Towards Distribution-Agnostic Generalized Category Discovery
Authors: Jianhong Bai
Zuozhu Liu
Hualiang Wang
Ruizhe Chen
Lianrui Mu
Xiaomeng Li
Joey Tianyi Zhou
YANG FENG
Jian Wu
Haoji Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Generalized Category Discovery
Open-world Recognition
Long-tail Learning
Contrastive Learning

Abstract: Data imbalance and open-ended distribution are two intrinsic characteristics of the real visual world. Though encouraging progress has been made in tackling each challenge separately, few works dedicated to combining them towards real-world scenarios. While several previous works have focused on classifying close-set samples and detecting open-set samples during testing, it's still essential to be able to classify unknown subjects as human beings. In this paper, we formally define a more realistic task as distribution-agnostic generalized category discovery (DA-GCD): generating fine-grained predictions for both close- and open-set classes in a long-tailed open-world setting. To tackle the challenging problem, we propose a Self-**Ba**lanced **Co**-Advice co**n**trastive framework (BaCon), which consists of a contrastive-learning branch and a pseudo-labeling branch, working collaboratively to provide interactive supervision to resolve the DA-GCD task. In particular, the contrastive-learning branch provides reliable distribution estimation to regularize the predictions of the pseudo-labeling branch, which in turn guides contrastive learning through self-balanced knowledge transfer and a proposed novel contrastive loss. We compare BaCon with state-of-the-art methods from two closely related fields: imbalanced semi-supervised learning and generalized category discovery. The effectiveness of BaCon is demonstrated with superior performance over all baselines and comprehensive analysis across various datasets. Our code is publicly available.

Towards Distribution-Agnosti Generalized Category Discovery
Jianhong Bai', Zuozhu Liu Hualiang Wang", Ruizhe Chen', Lianrui Mu', Xiaomeng Li,, Joey Tianyi Zhou", Yang Feng', Jian Wu², Haoji Hu 'Zhejiang University, "The Hong Kong University of Science and Technology, 3 'Centre for Frontier AI Research, A STAR,, "Angelalign Technology Inc Saanhonbbai@zju.o edu.cn
Abstract
Data imbalance and open-ended distribution are two intrinsic characteristics of the real visual world. Though encouraging progress has been made in tackling each challenge separately, few works dedicated to combining them towards real-world scenarios. While several previous works have focused on classifying close-set samples and detecting open-set samples during testing, it's still essential be able to classify unknown subjects as human beings. In this paper, we formally define more realistic task distribution-agnoste generalized category discovery (DA-GCD): generating fine-grained predictions for both close- and open-set classes in a long-tailed open-world setting. To tackle the challenging problem, we propose a Self-Balanced Co- (d-Advice contrastive framework (BaCon). which consists of a contrastive-lermingg branch and pseudo-labeling branch, working collaboratively to provide interactive supervision to resolve the DA-GCD task. In particular, the contrastive-leamingg branch provides reliable distribution estimation to regularize the predictions of the pscudo-labeling branch, which in turn guides contrastive learning through self-balanced knowledge transfer and proposed novel contrastive loss. We compare BaCon with state-of-the tfte-off-the-ar methods from two closely related fields: imbalanced semi- supervspe learning and generalized category discovery. The effectiveness of BaCon is demonstrated with superior performance over all baselines and comprehensive analysis across various datasets. Our code is available at: https::/p// / //g thub. com wwwmmmiiannngaaaiBBaaCo..
1 Introduction
Data imbalance [82, 79, 45] and open-ended distribution [14, 72, 17] are inherently intertwined with each other in the real visual world [40]. On the one hand, the frequency of different objects exhibits long-tailed distribution [48, 54] where a small portion classes dominate the distribution and many classes associated with only few examples, observed in healthcare [42, 75], autonomous driving [41, 36] and species classification [59]. On the other hand, humans or intelligent agents constantly encounter novel visual instances. Such complicated intertwined context poses great challenges to develop effective visual algorithms [28, 55] that apply to real-world scenarios [1, 12].
Tremendous efforts have been devoted to addressing the challenges of long-tail and open-set learning. In long-tail learning (LTL), previous works aim highlight minority categories by re-sampling [53,
83, 16] or re-weighting [5. 26] approaches, debiasing model prediction from statistical perspective [49, 43], decoupling feature learning and classifier training [29], or generalizing to semi-supervised [33, 31, 69] and self-supervised [38, 27, 81] scenarios. For open-world learning, existing work can be divided into three sub-domains based on how to handle open-set samples. Robust suni-super
Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Labeled Unlabeled Test Set
Imbolance
Number Somples
Long tall learning
Real- world scenorios (DA-SCD)
Novel
Not Availoble
Encountered in real world
12
Balance
Treditional mothods
world learning
Known
Manually annotated
Encountered real
10as Inclon
Close-set set
Open-set
Labeled
Unlabeled
(a) Data distribution of DA-GCD.
(b) Illustration of settings.
(c) Data composition.
Figure 1: Illustration of distributtion generalized category discovery (DA-GCD).
learning (SSL) [46] assumes that out-of-distribution (OOD) samples may undesirably appear in the unlabeled set, and several techniques [20, 77, 9] are designed to reject these potential open-set samples to improve model robustness. Another line of methods [3, 23, 37, 34, 61, 44, 64) attempt to detect them with close-set knowledge during testing and formulate the task as OOD detection. Furthermore, generalized category discovery (GCD) [60] suggests to classify open-set samples into each class [4, 56, 70]. An earlier work OLTR [40] manage to integrate LTL and open-world learning towards more realistic visual recognition. OLTR handles close-set data imbalance and is aware of open-set instances similar to OOD detection. However, this approach does not meet the requirements for real-world recognition scenarios as: a) it works in fully-supervised manner and does not effectively utilize large amount of unlabeled data in the real world. b) it can only detect unseen (open-set) samples rather than classifying them with novel visual concepts.
In this work, we consider a real-world yet more challenging setting: we assume that only portion of classes are collected and manually annotated (known classes) while massive unlabeled data from both known and open-set categories (novel classes) are available in the real world (Figure lc). These two types of data, both long-tailed form the semi-supervised training set, and the model is expected to classify all known and novel classes in balanced test set, as illustrated in Figure 1. We name this Generalized Category Discovery (DA-GCD) and contrast it with other related settings in Table 1. DA-GCD differs from previous open-world tasks because it considers realistic long-tail settings. It also differs from the pionecring work OLTR [40] by further considering the sumi-super setting and classifying novel classes, in up-to-date open-world settings [60, 4].
setting Distribution-Agnosti
The DA-GCD setting presents significant challenges that cannot be resolved with existing methods. Firstly, as confirmed by numerous studies in the field of long-tail learning, imbalanced datasets lead severe performance degradation and model bias [79, 29, 27]. This issue is even more challenging in DA-GCD, where the prior dataset distribution is unavailable in open- world scenarios. Therefore, most LTL methods [43, 50] could not be directly extended to resolve the DA-GCD task. Meanwhile, since only unlabeled samples are available for novel classes (as shown in Figure Ic), the performance on these open-set classes is severely limited. Existing contrastive-bar -baed methods in GCD [60, 56] struggle to optimize the feature space of unlabeled samples, resulting in poor alignment and uniformity [66]. Lastly. GCD methods lack tailored designs for imbalanced datascts, leading to significant performance degradation on long-taile -tiglt datasets (as discussed in Section 5.2).
To conquer the aforementioned challenges, we propose BaCon, a novel contrastive framework. BaCon comprises contrastive-leaming branch and a pseudo-labeling branch, which work collaboratively to provide interactive supervision to tackle the imbalanced recognition task under open-world scenarios. Specifically, the contrastive-learinn branch provides distribution estimation during training, which is used to regularize the prediction results of the linear classifier for better pseudo-labeling. On the other hand, the generated pseudo-labels are further sampled and debiased to re- balancee and provide additional supervision for the contrastive-lcariin branch. To fuse the knowledge of the two branches and learn a better feature space, we design novel pseudo-label-be based contrastive loss that clusters samples based on their positiveness scores.
Co- Co-Advice
We make the following major contributions. 1) We formulate a more realistic DA-GCD task that unifies long-tail and open-world learning and identify unique challenges that cannot be adequately addressed by existing techniques in both fields. 2) We design novel Delf-Balanced Co-advice contrastive (BaCon) framework for DA-GCD, which comprises contrastive-leaming branch and pseudo-labeling branch that work collaboratively to tackle the data imbalance and classify novel (open-set) categories simultaneously. 3) We conduct extensive experiments on various datasets and perform fine-gra -neigraided comparisons with SOTA basclines to validate the effectiveness of BaCon.

---

Table 1: Comparison of DA-GCD with other similar settings.
Known classes
Novel classes
Involved in training?
Setting
Scenario
Data Distribution
(novel classes)
Robust SSL
Semi-supervised Depend on method
Classify
Reject
Balanced
OOD Det
Classify
Detect
Depend on method
Balanced
GCD
Semi-supervised
Classify
Classify
Balanced
LT-SSL
Semi-supervised
Classify
N.A.
x
Long-tailed
OLTR
Fully-supervised
Classify
Detect
X
Long-tailed
DA-GCD
Semi-supervised
Classify
Classify
Long-tailed
2
Related Works
Open- Open-World World Learning tackling the scenario where some unknown (open-set) visual concepts or
categories are involved during training real-world applications. OOD detection [37, 24, 39], as well-known sub-field, considering a distributional shift (either semantic shift [23] or covariate shift
[35]) between the train and test stage, with the goal of detecting these OOD samples during testing. Robust comi-supervised learning [46, 20, 77, 9] apply different techniques to reject undesirable OOD samples the unlabeled set to improve model robustness. More recently, a line of work [60, 4, 56, 70, 52, 51] propose to classify the open-set samples according to their visual concepts. Nevertheless, these methods are built on the class balance assumption, and we observe a consistent performance degradation when generalizing to DA-GCD scenarios. Distribution-agnon is also investigated by [73], which focuses on the task of novel category discovery.
recognition
Long-Taie Tail Learning refers to scenario where the majority of samples belong to a few classes, while the minority classes have only small number of samples [63, 15]. Early research on long-tail learning mainly focused on fully-supervised scenarios, with representative methods that highlight the minority samples via re-samplnng [53, 83, 16], re- reightit [5, 26]. or knowledge transferring [68, 11, 76]. Long-tail learning under semi-supervvsed [33, 31, 69] and self-supervised [2, 74, 38, 27, 81, 78] settings are also widely investigated and have notable achievements. However, the requirement dataset distribution obstructs the application of fully/semi-superviseee methods in DA-GCD, while self-supervised methods would disregard the available label information for known samples.
the
Contrastive Learning [8, 22, 18] provide distinctive and transferable representations by controlling the instance similarity feature space, achieving significant progress in recent years. The following literature generalizes the idea to different scenarios [30, 62, 27], implements it for solving various downstream tasks [7, 71], and provides theoretical support [66, 67]. However, few attempts have been made at contrastive-bases open- open-world recognition. GCD [60] proposes performing supervised CL on labeled samples and unsupervised CL on all data. OpenCon [56] generating pseudo-positive pairs for closely aligned representations, but this paradigm could lead to another dilemma: the representations and pseudo-labels are interdependent, which means an inferior feature space (e.g.. lack of intra-class consistency) could lead to false positive pairs, and in turn deteriorate the learning of feature space. In short, those methods are incompatible with the proposed DA-GCD setting and have inferior representations for samples belonging to novel classes.
3 Preliminaries
3.1 Problem Formulation
As illustrated in Figure 1a, training set is composed of labeled subset D' = { v/}} xx)) and unlabeled subset x {(x) v/ E X x V.. where we don't have the label information for any images in novel classes, i.e., 3/ = (34 Da = = ()x ).,). Both D' and D" present a long-tail distribution with imbalance ratio Pt, Pr > 1. The objective of DA-GCD to assign labels disjoint balanced test set containing all classes in V The total number of classes in the whole dataset D is commonly regarded as a known prior [60, 4, 70], as we can effectively estimate through off- off-the-shehl shelf methods [21].
3

---

3.2 Revisiting Contrastive Learning
Contrastive- based representation learning methods [8. 22, 18, 30] aim find an embedding function to acquire optimal feature representation z E RR of the input image a E RRHW with z f(x), such that retains the discriminative semantic information of the input image. The general contrastive learning loss can be defined as:
exp(z: Zp/T)
(D)
Σ
log
D
P(1)
A())()) exp(z; za/T)
PEP(1)
where D is the training set. P(i) is the set of indices of positive pairs in A |-| denotes the operation to compute the cardinality, and is ttyper-paameete formulated as the two views of the same image [8, 22]. [30] further generalizes the contrastive loss into supervised scenarios where the positive set is the images that share the same ground truth label.
In unsupervised learning, the positive set is
4 Self-Balanced Co-Advice Contrastive Framework
As illustrated in Figure 2, design two-branch structure framework for DA-GCD. In Section 1. we propose the contrastive-lerming branch to estimate the training distribution and further regularize the outputs of the geuuo-labeling branch. We then describe techniques for self-balanced knowledge transfer from the pscudo-labeling branch to the contrastive-lcamiig branch Section 4.2. Finally, in Section 4.3, novel contrastive loss is proposed for better novel class learning.
4.1 Regularize Predictions with Dynamic Distribution Estimation
To enhance and re-balance the contrastive representation learning, we propose to learn an auxiliary classifier for pseudo-labeling. Our pseudo-labeling branch includes cross-entropy loss L, on labeled data, semi-supervised objective L. (e.g., cross pseudo supervision in [70]) on all samples, and regularization term Greg to align the predictions with the data distribution avoid non-activated classifiers. However, the distribution of the training set is agnostic in DA-GCD, and simply using the distribution of labeled set TD' or a balance prior results in inferior performance (Table 6a). Furthermore, we observed that estimating the distribution from the pseudo- branch itself could accumulate the estimation error and deteriorate model performance.
To this end, we perform estimation on the contrastive-leariing branch as an alternative to avoid the accumulation of estimation bias. Concretely, we perform k-means clustering on all samples in the training set to obtain C' clusters with size n, where C' = (2). and normalized the sample number n
to frequency æe. i.c., ne n > (n[c)). Moreover, we also need to determine the corresponding relationship between clusters and categories. Therefore, we use the Hungarian optimal assignment algorithm [32] map | clusters to each known class. For the remaining Y ||* clusters, sort them by the cluster size and assign them sequentially to the novel classes. Finally, we regularize the mean prediction of the pseudo-labeling branch with the aligned estimated distribution n.::
Greg = KL
Σ softmax((at((xx.)) || (align( )P
(2)
B
iEB
where B refers to the batch size during training, faie denote the backbone and the linear classifier of the auxiliary branch, KL[- [| iss the Kullback-Leiler (KL) divergence between the two distributions. and p E J0, 1] is a hyper-parameter used to smooth the target long-tailed distribution. To balance estimation accuracy and computation overhead, we re-estimate the dataset distribution every r epoch. The overall training objective of the pseudo-labelin branch is as follows:
Late L. 7/1 C, 72
During inference, we use optimal assignment to determine the correspondence between classifiers and cate- gories (detailed in Section 5.1). hence sequentially assigning the clusters to classes will not affect performance.

---

Dotoset Composition
Pseudo-labeling Branch
Test-time Forword
Labetied See
Casuffiation backbone
Classfier
Balanced
Sampling Detiviing
Contrastive bacibone
calmated Durriution
poutiveniss snare W
1110 Grvestinate coouls
Gustering Algyment
Features
Contrastive
Projector
Clustering
Controstive-learning Branch
Figure 2: Overview of the self-balanced co-advice contrastive framework (BaCon).
4.2 Self-Balanced Knowledge Transfer: Debiasing and Sampling
In Section 4.1, regularize the pseudo-labeling branch with the estimated dataset distribution Te acquired from the contrastive-lerning branch. In this step, we aim to transfer the knowledge from the pseudo-labeling branch to the contrastive-leaminng branch in turn and further enhance the representation learning, which is also beneficial to the estimation of data distribution т. However, the long-tail distribution of the underlying dataset places additional requirements transferring. On one hand, we should ensure that the knowledge to be transmitted is not affected by the long-tailed distribution. Meanwhile, hope that this process help the contrastive-learinng branch cope with the issue of data imbalance and the lack of supervision for novel classes.
knowledge
We design debiasing and sampling step of the pscudo-labels to meet the two aforementioned requirements respectively for knowledge transfer. First, we apply post-hoc logits adjustment [43] based the estimated distribution the predicted logits fata (2;)) to eliminate the bias caused by long-tailed distribution:
P: = (f - k logmr)))
where p. denotes the rectified class probability prediction of sample æ, and l: is in [43]. Next, in order to re-balance the learning process and filter fow-precision pseudo-labele we propose to sample the unlabeled instances. Specifically, instances in a training batch Ps = {p= = -..... B} according to their prediction class c, where C arg max P;. Formally, the class-wise sampling rate SR can be defined as:
sample the pseudo-labels of unlabeled
π.: min v.
CE VB.
SR
(5)
otherwise,
min
where n. the estimated distribution introduced in Section 4.1, VB V denotes the set of labels which involved in the current batch, and a, 3 E [0, 1] are two hyper-parameters. Setting a 3 0 = 1 means setting the sampling rate to be inversely
means sampling all pseudo-labels, while proportional to the estimated number of samples in its category. Note that we prioritize selecting samples with higher prediction confidence in each class to remove the potentially false aseudo-labels The sampled instances (M(D.)) complement the original long-tailed distribution to serve re-balance term for the labeled subset {Dt} and also provide additional supervision for novel classes. With this debiasing and sampling step, the sampled instances (D, UD,UME MDD)) and their corresponding rectified pseudo-labels mitigate the impact of the imbalance issue and compensate the unlabeled open-set samples simultancously.
5

---

4.3 Pseudo-label-aseed Soft Contrastive Learning
In Section 4.1 and Section 4.2, we generate pseudo-labels via the pseudo-labeling branch with help from the contrastive-leaming branch. Those pseudo-labels could provide additional information to improve the contrastive- fully leverage the probabilistic information in pseudo We design a pair-wise positiveness score to adjust the contribution of different samples to the anchor instance. For image pair (æ,, æy), the positiveness score Wij (p, Pj). Waj is obtained by calculating the similarity of the rectified class probability distribution between æ; and z. In practice, We implement the similarity metric with the dot product operation, which can be mathematically interpreted the probability of instance 2; and zy belonging to the same class. We compare different definitions of the positiveness score in Section 5.4. Finally, we formulate the soft contrastive loss by incorporating into Eq. 1:
branch as well. particular, we adopt a soft contrastive strategy to
Coof (D)
1 ID
Σ IED
Σ Wij" log
exp(zi z5/
EJEA(O)
Wij JEA())
EDEA(S) exp(zi Za/T)
Claim 1. Assume we train model 0 with the proposed soft contrastive loss, and æ; and 2 are two samples within a training batch B. The optimal value of the contrastive logit Pij is
Who
where
Σ mig
Waj is the corresponding positiveness score for the sample pair (2i,,.)) in Eq. 6.
The proof can be found in Appendix A. Claim 1 indicates that minimizing Eq. 6 encourages the similarity of features between two samples to be proportional to the corresponding positiveness score. In this way, we effectively transfer knowledge from the pseudo-labeling branch to contrastive learning. The overall training objective of the contrastive-lcarning branch consists of unsupervised CL loss on all samples, a supervised CL loss on labeled instances, and the proposed soft CL loss on both labeled and sampled unlabeled subset:
Lcon = CD) + + + 12 (D,UM((.))
(7)
The pipeline of our method is illustrated in Figure 2, and the pseudo-code of BaCon can be found in Appendix B. It's worth noting that the backbone of the two branches is shared and frozen during training (except for the last block). Therefore the two-branch structure only slightly increases computational overheads. During inference, we only utilize the backbone of the contrastive-learing branch and obtain predictions via k-means clustering on the backbone features.
5 Expriments
5.1 Experimental Setup
Data Preparation We conduct experiments on four popular datasets. CIFAR-IO-LTICIFAN 100-LL are long-tail subsets sampled from the original CIFARIO/CIFAR 100 [10]. We set the imbalance ratio to 100 in default. 100-- is proposed by [27] with 12K images sampled from ImageNet- 100 [57] with Pareto distribution. Places-LT [40] contains about 62.5K images sampled from the large-scale scene-centric Places dataset [80] with Pareto distribution. For each dataset, we first sub-sample [Dee classes from the whole long-tail -til datasets to stimulate known classes and treat the rest as novel classes, then sub-sample 50% instances in each known class as D and concentrate others with all samples in novel classes to form D The default D& for CIFAR- LIFAR-IO-LT, SIFAR-100-LY, MongeNNet1100-L7 Places-LT are 5, 80, 50, 182 respectively. We also evaluate BaCon with different ratios of Die and D..l set up D' with few annotated samples, Pt * Pu. in E.
Evaluation Protocols and Metrics Following previous works [60, 4], we calculate the accuracy by assigning the model prediction (cluster result) ÿ to ground truth y via the Hungarian assignment algorithm [32]. With the assignment fixed, we report the overall classification accuracy on 3%. (Old), )), (New), and D% U Yn, (All). To observe the impact of long tail distribution on performance, we evenly divide Y' and Vn into three disjoint groups (Many, Median, Few) respectively in terms of the number of instances of each class. We also calculate the standard deviation (Std) of the accuracy of the three groups, which quantitatively analyze the balancedness of feature space [27].

---

Table Test accuracy (%) and balancedness (Std.) of existing methods on CIEAR-I00-LT.
Known
Novel
Overall
Method
Subset
Manyt
Med
Fewt
Std
A
Manyt
Medt
Fewt
Stdy
Allf
D
66.2
75.4
73.6
4.0
71.8
60.3
54.1
51.5
3.7
55.2
68.5
GCD
D;
75.9
69.4
52.9
9.7
65.5
41.9
54.2
49.8
5.1
49.0
62.2
D2
75.1
75.3
72.2
1.4
74.2
60.2
58.8
59.7
0.6
59.5
71.3
SimGCD
D;
72.8
71.1
34.6
17.6
59.8
32.9
25.8
15.5
7.1
24.8
52.8
ABC
DD
77.4
51.6
16.3
25.0
48.5
20.8
32.0
5.9
10.7
20.8
43.0
DARP
D; 74.8
55.2
33.4
16.9
54.5
29.9
30.1
10.0
9.4
24.0
48.4
Table 3: Test accuracy (%) on four generic long-tale taigetailed image recognition datasets. (bold: best performance among all methods, underline: best performance among the baseline methods.)
CIFAR-10-LT
CIFAR-100-LE ImageNet- 100-LT
Places-Li
Methods
Old
New
All
Old
New
All
Old
New
All
Old
New
All
ABCT
77.7
20.2
48.9
48.5
20.8
43.0
DARP*
77.6
35.2
56.4
54.5
24.0
48.4
TRSSL
78.4
66.8
72.6
58.7
35.8
54.1
ORCA
77.5
55.6
66.6
55.0
30.8
50.1
69.3
29.0
49.2
21.5
6.9
14.2
SimGCD
75.1
41.5
58.3
59.8
24.6
52.8
81.1
33.4
57.8
31.4
18.2
24.8
GCD
78.5
71.7
75.1
65.5
49.0
62.2
81.0
76.8
78.9
29.8
22.7
26.2
OpenCon
87.2
47.2
67.2
64.2
40.9
59.6
83.3
42.8
63.0
30.6
12.4
21.6
BaCon-O 83.3
78.0
80.7
66.5
69.6
67.1
84.2
80.0
82.1
30.7
25.6
28.1
BaCon-S
94.2
88.1
91.1
67.4
66.5
67.2
84.6
82.8
83.7
31.1
28.4
29.9
Training Settings Following GCD [60], we load a DINO [6] pre-trained ViT-B/16 [13] as the backbone for both branches and only fine-tune the last block for all datasets. The classification head and projector are randomly initialized. To achieve a fair comparison, we replace the backbone model of all baseline methods with the ViT pre-trained by DINO (the same as ours), and denotes adapted methods. More training details can be found in Appendix C.
5.2 Migration of Existing Methods to DA-GCD
Here first state that migrating existing methods from two similar scenarios (GCD [60] and imbal- anced SSL [31]) to the proposed DA-GCD result inferior performance due to the co-occurrence of data imbalance and open-set samples. Existing GCD methods, whether based classifier [60] or non-parametric classification methods [70], are insufficient in addressing the issue of data imbalance. In Table 2, report two SOTA methods (GCD [60] and SimGCD [70]) performances the bal- anced set DD, and its long-tailed counterpart DD, It's observed that they exhibit significant performance degradation (especially on minority classes) on D, compared their performance on Dg. While imbalanced SSL methods [33, 31] can barely classify novel class samples under the DA-GCD setting.
5.3 BaCon's Accuracy, Balancedness and Versatility
The results of the BaCon in various datasets are presented in Table 3. BaCon-O and BaCon-S refer to training the pseudo-labeling branch in ORCA-style [4] and SimGCD-style [70] respectively. We compare BaCon with two SOTA imbalanced-SSI methods ABC [33] and DARP [31], and five latest open-world recognition techniques including contrastive- based methods GCD [60] and OpenCon [56] and classifierbbased methods TRSSL [52], ORCA [4] and SimGCD [70]. As shown in Table 3, BaCon significantly outperforms all these methods by large margin under various datasets. Specifically, on CIFAR-I00-LL BaCon-S outperforms the best baseline by 17.5% and 5.0% on novel classes accuracy and overall accuracy respectively.
We don't 't compare BaCon with OLTR [40] since they perform in supervised scenarios without the presence of novel classes (Table 1), resulting in inferior performance novel class classification.
7

---

(a) DINO.
(b) GCD.
(c) Ours.
Figure 3: t-SNE visualization the test of CIFAR-10.
Table
Test accuracy (%) and balancedness (StdL) CIFAR-IOO-LT.
Known
Novel
Overall
Methods
Manyt
Medt
Fewt
Std|
Allf
Manyt
Medt
Few?
Std
Allf
ABC
77.4
51.6
16.3
25.0
48.5
20.8
32.0
5.9
10.7
20.8
43.0
DARPt
74.8
55.2
33.4
16.9
54.5
29.9
30.1
10.0
9.4
24.0
48.4
TRSSL
78.8
73.0
24.1
24.3
58.7
32.7
50.9
18.8
13.1
35.8
54.1
ORCAt
73.6
60.0
31.0
17.8
55.0
47.5
28.4
17.2
12.5
30.8
50.1
SimGCD
71.1
72.8
34.6
17.6
59.8
32.9
25.8
15.5
7.1
24.6
52.8
GCD
75.9
69.4
52.9
9.7
65.5
41.9
54.2
49.8
5.1
49.0
62.2
OpenCont
86.8
69.7
35.8
21.2
64.2
55.8
48.9
15.3
17.7
40.9
59.6
BaCon-O 72.9
64.7
62.0
4.6
66.5
72.6
70.6
65.3
3.1
69.6
67.1
BaCon-S
74.4
67.1
60.8
67.4
73.7
66.1
59.8
5.7
66.5
67.2
Table 5: Test accuracy (%) on CIFAR-100-LT with different imbalance ratio p.
CIFAR-100-LT (p: Ps. p)
p 20
50
p 100
p 150
Methods
Old
New
All
Old
New
Old
New
All
Old
New
All
ABCt
64.0
23.0
55.8
53.7
21.5
47.3
48.5
20.8
43.0
41.9
18.5
37.2
DARP
66.8
27.4
58.9
58.9
25.1
52.1
54.5
24.0
48.4
49.7
23.3
44.3
TRSSL
71.8
43.6
66.2
62.9
49.0
60.1
58.7
35.8
54.1
55.2
33.3
50.8
ORCAf
66.9
37.7
61.0
61.7
30.0
55.3
55.0
30.8
50.1
51.0
36.5
48.1
SimGCD
71.8
23.1
62.1
63.7
23.4
55.7
59.8
24.2
52.8
57.5
19.9
50.0
GCD
69.8
58.7
67.6
66.6
59.7
65.2
65.5
49.0
62.2
65.4
50.2
62.4
OpenCont
76.3
44.3
69.9
71.2
42.0
65.4
64.2
40.9
59.6
59.7
45.3
56.8
BaCon-O 71.5
67.9
70.8
70.0
63.7
68.7
66.5
69.6
67.1
67.4
64.2
66.8
BaCon-S 71.6
68.5
71.0
70.4
64.2
69.2
67.4
66.5
67.2
67.0
64.3
66.5
To better evaluate the impact of dataset imbalance, we make a fine-grained comparison in Table 4. The lower standard deviation of the performance of (Mary, Median, Few) on both known and novel classes suggest BaCon yields a more balanced feature space. Besides, our method significantly improves the performance of the minority classes without sacrificing the accuracy of the majorities.
We also report the performance of BaCon when the training set has a different imbalance ratio p in Table 5. Note that we set the imbalance ratio of 'D' and D to be the same, i.e., Pt Pa. Empirical results with Pt * Pv. are Appendix Notably, BaCon achieves the best overull accuracies on all settings, indicating the universal effectiveness across various scenarios, and the performance gain becomes more evident under extremely imbalanced data. Also, open-world recognition emphasizes novel class learning, where BaCon achieves more notable performance gain.

---

Strategy
Old
New
All
Strategy
Old
New
All
Loss
Old
New
All
reg with n
76.4
13.2
44.8
baseline
65.1
52.0
62.5
baseline
65.1
52.0
62.5
balance prior
75.6
31.7
58.3
vanilla
66.3
62.4
65.5
hard
66.0
60.7
64.9
cls k-means
76.3
36.4
56.4
w/ debiasing
67.1
64.1
66.5
soft
67.4
66.5
67.6
con k-means
81.7
42.3
62.0
w/ sampling
66.8
65.6
66.6
oracle
85.0
44.2
64.6
w/ both
67.4
66.5
67.2
(a)
Regularization Term.
(b) Debiasing and Sampling.
(c) Loss Design.
Interval r Old
New
All
Function
Old
New
All
k
Old
New
All
1
67.3
70.2
67.9
L1
64.6
64.0
64.5
0
65.8
67.8
66.2
5
66.4
70.6
67.2
L2
63.2
60.9
62.7
0.25
66.1
69.2
66.7
10
67.4
66.5
67.2
cosine
67.8
64.1
67.0
0.5
67.4
66.5
67.2
25
66.5
67.5
66.7
dot
67.4
66.5
67.2
1,0
67.5
65.0
67.0
50
66.6
65.8
66.4
1.5
66.9
66.2
66.8
(d) Re-estimate Interval r.
(c) Similarity Function.
Hyper-parameter k.
Table 6: BaCon ablation experiments. For each ablation, we report test accuracy (%) of known, novel and all classes, denote as 'Old', 'New', and 'All'. Our default settings are marked in orange
5.4 Analysis and Ablation Study
The effectiveness of Lreg Recall in Section 4.1, we propose to regularize the predictions of the pseudo-labeling branch by the estimated train-set distribution. In Table 6a, we show the performance of the seeuo-labellng branch on ImageNet- -ooseNeet-O--LL with different estimation strategies. 'Oracle' denotes we use the true distribution π of DD (unknown in practice) as the target distribution in Eq. upper bound of the performance. Compared to previous works (ORCA and SimGCD) that use a balance prior, regularizing the predictions with oracle long-tailed distribution significantly improves the performance both known and novel categories, showing the importance of the distribution estimation process. Mcanwhile, the similar results achieved by our estimation strategy imply ne could be reliable proxy to Furthermore, we investigate whether two alternative estimation strategies could help the pseudo-labeling branch: 1), only regularize known classes prediction with TDi 2), perform k-means clustering on the feature of the pseudo-labeling branch, and they all results in inferior accuracy and could in turn deteriorate the contrastive learning process.
2. and it serves
The effectiveness of sampling & debiasing In Section 4.2, we suggest adjusting the prediction logits according to the estimated distribution for debiasing and sampling unlabeled instances for re-balancing. We adopt step-by-step ablation experiments to the proposed approaches in Table 6b. 'Baseline' refers to not using the proposed pseudo-label-ban contrastive loss, and it leads to inferior performance novel classes since they only get supervision from the self-supervised contrastive loss. 'Vanilla' means incorporates the designed sscudo-label-basee loss into the optimization objective with Eq. 7 without the debiasing and sampling module. It brings performance gain on novel classes due to the leverage of additional supervision in pseudo labels. Finally, we achieve higher test accuracy on both known and novel classes by combining the two techniques together.
Definition of the pseudo-label based contrastive loss In Section 4.3, we design a novel soft contrastive loss based on pseudo-labels to transfer the knowledge of fat into Scon As opposite, we could also construct the loss hard manner where we formulate the positive pairs on top the prediction class with the largest logit and further perform the supervised contrastive loss. Intuitively, the hard design discards the probability distribution information and is more susceptible to the potential false pseudo-labeli while the soft contrastive loss utilized in our method could help alleviate the influence of erroncous pseudo-labels. The empirical results also support the intuition, as shown in Table 6c, the proposed C.., (soft) outperforms the supervised CL loss (hard) by large margin. This phenomenon is also observed in knowledge distillation [25, 65] that transferring knowledge by using soft labels rather than one-hot predictions achieves better performance.
Changing of metrics and co-efficient to adjust the contribution of different samples to the anchor instance. We apply various similarity metrics on the pseudo-labels of a; and T.p to calculate the positiveness score Wip- The results are exhibited in Table 6e, we incorporate dot product in BaCon due to the superior performance
The proposed soft CL loss introduces pair-wise

---

in practice. We also show the effect of hyper-paramettern parameters involved in BaCon. Table 6d shows the empirical results of changing estimate interval r. We observe similar accuracy when scaling r. hence we use relatively large interval to reduce the computational overhead. Table 6f shows the classification accuracy when changing Eq. 4.
Feature Visualization We adopt the widely-used T-distributed stochastic neighbor embedding (t- SNE) [58] for feature space visualization. We show the feature space of BaCon (ours) with DINO [6] pre-trained model (without fine-tuning) and GCD [60] in Figure 3, different colors represent data belonging distinct categories. Compare to vanilla DINO (Figure 3a) and GCD fine-tuned model (Figure 3b), BaCon has more compact features for samples within the same category, suggesting better intra-class consistency, and also a larger margin between different classes.
6 Conclusion and Limitations
In this paper. we formulate more realistic DA-GCD task that umifies learning, and design a novel framework BaCon for DA-GCD to fight against data imbalance and classify open-set categories simultaneously. Extensive experiments effectiveness of the proposed method. There are nevertheless some limitations. First, our work focuses on the imbalanced open-world classification task, how to extend to object detection and instance segmentation is also worth exploring. Second, while the computation increment not severe, how to further accelerate the proposed method will be explored in the future. We hope our work can promote the research on imbalanced open-set recognition in real-world applications.
and spoldworld
various datasets verify the
Acknowledgements
We thank the anonymous NeurlPS reviewers for providing us with valuable feedback that greatly improved the quality of this paper!
This work is supported by the Zhejiang Provincial Key RD Program of China (Grant No. 2021C01119), the National Natural Science Foundation of China (Grant No. U21B2004, 62106222), and the Core Technology Research Project of Foshan, Guangdong Province, China (19200010004988, the Natural Science Foundation of Zhejiang Province, China (Grant No. LZ23F020008) and the Zhejiang University-Angelalign Inc. R&DD Center for Intelligent Healthcare.
References
[1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06500 2016.
[2] Jianhong Bai, Zuozhu Liu, Hualiang Wang, Jin Hao, YANG FENG, Huanpeng Chu, and Haoji Hu. On the effectiveness of out-of-distribution data in self-supervised long-tail learning. In The Eleventh International Conference on Learning Representations, 2022.
[3] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
[4] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In Interna- tional Conference on Learning Representations, 2022.
[5] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distribution-awaar margin loss. Advances in neural information processing systems, 32, 2019.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[7] Krishna Chaitanya, Ertune Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33: 13:16256-12558 2020.
10

---

[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[9] Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In Pruceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3569-3576, 2020.
[10] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268 9277, 2019.
[11] Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4109-4118, 2018.
[12] Thomas G Dietterich. Steps toward robust artificial intelligence. Ai Magazine, 38(3):3-24, 2017.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 / 1929, 2020.
[14] Nick Drummond and Rob Shearer. The open world assumption. In ESI Workshop: The Closed World of Databuses meets the Open World of the Semantic Web, volume 15, page 1, 2006.
[15] Marawan Elbatel, Robert Martí, and Xiaomeng Li. Fopro-kd: Fourier prompted effective knowl- edge distillation for long-tailes medical image recognition. arXiv preprint 1rXiv:2305.17421,, 2023.
[16] Yonatan Geifman and Ran Elniv..v. Deep active learning over the long tail. arXiv preprint arXiv:1711.00941, 2017.
[17] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recogni- tion: survey, IEEE transactions on pattern analysis and machine intelligence, 43(10))3614- 3631, 2020.
[18] -aan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self- self-supervised learning. Advances in neural information processing systems, 33:21271 21224, 2020.
[19] Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. In International Conference Machine Learning, pages 8082-8094. PMLR, 2022.
[20] Lan-Zhe Guo, Zhen- Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi- Hua Zhou. Safe deep semi- supervised learning for unscen-class unlabeled data. In International Conference Machine Learning, pages 3897-3906. PMLR, 2020.
[21] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409, 2019.
[22] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9733 2020.
[23] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and -isti--distribution examples in neural networks. International Conference on Learning Representations, 2017.
[24] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2018.
11

---

[25] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1e0::0233303/11 2015.
[26]
Muhammad Abdullah Jamal, Matthew Brown, Ming- Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7610-7619, 2020.
[27] Ziyu Jiang, Tianlong Chen, Bobak Mortazavi, and Zhangyang Wang. Self-damaging con- trastive learning. In International Conference on Machine Learning, pages 4927-4939, PMLR, 2021.
[28] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2021.
[29] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
[30] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673 2020.
[31] Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang. Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pseudo-labe for imbalanced sumi-supervised learning, Advances in neural information processing systems, 33:14567-14479, 2020.
[32] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1 (1--2):83-97) 97, 1955.
[33] Hyuck Lee, Seungjae Shin, and Heeyoung Kim. Abe: Auxiliary balanced classifier for class- imbalanced semi sumi-supereiied learning. Advances in Neural Information Processing Systems, 34:7082-7094, 2021.
[34] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distriuttion samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
[35] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEÉ international conference on computer vision, pages 5542 5550, 2017.
[36] Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit- Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection autonomous driving. In European Conference Computer Vision, pages 406-423. Springer, 2022.
[37] Shiyu Liang, Yixuan Li, and R Srikant. Enhaneing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[38] Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance, In International Conference on Learning Representattinss 2021.
[39] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection, Advances neural information processing systems, 33:21464-21475. 2020.
[40] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[41] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, al. One million scenes for autonomous driving: Once dataset. arXiv preprint arXiv:2106. 1137, 2021.
12

---

[42] Yassine Marrakchi, Osama Makansi, and Thomas Brox. Fighting class imbalance with con- trastive learning. Medical Image Computing and Computer Assisted InterventionMMICCM 2021:
24th International Conference, Strasbourg, France, September 27-October I, 2021, Proceedings, Part III 24, pages 466-476. Springer, 2021.
[43] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail tail learning via logit adjustment. In International Conference on Learning Representationss 2021.
[44] Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of-distribution detection with posterior sampling. In International Conference on Machine Learning, pages 35650-15666 PMLR, 2022.
[45] Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: A review. IEEE Transactions on Pattern Analysis & Machine Intelligence, 43(10):3388-3415, 2021.
[46] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Real- istic evaluation of deep supi-super learning algorithms. Advances in neural information processing systems, 31, 2018.
[47] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Veiioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[48] William J Reed. The pareto, zipf and other power laws. Economics letters, 74(1):15-15 2001.
[49] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in neural information processing systems, 33:4175-41866 2020.
[50] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances neural information processing systems, 33:4175-4186, 2020.
[51] Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openldn: Learning to discover novel classes for open-world semi-supervised learning. In European Conference on Computer Vision, pages 382-401. Springer, 2022.
[52] Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. Towards realistic semi-supervised learning. In European Conference on Computer Vision, pages 437-455, 2022.
[53] Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference on computer vision, pages 467-482. Springer, 2016.
[54] Merrielle Spain and Pietro Perona. Measuring and predicting importance of objects in our visual world. 2007.
[55] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144-157, 2021.
[56] Yiyou Sun and Yixuan Li. Opencon: world contrastive learning. Transactions of Machine Learning Research, 2023.
[57] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference computer vision, pages 776-794, Springer, 2020.
[58] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
[59] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 8769-8778, 2018
13

---

[60] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 7492-7501, 2022. 2022.
[61] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
[62] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico:
Contrastive label disambiguation for partial label learning. International Conference on Learning Representations, 2022.
[63] Hualiang Wang. Siming Fu, Xiaoxuan He, Hangxiang Fang, Zuozhu Liu, and Haoji Hu. Towards calibrated hyper-sphere representation via distribution overlap coefficient for long- tailed learning. In European Conference on Computer Vision, pages 179-196. Springer, 2022.
[64] Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. Clipn for zero-shot ood detection: Teaching clip to say no. arXiv preprint arXiv:2308. /2213, 2023.
[65] Huan Wang, Suhas Lohit, Michael N Jones, and Yun Fu. What makes a" good" data aug- mentation knowledge distillation-a statistical perspective. Advances in Neural Information Processing Systems, 35:1 35:13456-1333 13469, 2022.
[66] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning. pages 9929 -9399.9999933 PMLR, 2020.
[67] Yifei Wang, Qi Zhang. Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. International Conference on Learning Representations, 2021.
[68] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. Advances in neural information processing systems, 30, 2017.
[69] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced semi emi-suuervised learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10857-10866, 2021.
[70] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. A simple parametric classification baseline for generalized category discovery. arXiv preprint arXiv:2211.11727, 2022.
[71] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Rision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt III 16, pages 574-591. Springer, 2020.
[72] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized dustt---ist detection: A survey, arXiv preprint arXiv:21 10. 11334, 2021.
[73] Muli Yang, Liancheng Wang, Cheng Deng, and Hanwang Zhang. Bootstrap your own prior: To- wards distributiin-agnostie novel class discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3459-3468, 2023.
[74] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class- imballnceadddd learning. Advances in neural information processing systems, 19999-999311,, 2020.
[75] Zhixiong Yang, Junwen Pan, Yanzhan Yang, Xiaozhou Shi, Hong- Yu Zhou, Zhicheng Zhang, and Cheng Bian. Proco: Prototype-aware contrastive learning for long-tailed medical image classification. In Medical Image Computing and Computer Assisted Intervention 2022: 25th International Conference, Singapore, September 18-22, 2022, Proceedings, Part VIII, pages 173-182. Springer, 2022.
14

---

[76] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5704-5713, 2019.
[77] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for supervipee learning. In Computer Vision-ECCV 2020: 16th European Conference,
open-set semi- semn---ttemm----ee Glasgow, UK, August 23-28, 2020, Proceedings. Part XII 16, pages 438-454. Springer, 2020.
[78] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Advances in Neural Information Processing Systems, 35:34077-34090, 2022.
[79] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596. 2021.
[80] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452)144521464, 2017.
[81] Zhihan Zhou, Jiangchao Yao, Yan-Feng Wang, Bo Han, and Ya Zhang. Contrastive learning with boosted memorization. In International Conference on Machine Learning, pages 27367-27377. PMLR, 2022.
[82] Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 915-922, 2014.
[83] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289-305, 2018
15

---

A Proof for Claim 1
Proof. We start with recalling some related notations and definitions. In Section 4.3, we design novel soft contrastive loss by incorporating pair-wise positiveness score ur in Eq. 1:
exp(z: Zj/T)
(D)
Σ IED
Σ
Wij log
(8)
ESEA(O)
wi jEA(()
EoEA(S) exp(zi 2a/T)
where D is the training set, A(i) is the index of the set of instances involved in the same batch, 2; is the contrastive feature for instance ai and the temperature papee--arameter.
For each instance 3g the soft contrastive loss fa (ax)) be written as:
exp(zi ZT/T)
Ceoft (x:) C Σ
w log Ene A())(i) exp(zi Za Za/T)
JEA(()
where C. Escats wis
a constant value for fixed w, then the contrastive logit P.y is defined as:
exp(z, ZT/T)
Pij Ane exp 2; Za Za/T)
(10)
Then, we omit the constant C' in Eq. 9 and solve the optimization problem with the Lagrange multiplier. The problem is defined as follows:
Minimize: S(a,,.. PPi)) En (wix log Pik),
(11)
Subject to: g(Pa,'. PPn) Eh Pik
O,
where n refers to the number of instances in the training batch, i.e., n B [A(i). The Lagrangian function of Eq. 11 and its corresponding partial derivatives are:
C(pa, ... Pin A)
(wik log + A(
Pib - 1),
(12)
k=1
8C.
Wiy Pii
+ =
8ps
(13)
8C
km Pij
0
From Eq. 13, the optimal value of Pij pl Els was which concludes the proof for Claim 1.
Claim indicates that minimizing Eq. 6 encourages the similarity features between two samples to be proportional to the corresponding positiveness score. In this way, we effectively transfer knowledge from the pseudo-labeling branch contrastive learning.
16

---

B Pseudo-code
We summarize the pipeline of self-Balanced Co-advice contrastive framework (BaCon) in Algorithm 1, and the source code can be found here.
Algorithm The overall pipeline of BaCon.
Train set D (Di U )"), model parameter Seon: Jats, train epoch T, warm-up epochs w, estimation interval r, sampling rate a. 3, similarity metric Sim (-), hyper-parameter and k. Output: Trained model parameter 0r.
Input:
Initialize: Load DINO [6] pre-trained parameters for the backbone in two branches and the classifier and projector is randomly initialized.
for epoch = ..... T- do if epoch%r then
Re-estimate the training set distribution and re-place л. (Section 4.1);
end if'
Regularize the predictions of the classifier with Eq. 2;
Train fei with Eq. 3; Compute the self-supervised CL loss L (D) and supervised CL loss L&g (D)) with Eq. 1; if epoch 2 w then
Debiasing pseudo-labels with Eq. 4; Obtain the sampled pseudo-labels M (D") with sampling rate Eq. 5; Calculate the pair-wise posifiveness score w with w Sim(p: P;);
Compute the soft CL loss Ceaft (D' (D")) with Eq. 6;
Train fcon with Eq. 7.
else
Train Seom with (D) + (D') end if end for
After training, we test the model via k-means clustering on the backbone feature of the contrastive- learning branch. The test strategy is summarized in Algorithm 2.
Algorithm 2 The test stage strategy of BaCon.
Input: The balanced test set D' = +(x,,, vi) X' x V, fine-tuned backbone of the contrastive- learning branch fo. Output: Classification accuracy Acc.
for æ. E DD' do
Obtain the feature of an via fo
end for
Perform k-means clustering on all the features test samples with the cluster number = [D']
Calculate the optimal assignment between clusters and classes by Hungarian algorithm [32];
Compute the test accuracy Acc based on the optimal assignment.
17

---

C Detailed Experimental Settings
In this section, we provide comprehensive overview of the experimental setup, expanding upon the details presented in Section 5.1.
C.1 Benchmark Datasets
In Section 5.3, we present our experimental findings based on four generic long-tailed image recogni- tion datasets. CIFAR-IO-EE and CIFAR-I004 which are first introduced by [10]. Both datasets are long-tail subsets sampled from the original CIFAR-10 10 and CIFAR-100. The imbalance ratio is defined the ratio between the number of instances in the largest class and the smallest class. In our experi- ments, we default the imbalance ratio to 100 to reflect the performance disparities between classes. ImageNet- -eo-eeett-O-LL.. proposed by [27]. which comprises 12K images sampled from the ImageNet- 100 dataset [57] using a Pareto distribution. The instance number of each class in the training set ranges from 1,280 to 5. We also incorporate the Places dataset [80], large-scale scene-centric dataset, in our experiments. We utilize Places-LT [40], which consists of approximately 62.5K images sampled from the Places dataset using Pareto distribution. The training set of Places-LT is ranging from 4,980 to 5. A summary of the dataset statistics is provided in Table 7.
Table 7: Statistics of datasets.
Dataset CIFAR-IO-LT CIFAR-100-LT
Long-tail type
Imbalance ratio p #class
#train images
#test images
Exp
100
10
11.2K
10.0K
Exp
100
100
9.8K
10.0K
1000--
Pareto
256
100
12.0K
5.0K
Places-LT
Pareto
996
365
62.5K
36.5K
C.2 Construction of Training and Testing Sets
To construct the training set for distribution aggnosiiii generalized category discovery (DA-GCD), first sub-sample 15 classes from the entire long-tail datasets (introduced in Section C.1) to stimulate known classes and treat the rest novel classes, then sub-sample 50% instances in each known class as D' and concentrate others with all samples in novel classes to form D The default [D/e] for CIFAR-10-
poageNet--100-11 and Places-LT are 5, 80, 50, and 182 respectively. Our default training set of each dataset is summarized in Table 8. We also evaluate BaCon with different Vk. different labeled ratios rz (default rı 50%), and different imbalance ratios of both labeled set and unlabeled set.
CIFAR-100-LL7
Table 8: The default setting of DA-GCD's training set.
Known classes
Novel classes
Labeled images
Unlabeled images |
Dataset
D
|D'
CIFAR-IO-LE
5
4.5K
6.7K
CIFAR-100-
80
20
3.8K
6.0K
ImageNet-100-11
50
50
3.2K
9.0K
Places-LT
182
183
20.8K
41.7K
For the test set, we use a disjoint balanced set as common practice in the field of long-tail learning [29, 31, 27]. It's worth noting that the test set contains both known and novel classes, and the model is required to classify test set instances to every single class.
C.3 Implementation Details of BaCon
We implement all our techniques using PyTorch [47] and conduct the experiments using RTX3090 GPU. Following GCD [60], we load DINO [6] pre-trained ViT-B/16 [13] as the backbone for both
18

---

branches. The classification head and projector are randomly initialized. We only fine-tune the last block of the backbone, the classification head in the pseudo-labeling branch, and the projector in the contrastive-leariis
branch. We use the output of [CLS] token with a dimension of 768 as the backbone feature for an input image. The classifier is implemented with single fully-connected layer, and we use the cosine classifier due to its competent empirical results in long-tail learning [29]. The projector is an MLP with an output feature dimension of 65536. We train with a batch size of 256, and an initial learning rate of 0.1 decayed with cosine schedule. We train for 200 epochs on each dataset. The temperature is set to 1.0, the hyper-parameter p and k is set to 0.5, the sampling rates 0.8, 0.5, and the re-estimate interval r is 10 (epochs). We use dot product as the similarity metric to define the positiveness score w.
19

---

D Introduction of Baseline Methods
In our paper, we formally define more realistic task called distribution-agen -agnnosii generalized category discovery (DA-GCD) and design novel framework BaCon for the challenge setting. To evaluate the effectiveness of BaCon, we compare the proposed method with SOTA techniques two closely related fields: imbalanced semi-supervised learning (imbalanced SSL) and generalized category discovery (GCD). this section, we provide detailed introduction of baseline methods (including ABC [33] and DARP [31] in imbalanced SSL; GCD [60], ORCA [4], OpenCon [56]. and SimGCD [70] in GCD).
Imbalanced SSL [31, 33, 69, 19] considers the scenario where the training set has a long-tailed distribution and only small subset of data labeled. DARP [31] suggests refining pseudo-labels to the long-tailed distribution of the training set, and further designing a method to estimate the unknown distribution based on the assumption that the confusion matrix for labeled data and the unlabeled part are almost the same. Unfortunately, their assumption does not always hold in practice since the model tends overfit the labeled samples. ABC [33] introduces an auxiliary balanced classifier to mitigate the data imbalance issue. Nevertheless, they also require the class-wise data distribution which is agnostic (due to the presence of open-set class samples) DA-GCD.
Generalized category discovery (GCD), formulated by [60], is new-raised domain in open-set learning. GCD manages to jointly recognize known categories contained in the manually annotated subset as well as novel (open-set) classes which appeared in the unlabeled set. Note that GCD classified the open-set samples into fine-grained categories according their visual concept, which is different from OOD detection [37, 24, 39]. The pioneering work GCD [60] proposes to leverage the pre-trained VIT and contrastive learning to learn a discriminative feature space and further obtain predictions by the proposed semi-k-mean, clustering strategy. ORCA [4] suggests balancing the learning rate of known and novel classes by an uncertainty adaptive margin mechanism. OpenCon [56] attempts to supervise the unlabeled instances by generating pseudo-positive pairs for closely aligned representations. But this paradígm could lead to another dilemma: the representation and pseudo-label are interdependent, which means, an inferior feature space (e.g., lack of intra-class consistency) could lead to false positive pairs, which deteriorate the lcarning of feature space in turn. A recent work SimGCD [70]. argues that parametric classification could achieve better performance with elaborately designed pseudo-labeling techniques. However, the aforementioned methods in GCD are built the class balance assumption, and we observe consistent performance degradation when generalizing to DA-GCD scenarios.
20

---

E More Empirical Results
In this Section, provide more empirical results on different datasets, and experimental settings to further verify the effectiveness of the proposed BaCon.
E.1 Results with Different Ratio of Known and Novel Classes
Here show the accuracy when changing the number of known (close-set) and novel (open-set) classes. A small [/ means fewer categories have manually labeled samples (illustrated in Figure 1), making the learning process more difficult. The performance reported in Table 9. It's revealed that BaCon achieves the best accuracies on all settings, and brings larger performance gain under more challenging settings, i.e., few known classes and large number of novel categories (outperforms best baseline ~7% overall accuracy on CIFAR- 100-LT when D 20). Note the default [3's for 5, 80, 50, and 182 respectively.
CIFAR-1O-LI, CIFAR-I00-LI) ImageNet- 100-LT, and Places-
Table 9: Test accuracy (%) on CIFAR-100-LT with different Disl-
Diel 20
Diul 50
Dicl
80
Methods
Old
New
All
Old
New
All
Old
New
All
TRSSL
65.5
30.7
37.6
62.2
30.5
46.4
58.7
35,8
54.1
ORCA!
43.5
29.3
32.1
56.6
30.2
43.4
55.0
30.8
50.1
SimGCD
51.2
27.3
32.1
66.3
19.6
42.9
59.8
24.2
52.8
GCD
63.4
50.8
53.3
67.0
53.0
60.0
65.5
49.0
62.2
OpenCon
60.3
37.2
41.8
69.0
33.3
51.2
64.2
40.9
59.6
BaCon-O
62.7
56,9
58.1
66.8
59.4
63.1
66.5
69.6
67.1
BaCon-S
64.2
59.2
60.2
67.9
60.2
64.0
67.4
66.5
67.2
21

---

E.2 Results on Few-annotated Scenarios
In Table 10, we also conduct experiments under few- anwotaedt scenarios, i.e., ri {10%, 30%, 50% The annotation ratio 77 is computed by rı #labeled samples
#all samples in known class since we only have access to manually annotated images from known categories (illustrated in Figure lc), and all novel class instances are unlabeled. Note that set 72 50% in default in the main paper.
Table 10: Test accuracy (%) with different annotation ratio rt.
F1
10%
F1
30%
F1 50%
Methods
Old
New
All
Old
New
All
Old
New
All
TRSSL
53.7
32.6
49.5
56.0
32.6
51.4
58.7
35.8
54.1
ORCA
43.8
41.3
43.3
51.2
32.2
47.4
55.0
30.8
50.1
SimGCD
51.2
17.6
44.5
55.7
25.6
49.7
59.8
24.2
52.8
GCD
61.7
54.9
60.4
63.1
58.8
62.2
65.5
49.0
62.2
OpenCon
52.1
40.9
49.9
61.8
43.9
58.2
64.2
40.9
59.6
BaCon-O
63.2
58.8
62.3
64.9
58.8
62.3
66.5
69.6
67.1
BaCon-S
64.0
60.5
63.3
65.6
65.6
65.6
67.4
66.5
67.2
22

---

E.3 Results with Different Imbalance Ratio on Labeled and Unlabeled Set
In Table we report the performance of BaCon and baseline methods in GCD when the training set has a different imbalance ratio p. Now, we further set the imbalance ratio of labeled data and unlabeled data be unequal (p: * Pu). which is common practice. For example, in medical image processing, the labeled data and unlabeled parts may come from hospitals in different countries. When comes to autonomous driving, the occurrence frequency of stones or pedestrians may slightly change when the training data is from city areas or mountain roads.
The results summarized in Table 11. We set the imbalance ratio of labeled data 100 and vary the value of P. across different settings, namely {20, 50, 100, 150}. It's observed that the proposed BaCon outperforms all baseline methods by large margin, indicating that BaCon is robust the shift of imbalance ratio between labeled and unlabeled samples.
Table 11: Test accuracy (%) CIFAR- 100 (pr 100) with different Pu.
Pu 20
Pu
50
Pu
100
Pu
150
Methods
Old
New
All
Old
New
All
Old
New
All
Old
New
All
TRSSL
63.1
38.8
58.2
57.2
44.0
54.6
58.7
35.8
54.1
55.5
39.0
52.2
ORCA
56.4
45.5
54.2
56.9
42.1
53.9
55.0
30.8
50.1
56.9
31.9
51.8
SimGCD
61.1
27.9
54.4
59.2
29.2
53.2
59.8
24.2
52.8
59.6
24.7
52.6
GCD
65.6
56.1
63.7
65.5
54.1
63.2
65.5
49.0
62.2
64.5
50.9
61.8
OpenCont
66.8
48.8
63.2
64.7
51.7
62.1
64.2
40.9
59.6
64.1
40.2
59.3
BaCon-O
67.1
63.4
66.4
67.0
61.0
65.8
66.5
69.6
67.1
65.3
54.4
63.1
BaCon-S
69.2
66.2
68.6
68.7
65.3
68.0
67.4
66.5
67.2
67.8
64.2
67.1
23


Reviews:
[{"review_id": "iDEGjK3Ocz", "reviewer": "Reviewer_JDMB", "summary": "This paper introduces and provides a formal definition for a real-world task, the long-tailed open-world classification (LT-OPC): it entails generating predictions for old and novel classes within a long-tailed open-world context. The proposed method incorporates a contrastive-learning branch and a pseudo-labeling branch to offer mutual guidance. Specifically, the contrastive-learning branch ensures reliable distribution estimates to standardize the pseudo-labeling branch's predictions. This subsequently directs contrastive learning via self-balanced knowledge transfer via a novel soft contrastive loss. The paper shows its effectiveness compared to methodologies from related fields of imbalanced semi-supervised learning and generalized category discovery. ", "questions": "1- In Table 4, the Std for ABC, DARP, and other methods are strangely high. Is there any reason for this? Also, why ORCA performs poorly in comparison to other methods? Since the results in their paper are reasonable, there might be something unfair to their method, especially in Table 4. Having 0.3 for \"few\" classes looks pretty odd compared to the highest number, 65.3. The same goes for SimGCD, which should have close results to GCD because of their high similarity.   \n\n2- There is a large gap between the paper's results and the next one in cifar100-LT, especially for new classes, and also, the model has a higher accuracy for new categories than the old ones, which seems quite strange. Is there any particular intuitive reason for this? \n\n3-  The reason behind the paper outperforming other methods with this high margin has yet to be explored and explained well. It is necessary to explain why methods behave the way the paper has reported for each dataset. Also, numbers for each dataset have drastic variances between different methods. It is worthwhile to investigate and explain the reasons for these differences so that they become more comprehensible.\n\n4- While it is interesting to consider common unseen categories, as in Figure 1, however, the real-world data usually has the long-tailed categories as unseen. With this new way of considering the open world, a model that assigns most novel samples to the most common novel category or, in other words, detects the open world samples and then assigns them to the most common category will triumph over a model which considers the unknown categories all in long-tailed. For instance, ORCA has this uniform objective loss, which is in direct contrast with the way data has been assigned (maybe this is the reason for ORCA's poor performance in the tables), while for the real world, the more traditional long-tailed split is in fact more plausible. Although Data preparation is explained in the paper, is it similar to how Figure 1 has depicted? Are there any experiments to also compare methods against traditional long-tailed versions of datasets?\nAlso, a base method that only detects open-world samples and assigns them to the most common novel categories might help to decipher how much of this assignment is just because of the model's knowledge of common novel categories. \n", "limitations": "The limitations of the work have been adequately addressed. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Clarity: This paper is well-structured and clearly presents the motivation and method.  \n\n-  Significance: The problem of long-tailed open-world classification is a significant real-world problem that this paper strives to define and address. \n\n- Originality: The paper introduces an intriguing concept, \"soft contrastive learning,\" which makes it possible to benefit from pseudolabeling and soft labels for contrastive learning. The pseudolabeling branch provides the contrastive branch with classification signals, while the contrastive branch standardizes the distribution of pseudolabeling branch.\n\n- Quality: The supplementary material and experiments offer a detailed analysis, enhancing the paper's value. ", "weaknesses": "- It is not a weakness, but since the OLTR work already exists, it is better to use a more distinct name for the problem the paper addresses. Especially since this problem is more in line with generalized category discovery, a different name for the problem seems less confusing.  \n- Ablations studies could be more well explored, and the explanation of the trends seen in Table 6 can be improved and explain these trends with more depth. \n- Calling theorem 1 a theorem does not seem mathematically correct. It is solving a constrained optimization problem specific to the method.  \n- Also, in the proof of theorem 1 in supplemental, is it correct to omit $C$ for a more rigorous consideration because it depends on $w_{ij}$, which depends on $z_i$ and $z_j$s?\nResults for other methods are by applying code of other methods to the new datasets. Since methods like ORCA on cell dataset, GCD and SimGCD on Herbarium have a decent performance. Since Herbarium or cell datasets are long-tailed, comparing these methods on the same datasets and report numbers from their paper seems a more fair comparison. \n- Although the number of experiments is enough, the numbers have not been explained and reasoned about. Since numbers for different methods have drastic changes, a section to analyze these numbers (at least in supplemental) seems necessary. \n- Minor: there are a few mistakes in the tables: (Table 4/known/Many/GCD: 75.9) (Table4/known/Med/SimGCD: 72.8) (Table5/CIFAR10-LT $\\rho =150$/ OpenCon: 83.2) (Table5/CIFAR100-LT$\\rho =150$/ BaCon-O: 66.8) should be bold \n\nAlso, indicating the second-best method or improvement can be beneficial for the reader to see the efficacy of the paper’s contributions.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> A different name for the problem seems less confusing.\n\nThanks for the advice! We will change the name to 'Imbalanced Open-World Classification'.\n\n> Ablations studies could be more well explored.\n\nThanks for the advice! We will add a more detailed analysis in the ablation. \n\n> Calling theorem 1 a theorem does not correct.\n\nThanks for the advice! We will change 'Theorem 1' to 'Claim 1'.\n\n> Is the proof of theorem 1 correct to omit C? \n\nThank you for your reminder! We will modify the $C$ in Eq. 2 (Appendix) to $C_i$, as $C$ may vary among different samples. But we believe that this should not affect subsequent proofs, because, for a training batch, its *positiveness* score matrix is obtained by calculating the similarity of the rectified class probability distribution between each sample pair $\\boldsymbol{x}_i$ and $\\boldsymbol{x}_j$ (line 190- line 192), so each element is a fixed value that does not affect the subsequent differentiation process and can be regarded as a constant. \n\n> Results on Herbarium or Single-Cell.\n\nThanks for the advice! We compare the proposed BaCon with other methods on Herbarium-19 in Table F in the rebuttal pdf. Our method outperforms baselines on open-set ('New' split) classes and overall accuracy ('All' split) by a large margin.  The authors of ORCA didn't provide the source code for training methods on the Single-Cell dataset, which is a dataset in the field of biology and may need some unique preprocessing or different hyper-parameters. We have emailed the authors to ask for the source code, and we will conduct the experiments on Single-Cell when we get the code from the authors.\n\n> The results have not been fully explained.\n\nThanks for the advice! We will add the explanation below:\n\nIn Table A (in the rebuttal PDF), we compared the proposed BaCon with two SOTA imbalance-SSL methods, ABC and DARP, as well as the six latest open-world recognition methods. Among them, GCD and OpenCon are contrastive-based methods, while ORCA, SimGCD, OpenLDN, and TRSSL are classifier-based methods.  **For the imbalanced-SSL methods,** thanks to their carefully designed semi-supervised learning approaches for long-tail data, they achieve decent performance on known (close-set) classes. However, they perform poorly on novel (open-set) classes because the imbalance-SSL scenario does not consider the presence of open-set samples in the unlabeled data.  **Regarding the open-world recognition methods,** we found that contrastive-based methods outperform classifier-based methods in the LT-OPC scenario. Reasonable explanations can be found in [1], which indicate that self-supervised contrastive training, is more robust to class imbalance than supervised methods. Nevertheless, existing contrastive-based methods struggle to optimize the feature space of unlabeled samples, e.g., GCD only uses a self-supervised contrastive loss for unlabeled data. Moreover, GCD methods lack tailored designs for imbalanced datasets, leading to significant performance degradation when the training set has a long-tailed distribution (as shown in Table 2 in the main text).  **For the proposed BaCon,** we design a dual branch structure that works collaboratively to provide interactive supervision and achieve self-rebalancing. The pseudo-labeling branch is enhanced by the proposed 'dynamic distribution estimation' algorithm for regularizing the predictions, while the contrastive-learning branch takes advantage of the 'self-balanced knowledge transfer' module, which helps to learn a balanced and reasonable feature space. It outperforms baseline methods by tackling the imbalanced recognition task and the open-world challenge at the same time.\n\n> Minors in tables.\n\nWe would like to thank the reviewer for a very detailed review. We will fix the mistakes in the experiment tables. \n\n> There might be something unfair in experiments.\n\nPlease kindly refer to 'The Fairness of the Experiments' in the 'General Response' column at the top.\n\n> In Table 4, the Std for some methods are strangely high.\n\nAs you mentioned, it's observed that classifier-based methods have a relatively large Std than contrastive-based methods (GCD and OpenCon). The phenomenon is aligned with [1], which suggests that self-supervised contrastive training, is more robust to class imbalance than supervised methods. We will add the explanations in the final version.\n\n> Although Data preparation is explained in the paper, is it similar to how Figure 1 has depicted? Are there any experiments to compare methods against traditional long-tailed versions of datasets?\n\nIn our paper, we assume that both close-set and open-set categories follow a long-tailed distribution, meaning that both parts of the data contain both common and rare classes. For example, in Figure 1 (a), we assume classes 1-5 are close-set categories and 6-10 are open-set categories, with roughly the same distribution for the two parts. The core idea of setting the problem in this way is that when manually collecting data, we can consider it as randomly sampling some categories from a large (including plenty of classes) real-world long-tailed distribution as the close-set. Similarly, the open-set categories can also be considered as a sampling process in the large long-tailed distribution. For instance, we consider training a species classification model where the close-set data we collect covers species in the forest, while open-set classes can be species on plains or marine organisms. In this case, both closed-set and open-set samples are long-tailed.\n\nIn the experiment, we compared BaCon with the semi-supervised long-tail learning methods ABC and DARP. On the other, due to we can't obtain a prior distribution of the training set in LT-OPC, we are unable to compare the proposed method with supervised long-tail methods.\n\n\n\n[1] Hong Liu, et al. Self-supervised learning is more robust to dataset imbalance. ICLR, 2021."}, {"review_id": "aZabyVibf7", "reviewer": "Reviewer_kmyS", "summary": "This paper studies the long-tailed recognition in the presence of open-set samples, which is never studied by previous works. Moreover, existing long-tailed learning methods can not be directly extended to open-set classification. To solve this problem, the authors design a new method termed BaCon, which utilizes contrastive learning to help estimate the label distribution and generate pseudo-labels for the unlabeled data, thus relieving the data imbalance and providing more data supervision. Besides, a pseudo-label-based contrastive loss is proposed to cluster similar samples for better open-set classification. The authors conduct experiments on multiple long-tailed datasets including CIFAR10/100-LT, ImageNet100-LT, and Places-LT. The results show that BaCon can obviously improve performance, especially when the imbalance ratio is high.", "questions": "See \"Weaknesses\".", "limitations": "The authors have discussed the limitations in the main paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper firstly studies the long-tailed recognition with unlabeled data in both open-set and close-set samples. This studied problem is more complex and the performances of most previous methods are not satisfactory, indicating the necessity of proposing a corresponding method.\n\n2. The authors propose a new method for long-tailed open-set classification. The experimental results demonstrate the superiority of the proposed method compared to most existing methods.\n\n3. The authors conduct several ablation studies to verify the effectiveness of each component.", "weaknesses": "1. The studied problem is novel. However, the motivation is insufficient. By simply integrating open-set classification methods with re-balancing strategies, the long-tailed problem may be alleviated. The authors should also compare with the simply integrated baseline methods, or explain the difficulty of the integration.\n\n2. The pseudo-labeling module is important since it can affect the additional supervision and the pseudo-label-based contrastive loss. So how to ensure the quality of the pseudo-label? What if the pseudo-label is incorrectly assigned? To verify the quality of the pseudo-label, I suggest the authors calculate the accuracy of the pseudo-labels for both head and tail classes and report the results.\n\n3. Some typos. The title of Section 5 should be \"Experiments\". In Table 4, GCD performs better in Many and Med shots, but the results are not bolded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The studied problem is novel. However, the motivation is insufficient. By simply integrating open-set classification methods with re-balancing strategies, the long-tailed problem may be alleviated. The authors should also compare with the simply integrated baseline methods, or explain the difficulty of the integration.\n\nThanks for the advice! We would like to claim that most supervised or semi-supervised long-tail learning methods [1, 2, 3] require the prior training set (long-tailed) distribution to tackle the data imbalance issue, and it prevents them from being directly extended to resolve the proposed LT-OPC task, since the prior dataset distribution is unavailable in LT-OPC (line 56 - line 58). \n\nMeanwhile, though several self-supervised methods [4, 5] could alleviate the imbalance issue without knowing the dataset distribution, they discard all the accessible label information and bring marginal accuracy gain. As shown in Table C (please kindly refer to the rebuttal PDF file), we combine two best-performing baseline methods GCD and SimGCD with  BCL [5], which is one of the latest self-supervised long-tail learning methods. It's observed that BCL can not effectively tackle the data imbalance issue, while the proposed BaCon improves both accuracy and balancedness by a large margin via dynamic distribution estimation (Section 4.1) and the self-balanced knowledge transfer module (Section 4.2).\n\n> The pseudo-labeling module is important since it can affect the additional supervision and the pseudo-label-based contrastive loss. So how to ensure the quality of the pseudo-label? What if the pseudo-label is incorrectly assigned? To verify the quality of the pseudo-label, I suggest the authors calculate the accuracy of the pseudo-labels for both head and tail classes and report the results.\n\nThanks for the advice!  We report the test accuracy of the pseudo-labeling branch during training in Table D (please kindly refer to the rebuttal PDF file). Furthermore, we have two elaborately designed components in our method to ensure the quality of the pseudo-label-based contrastive loss. Firstly, to mitigate the impact of the long-tail distribution on pseudo-labels, we designed a debiasing module to post-hoc adjust the logits output from the pseudo-labeling branch (introduced in Section 4.2). Secondly, when calculating the *positiveness* coefficient $w_{ij}$ using the rectified pseudo-labels, we utilized the soft labels of each sample instead of hard labels (i.e., converting labels to one-hot labels based on the class with the highest logits). This design helps alleviate the influence of erroneous pseudo-labels. Empirical evidence supporting the effectiveness of these two components can be found in Table 6 (b) and (c).\n\n> Some typos. The title of Section 5 should be \"Experiments\". In Table 4, GCD performs better in Many and Med shots, but the results are not bolded.\n\nWe would like to thank the reviewer for a very detailed review! We will fix the typos.\n\n[1] Menon, Aditya Krishna, et al. \"Long-tail learning via logit adjustment.\" ICLR 2021.\n\n[2] Kim, et al. \"Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning.\" NIPS 2020.\n\n[3] Lee, et al. \"ABC: Auxiliary balanced classifier for class-imbalanced semi-supervised learning.\" NIPS 2021.\n\n[4] Jiang, Ziyu, et al. \"Self-damaging contrastive learning.\" ICML, 2021.\n\n[5] Zhou, Zhihan, et al. \"Contrastive Learning with Boosted Memorization.\" ICML, 2022."}, {"review_id": "8PNS0Ydi6D", "reviewer": "Reviewer_udGd", "summary": "This paper tackles long-tailed open-world classification problem. It handles data imbalance in the long-tailed problem, and it also needs to handle the closed-world/open-world classifications.  For open-world classifications, there are only unlabeled data. Moreover, it assumes that unlabeled data can come from the \"closed-world\"classes as well.\n\nIt proposes a framework, called \"Self-Balanced Co-Advice contrastive framework (BaCon)\", to tackle data imbalance and handle the open-set classifications. The framework consists of a contrastive-learning branch and a pseudo-labeling branch, and the two branches also work collaboratively. Experiments are performed on image classification datasets.", "questions": "The experiments are done on CIFAR, ImageNet, and Places datasets. Are there any experiments on other fine-grained long-tailed datasets?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed problem of \"long-tailed open-world classification\" is valid. Both \"long-tailed\" and \"open-world\" are popular and important problems in vision.  This paper combines problems and approaches from a couple of previous works. The proposed approaches may work.", "weaknesses": "The proposed problem setting is not very novel. For example, the previous work of \"Large-scale long-tailed recognition in an open world\" (reference [36]) handles long-tailed problems in an open world. Also the paper \"Generalized Category Discovery\" (reference [52]) proposes \"the unlabelled images may come from labelled classes or from novel ones.\"\n\nThe proposed approach is not novel. For example, the contrastive learning idea has been used in the \"Generalized Category Discovery\" (reference [52]).  Or the key insights need to be clearly stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The proposed problem setting is not very novel. For example, the previous work of \"Large-scale long-tailed recognition in an open world\" (reference [36]) handles long-tailed problems in an open world. Also the paper \"Generalized Category Discovery\" (reference [52]) proposes \"the unlabelled images may come from labelled classes or from novel ones.\"\n\nGCD [1] and OLTR [2], are indeed related to the proposed LT-OPC setting, but we would like to clarify that the proposed problem setting still has significant differences from them and introduces unique challenges in this scenario. **Compared to GCD,** they only focus on the problem of the open world but ignore the data imbalance issue in the real world, i.e., their method falls in the lower right corner of Figure 1 (b); while our setting aims to solve both the problems of the open-world and data imbalance, corresponding to the upper right corner of Figure 1 (b). Moreover, since we do not know the prior distribution of the dataset in open-world scenarios, we cannot simply transfer existing long-tail learning methods to solve the problem of data imbalance in LT-OPC scenarios, making it more challenging. **Compared to OLTR,** though we both study the long-tail problem in open-world scenarios, our method pushes 'one step further'. That is, not only learn a model to detect the open-set samples, but it also requires the model to classify them according to their semantics. On the other, OLTR considers a fully-supervised scenario that only utilizes manually annotated samples (lower left in Figure 1 (c)); while we also make use of unlabeled (close-set and open-set) data (upper right and lower right in Figure 1 (c)), which makes our approach more powerful and generalizable.\n\n\n\n> The proposed approach is not novel. For example, the contrastive learning idea has been used in the \"Generalized Category Discovery\" (reference [52]). Or the key insights need to be clearly stated.\n\nOur **key insight** is methods like GCD [1] lack effective supervision of novel categories, since they only perform self-supervised contrastive loss on samples in novel classes, which leads to inferior performance. Meanwhile, GCD overlooks the long-tail distribution in the real world. '***How to simultaneously tackle the issue of data imbalance and provide adequate supervision for open-set classes***' motivates our proposed method BaCon, where the **key technical novelty** is the design of a dual branch structure and the co-advice mechanism which lets the two branches work collaboratively to provide interactive supervision to tackle the imbalanced recognition task under open-world scenarios. To be specific, the contrastive-learning branch provides distribution estimation to regularize the prediction results of the linear classifier for better pseudo-labeling. On the other, the generated pseudo-labels are sampled and debiased to re-balance and provide additional supervision for the contrastive-learning branch. To fuse the knowledge of the two branches and learn a better feature space, we design a novel pseudo-label-based contrastive loss that clusters samples based on their *positiveness* scores.\n\nThanks for the advice! We will highlight our key insight and key technical novelty in the final version.\n\n> The experiments are done on CIFAR, ImageNet, and Places datasets. Are there any experiments on other fine-grained long-tailed datasets?\n\nIn addition to conducting experiments on CIFAR-10-LT, CIFAR-100-LT, ImageNet-100-LT, and Places-365-LT as described in the Experiments section of the main text, we also present the experimental results of our method on ImageNet-1k-LT and iNaturalist [3] in Appendix E.5, and the iNaturalist dataset is highly imbalanced which is consists of 8,142 categories. Furthermore, we also compare the performance of our method with other baseline methods on the Herbarium-19 [4] dataset in Table F (please kindly refer to the rebuttal PDF file). We believe that these experiments and comparisons serve to demonstrate the effectiveness of our method.\n\n\n\n[1] Sagar Vaze, et al. Generalized category discovery. CVPR, 2022.\n\n[2] Ziwei Liu, et al. Large-scale long-tailed recognition in an open world. CVPR, 2019.\n\n[3] G. Van Horn, et al. The inaturalist species classification and detection dataset. CVPR, 2018.\n\n[4] Kiat Chuan Tan, et al. The herbarium challenge 2019 dataset. In Workshop on Fine-Grained Visual Categorization, 2019."}, {"review_id": "WJ0oXbiGSu", "reviewer": "Reviewer_B9Lc", "summary": "The paper proposes a combination of contrastive learning and semi-supervised learning approach to label images in open-world classification. The classes are assumed to be long-tailed, as is the case in real-world cases. Not all the classes have a labeled example, but the number of classes are assumed to be known beforehand. \n\nThe proposed method follows a \"two branch structure\", which one branch doing contrastive learning, and another doing pseudo-labeling.  ", "questions": "1. Line 146: What do you mean sample number n? \n2. Line 146: What does the symbol (n^c) mean? Are you doing \"n choose c\" or \"n raised to c\"? \n3. How is Hungarian optimal assignment used in the context of the problem? How do you define \"cost\" in the adjacency matrix? The citation provided does not explain this. \n4. What is the \"align\" operation in equation 2? \n5. Why does equation 4 lead to a \"post-hoc logits adjustment\"? It is better to assume the reader has not read citation 39 and explain the concept briefly\n6. How is equation 5 leading to \"we prioritize selecting samples with higher prediction confidence in each class\"? Line 179.\n7. Line 183. How do you mitigate imbalance issue with this sample process? Even the ground truth labels are highly skewed.\n8. What is A(i) in equation 1 and 6?\n", "limitations": "Good paper overall. I would like to see a clearer explanation of methods as listed above.\nThe experiments where the number of classes are unknown are perhaps of higher importance, but they have been pushed to Appendix. It would be good to include a summary in the main paper. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A highly relevant problem statement for real-world application of image classification\n2. Thorough comparison against related works, both in conceptual differences and in evaluation\n3. Soft contrastive loss is an interesting idea\n", "weaknesses": "Some of the algorithmic steps are not clearly described. Questions listed below.\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Line 146: What do you mean sample number n?\n\nThe sample number **n** is a vector with size $C$, where the $i$-th element represents the number of samples in $i$-th cluster (line 144- line 145).\n\n> Line 146: What does the symbol (n^c) mean? Are you doing \"n choose c\" or \"n raised to c\"?\n\n$n^c$ refers to the $c$-th element in vector $\\boldsymbol{n}$, which is the number of samples in the $c$-th cluster. We realize that there may be ambiguity here, so we will change $n^c$ to $n[c]$ in the final version. Thank you for your reminder.\n\n> How is Hungarian optimal assignment used in the context of the problem? How do you define \"cost\" in the adjacency matrix? The citation provided does not explain this.\n\nAssuming we want to calculate the accuracy of a classification task with a number of categories $C$, the cost matrix $W \\in \\mathbb{R} ^ {C \\cdot C}$. In this case, $-w_{ij}$ is the number of samples with the highest logits value in the $i$-th dimension of the model classifier (i.e., predict to dimension $i$), and the true label should be the $j$-th class of samples. We take the minus sign because the Hungarian optimal assignment algorithm will **minimize** the cost, and we want to find an optimal assignment that **maximizes** the test accuracy. After defining such a cost matrix, we can use the Hungarian algorithm to obtain the optimal assignment, under which the model has the maximum test accuracy. We will add the explanation in the appendix in the final version.\n\n> What is the \"align\" operation in equation 2?\n\nThe 'align' operation means finding the correspondence between the clusters and categories using the method described from line 146 to line 149. In a word, we first map the 'known' classes via the Hungarian algorithm performed on labeled samples, then we sort the remaining clusters by the cluster size and assign them sequentially to the novel classes.\n\n> Why does equation 4 lead to a \"post-hoc logits adjustment\"? It is better to assume the reader has not read citation 39 and explain the concept briefly\n\nThanks for the advice! Equation 4 implements the Logits Adjustment [1] method, which is one of the SOTA methods in supervised long-tail learning, its core idea is to adjust the output logits at test time to mitigate the issue of data distribution differences between testing (balanced) and training (long-tailed). In other words, the adjustment term $-k \\cdot \\pi_e$ introduces a label-relevant offset, which amplifies the tail class probability according to the data distribution during training, leading to an unbiased probability estimation. We will add a brief explanation to Section 4.2.\n\n> How is equation 5 leading to \"we prioritize selecting samples with higher prediction confidence in each class\"? Line 179.\n\nIn fact, we only use Equation 5 to determine the number of samples that should be sampled for each category. After determining the number of samples, we then rank each category of samples in descending order of confidence and sequentially select the sample with the highest confidence according to the number of samples needed.\n\n> Line 183. How do you mitigate imbalance issue with this sample process? Even the ground truth labels are highly skewed.\n\nThe long-tailed distribution of the dataset can indeed bring difficulties to re-balancing. However, since the sampling rate we set is inversely proportional to the number of samples in that category, we can assume that the number of samples sampled in each category is roughly the same, which can still alleviate the problem of long-tailed distribution, i.e., the data for contrastive learning has a smaller imbalance ratio after incorporating the sampled images.\n\n> What is A(i) in equation 1 and 6?\n\n$A(i)$ is the set of indices of all other features (both positive and negative pairs) in contrastive learning. Take SimCLR [2] for instance, assume we train the model with a batch size B, and all samples are augmented twice in different augmentations ($2B$ views), then feed into the model to obtain $2B$ features $\\left\\\\{z_i\\right\\\\}_{i=1}^{2B}$. For feature $z_i$, $A(i)$ refers to the set of indices of all other $(2B-1)$ features in the training batch. We will add the clarification in the main text.\n\n> The experiments where the number of classes are unknown are perhaps of higher importance, but they have been pushed to Appendix. It would be good to include a summary in the main paper.\n\nThanks for the advice! We will add a summary of the experiments in the Appendix in Section 5.1.\n\n\n\n[1] Menon, Aditya Krishna, et al. \"Long-tail learning via logit adjustment.\" ICLR, 2021.\n\n[2] Ting Chen, et al. A Simple Framework for Contrastive Learning of Visual Representations. ICML, 2020."}, {"review_id": "btLpJTq8n5", "reviewer": "Reviewer_7Jpa", "summary": "This paper presents a novel real-world driven problem setting named long-tailed open-world classification (LT-OPC) setting, where a model should predict both closed-set and open-set classes with long-tailed distribution. To address such challenges, a Self-Balanced Co-Advice contrastive framework (BaCon) is proposed, where both contrastive learning branch and pseudo-labeling branch provide supervision for the LT-OPC task. First, the contrastive learning branch estimates the training distribution of a data to regularize the pseudo-labeling branch. In the pseudo-labeling phase, a debiasing and sampling phase is designed to resolve the data imbalance and insufficient supervision for the novel classes. In final, pseudo-label branch further provides supervision to the contrastive learning branch, in the form of  pseudo-label-based soft contrastive learning objective. A series of experiments are carried out under the proposed setting and the effectiveness of each of the proposed component is studied. ", "questions": "- The notation of $\\mathcal{Y}_{n}$ in line 113 is used without definition.\n\n- What exact type of loss function is utilized for the pseudo-labeling loss $\\mathcal{L}_{u}$ from the equation 3?\n\n- It can be found that the codes are attached, but it is unclear the codes can be executable. If you decided to include the codes in the submission, making the codes more understandable would be appreciated.", "limitations": "Brief limitation of the work can be found, but potential negative societal impact has not been addressed.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed LT-OPC setting with long-tailed distribution data considers a more practical scenario in real world, compared to the traditional semi-supervised learning and generalized category discovery setting.\n\n- It seems that the contrastive learning strategy and pseudo-labeling strategy with interaction between them are well designed to deal with the proposed LT-OPC setting. The ablation results from the Table 6 can support its design.\n\n- Experimentally, the proposed BaCon framework shows competitive results from the baseline methods in imbalanced SSL and generalized category discovery setting.", "weaknesses": "- It is unclear that the proposed problem setting of LT-OPC is novel. There have been some published works that tried to combine semi-supervised learning and generalized category discovery setting in the context of open-world semi-supervised learning [C1, C2], but any discussions or comparisons with them are not provided in this paper. To my view, the only difference is whether the training data has long-tailed distributions or not, which can weaken the task novelty of this work.\n\n- It is unclear that the comparisons with imbalanced SSL methods are fair. To my understanding, the model trained with the proposed BaCon objective gets supervision for the novel classes from the unlabeled data with the pseudo-labeling term, while the baseline SSL methods do not. In that sense, it is natural that the proposed method shows higher accuracy in ‘New’ category classes. Any clarification or opinion from the authors is expected. In addition, for the imbalanced SSL baselines, both DARP and ABC do not contain contrastive branch. To make the comparison more fair, considering additional SSL baselines including contrastive learning branch is necessary [C3, C4].\n\n- For the training the baseline methods, is the training setup identical to the procedure in the proposed BaCon method? Training BaCon model relies on transformer architecture with strong pretraining method of DINO. It is unclear that the training protocol for the other baseline methods is the same with BaCon for fair comparison.\n\n- Due to combining two different mechanisms, which are pseudo-labeling and contrastive learning, many hyper-parameters are introduced. How those values can be determined in a new dataset without simply tuning the real test set of each benchmark? Considering a real-world scenario, the prior for the novel class categories can be prohibited. To simplify the question, can a model tuned with a dataset including both known classes and novel classes (e.g., validation data) generalize to a new dataset including 'unseen' novel classes (e.g., test data)? Detailed discussion on such assumption would be appreceated.\n\n---\n\nReferences\n\n[C1] Rizve, Mamshad Nayeem, et al. \"Openldn: Learning to discover novel classes for open-world semi-supervised learning.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2022.\n\n[C2] Rizve, Mamshad Nayeem, Navid Kardan, and Mubarak Shah. \"Towards realistic semi-supervised learning.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2022.\n\n[C3] Park, Jongjin, et al. \"Opencos: Contrastive semi-supervised learning for handling open-set unlabeled data.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2022.\n\n[C4] Zheng, Mingkai, et al. \"Simmatch: Semi-supervised learning with similarity matching.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Novelty of LT-OPC. \n\nPlease kindly refer to 'The Novelty of the Proposed Setting' in the 'General Response' column at the top.\n\n> Discussions or comparisons with [C1, C2] are not provided.\n\nThank you for the reminder! We have migrated the two methods [C1, C2] you mentioned to the proposed LT-OPC scenario, and we will include them as baseline methods in the final version. The results are shown in the rebuttal PDF file as Table A and Table B. It is worth noting that for a fair comparison, we have replaced the backbone model of both methods with the more powerful DINO pre-trained ViT, which is consistent with the backbone model of our method.  Moreover, we will provide introductions to these two methods in the 'Related Works' section. \n\nIn addition, the two methods share similarities with the GCD scenario, which involves the consideration of open-set class samples in unlabeled data within a semi-supervised setting. During testing, the model is required to classify both close-set and open-set samples. In Table 1 and lines 28 to line 44 of the main text, we provide a detailed comparison between the proposed LT-OPC and relevant scenarios such as GCD. We will incorporate the comparisons with these two methods in the respective location.\n\n> Model trained with BaCon gets supervision for the novel classes from the pseudo-labeling term, while the baseline SSL methods do not.\n\nFirstly, we would like to clarify that the dataset used for training the baseline SSL methods and our method are exactly the same, as shown in Figure 1(a) in the main text. The training set consists of labeled close-set samples, and unlabeled close-set and open-set samples.\n\nSecondly, we would like to claim that **the baseline SSL methods also utilize ‘the pseudo-labeling term’ during training** to obtain supervision for the unlabeled samples, which is the same as our method. Specifically, their methods first generate pseudo-labels for each sample, then filter out pseudo-labels with confidence scores lower than a pre-set threshold. The remaining pseudo-labels are then used as supervision for training. On the other hand, our method trains a pseudo-labeling branch to generate pseudo-labels, which in turn guides contrastive learning. In this process, **no additional supervision**, such as the true labels of the 'new' categories samples, is used. Therefore, we assume that our comparison with the baseline SSL methods is fair.\n\n> Considering additional SSL baselines including contrastive learning branch is necessary [C3, C4].\n\nThank you for your suggestion! We conducted experiments on the two methods, Opencos [C3] and Simmatch [C4], and reported the results in Table E. To ensure fairness, we re-implemented both methods using DINO pre-trained ViT as the backbone model. We will include the results in the final version. \n\n> It is unclear that the training protocol for the baseline methods is the same with BaCon.\n\nPlease kindly refer to 'The Fairness of the Experiments' in the 'General Response' column at the top.\n\n> How hyper-parameters can be determined in a new dataset without simply tuning the real test set?\n\nTo verify whether our method can be extended to 'unseen' novel classes, we conducted experiments on ImageNet. Specifically, we first randomly sample 50 categories as 'known' classes and keep these 50 classes consistent across all experiments. Then, we **non-repetitively** sample 6x50 categories from the remaining 950 categories, resulting in six groups of classes named 'novel_val' and 'novel_test_A/B/C/D/E'. Next, we select the optimal hyperparameters based on the performance on the {'known' + 'novel_val'} dataset. **By Keeping all the hyperparameters fixed**, we further train and evaluate the model on the five different test datasets.  The experimental results shown in Table G, revealed that the hyperparameters performing best on the validation set achieved similar performance on different 'unseen' novel classes. This indicates that our method exhibits good generalization, implying its effectiveness in handling 'unseen' novel classes. We will report the experiment results in the final version.\n\n> y_n in line 113 is used without definition.\n\nWe would like to thank the reviewer for a very detailed review. $\\mathcal{Y}_n$ represents the set of labels for novel classes, which corresponds to the total number of categories in the open-set data. We will include this clarification in Section 3.1.\n\n> What exact type of loss function is utilized for L_u from Eq. 3?\n\nIn Equation 3, the term $\\mathcal{L}_u$ can be any unsupervised loss function used in classifier-based GCD methods. In our experiments, we choose two different $\\mathcal{L}_u$ to demonstrate the effectiveness of our method. These are the cross-pseudo supervision loss from SimGCD (BaCon-S) and the pairwise objective from ORCA (BaCon-O).  Specifically, the cross-pseudo supervision loss is a loss function similar to the idea of self-distillation and SwAV [1]. It uses the predictions of two views (images augmented differently) of the same image as pseudo-labels for each other and calculates the cross-entropy loss. On the other hand, the pairwise objective in ORCA encourages intra-class consistency by pulling together samples with similar representations.  From the experimental results in Section 5, it's observed that our method can adapt to different $\\mathcal{L}_u$ and achieve significant improvements, demonstrating the effectiveness and versatility of the proposed approach.\n\n> It is unclear the codes can be executable.\n\nWe apologize for the inconvenience. Some of the code was not successfully synchronized from my GitHub repository to the anonymous repository. We have now resolved this issue and have attached a readme file as an explanation. Thanks for the reminder!\n\n[1] Caron M, et al. Unsupervised learning of visual features by contrasting cluster assignments. NIPS, 2020."}]

Author rebuttal:
# General Response

## To All Reviewers.

Dear Reviewers:

We would like to thank you for your time and insightful comments! We have carefully read your review comments and conducted additional experiments as required to answer the questions (please kindly refer to the rebuttal PDF file and the rebuttal reply to each reviewer below). We hope we have addressed your concerns. We would be grateful if you would kindly let us know of any other concerns and if we could further assist in clarifying any other issues.

Thanks a lot again, and with sincerest best wishes

Authors

## Explanation of the Rebuttal PDF.

In the rebuttal PDF file, we indicate the best performance among **all** methods with **bold numbers** in all tables (consistent with the submitted version), and we will use underlined numbers to represent the best performance among the **baseline** methods. We believe this will help highlight the differences between our method and the baseline methods.

## The Novelty of the Proposed Setting.

GCD [1] and OLTR [2], are closely related to the proposed LT-OPC setting, but we would like to clarify that the proposed problem setting still has significant differences from them and introduces unique challenges in this scenario. **Compared to GCD,** they only focus on the problem of the open world but ignore the data imbalance issue in the real world, i.e., their method falls in the lower right corner of Figure 1 (b); while our setting aims to solve both the problems of the open-world and data imbalance, corresponding to the upper right corner of Figure 1 (b). Moreover, since we do not know the prior distribution of the dataset in open-world scenarios, we cannot simply transfer existing long-tail learning methods to solve the problem of data imbalance in LT-OPC scenarios, making it more challenging. **Compared to OLTR,** though we both study the long-tail problem in open-world scenarios, our method pushes 'one step further'. That is, not only learn a model to detect the open-set samples, but it also requires the model to classify them according to their semantics. On the other, OLTR considers a fully-supervised scenario that only utilizes manually annotated samples (lower left in Figure 1 (c)); while we also make use of unlabeled (close-set and open-set) data (upper right and lower right in Figure 1 (c)), which makes our approach more powerful and generalizable.

The experimental results empirically validate the highly challenging nature of the proposed setting. GCD methods as well as long-tailed SSL methods, all exhibit significant performance drops under the newly proposed setting, as shown in Table 2 in the main text, and Table A, Table B in the rebuttal PDF file.

Additionally, another significance of our work is that existing research on long-tailed learning and open-set learning has developed independently, and very few previous works attempt to combine both of these areas. However, data imbalance and open-ended distribution are inherently intertwined with each other in the real visual world, which renders existing methods ineffective in terms of deployment. The proposed LT-OPC addresses both of these problems simultaneously, which can facilitate the deployment of existing methods in practical application scenarios.

## The Fairness of the Experiments.

In the submitted version, we have kept the original settings of the baseline methods. Namely, ABC, DARP, Opencon, and ORCA, which use the ResNet backbone model as they implement in their papers. On the other hand, GCD, SimGCD, and our method utilize the DINO pre-trained ViT network, which may lead to an unfair comparison. To address this issue, **we have replaced the backbone networks of all baseline methods with the DINO pre-trained ViT, consistent with the proposed method**. Furthermore, we have ensured that all methods are trained for 200 epochs, with an equal amount of data used per epoch. Table A and Table B in the rebuttal PDF file present the modified versions of Table 3 and Table 4 in the main text, respectively, where $\dagger$ denotes adapted methods. It can be observed that the baseline methods benefit from the powerful feature extraction capability of the DINO pre-trained ViT, achieving higher accuracy than using ResNet as the backbone on all datasets. At the same time, our method still demonstrates significant performance improvement.

In the final version, we will replace the original tables and report the experimental results of all methods when using the DINO pre-trained ViT as the backbone.



[1] Sagar Vaze, et al. Generalized Category Discovery. CVPR, 2022.

[2] Ziwei Liu, et al. Large-scale long-tailed recognition in an open world. CVPR, 2019.

Meta-review: This was a slightly positive “borderline accept” paper. Initially, there were common concerns on the novelty of the proposed setting and the proposed method. The submitted paper was not presented well in highlighting the difference, key challenge, and key component of the proposed setting and method. However, after a deep multi-round discussions between authors and Reviewers JDMB (from 6 -> 7) and 7Jpa (from 3 -> 6), the contributions of the paper have been made much clearer. Yet it is unfortunate that Reviewer udGd (4: Borderline reject; Confidence: 3), who also had critical concerns about the novelty of problem setting and proposed method, did not come back after authors’ rebuttal. However, from authors’ feedback and the above-mentioned discussions, AC regarded that these critical concerns have been addressed. There were a number of other concerns, such as experimental comparisons, which were also addressed by authors. Finally, except Reviewer udGd who held the only negative rating but did not come back, all other review ratings were positive, recommending acceptance of the paper. AC therefore is happy to accept the paper. Authors are required to address reviewer comments and incorporate the rebuttal and discussion material to the camera-ready version of the paper.
Decision: Accept (poster)