OwMatch: Conditional Self-Labeling with Consistency for Open-World Semi-Supervised Learning
Authors: Shengjie Niu
Lifan Lin
Jian Huang
Chao Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Open-world Semi-Supervised Learning
self-labeling
consistency loss
Primary area: machine_vision
TL;DR: Boosting open-world semi-supervised learning with conditional self-labeling and open-world hierarchical thresholding

Abstract: Semi-supervised learning (SSL) offers a robust framework for harnessing the potential of unannotated data. Traditionally, SSL mandates that all classes possess labeled instances. However, the emergence of open-world SSL (OwSSL) introduces a more practical challenge, wherein unlabeled data may encompass samples from unseen classes. This scenario leads to misclassification of unseen classes as known ones, consequently undermining classification accuracy. To overcome this challenge, this study revisits two methodologies from self-supervised and semi-supervised learning, self-labeling and consistency, tailoring them to address the OwSSL problem. Specifically, we propose an effective framework called _OwMatch_, combining conditional self-labeling and open-world hierarchical thresholding.  Theoretically, we analyze the estimation of class distribution on unlabeled data through rigorous statistical analysis, thus demonstrating that OwMatch can ensure the unbiasedness of the label assignment estimator with reliability.  Comprehensive empirical analyses demonstrate that our method yields substantial performance enhancements across both known and unknown classes in comparison to previous studies. Code is available at [https://github.com/niusj03/OwMatch](https://github.com/niusj03/OwMatch).

OwMatch: Conditional Self-Labeling with Consistency for Open-World Semi-Supervised Learning
Shengjie Niu',, Lifan Lin Jian Huang', Chao Wang²t
Hong Kong Polytechnic University, "Southern University of Science and Technology shengjie, ie. niu@connect. polyu. hk, 120128160mail. sustech, edu. en, j huangopolyu. hk, wangc6@sustech. edu. cn
Abstract
Semi-supervised learning (SSL. offers a robust framework for harnessing the po- tential of unannotated data. Traditionally, SSL mandates that all classes possess labeled instances. However, the emergence of open-world SSL (OwSSL) intro- duces a more practical challenge, wherein unlabeled data may encompass samples from unseen classes. This scenario leads to the misclassification of unseen classes as known ones, consequently undermining classification accuracy. To overcome this challenge, this study revisits two methodologies from self superviede and semi- supervised learning, self-labeling and consistency, tailoring them to address the OwSSI. problem. Specifically, we propose an effective framework called OwMatch, combining conditional lelf-labeling and open-world hierarchical thresholding. The- oretically, we analyze the estimation of class distribution on unlabeled data through rigorous statistical analysis, thus demonstrating that OwMatch can ensure the un- biasedness of the self-label assignment estimator with reliability. Comprehensive empirical analyses demonstrate that our method yields substantial performance enhancements across both known and unknown classes in comparison to previous studies. Code available at httpss//gthhbb.oommmuuuS//XXXOK
Introduction
Deep learning has made remarkable success in various tasks by leveraging substantial labeled training data [22 21 [3]. However, the costly and time-consuming labeling process limits their application in practical scenarios. Semi-supervised learning (SSL) significantly reduces the dependency on labeled data by exploring the inherent structure of unlabeled data [I6]. Despite promising results, SSL methods assume a glosed-world scenario where, though limited, all classes possess labeled instances. This assumption may be violated due to difficulties in data collection, such as in medical diagnostics, where it common encounter new symptoms or fail annotate due to technical constraints.
result, only a subset of the categories can be precisely labeled during the annotation process. Recently, numerous studies have sought to identify such novel classes effectively. Open-world SSL (OwSSL) is innovative in promoting dual objectives: classifying instances of seen classes and discovering instances of novel classes [5]
A notable challenge in OwSSL is the confirmation bias of model: model tends to predict instances as seen classes owing the lack of ground-truth supervision of novel-class instances. To eliminate this bias, existing works utilize unsupervised clustering methods, including contrastive loss and binary cross-entropy (BCE) loss, to group pairs identified by similarity metrics [5] [5]. Among these unsupervised techniques, self-labeling [1 6
has shown remarkable success, which involves
assigning self-labels to unlabeled data, with the generation of high-quality self-labels being the key
"The first two authors contributed equally to this work.
i Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

confidenca for age confillance for Classess Pradi confidence Thresholds
wdo condition condition
exsigtmmnt
Prediceivv Confidence
0.100
6.--
Thresholld
0025
39
Class Index
Class Index
(a)
(b)
Experimental results on the OwSSL problem. (a) Self-label assignment of seen classes (1-5) and novel classes (6-10) with or without conditional component in self-labeling, (b) Predictive confidence and hierarchical threshold for each class.
Figure 1:
factor. Previous studies utilize optimal transport align the self-labels for unlabeled data with given distribution. However, this self-label generation fully relies on the accurate prior distribution and lack of consideration of the supervision of labeled data. In TRSSL [34], the unlabeled data are assigned with a soft self-label based on the inaccurate class distribution, which raises biased estimation. Moreover, the confirmation bias still exists even if to align the unlaheled data in the same process. In addition to confirmation bias, a new issue called clustering misalignment arises when self-labeling depends solely unlaheled data: without proper guidance, the self-labell lalf-labeing process may adopt varying criteria for clustering. For example, it might cluster data based on superficial features like color rather than high-level semantic information. This misalignment can lead to results that deviate from expected outcomes and even contradict the classification criteria established by labeled data.
the ground -trtth distribution
Consequently, we propose new self- lelf-labeling scheme, conditional self-labeling, designed to address the challenges of OwSSL, particularly targeting issues related to confirmation bias and misalignment. This scheme limits self-labels for each class and incorporates labeled data to generate debiased and informative self-label assignments for all training data, further mitigating the confirmation bias as shown in Figure Ta Additionally, as illustrated in Figure Tb] seen classes typically exhibit higher predictive confidence, while novel clusters demonstrate variability in their internal learning progresses. The disparities learning paces between seen and novel classes, coupled with their distinct behaviors, necessitate the selection appropriate thresholds to f'cillitate cluster learning. To address these challenges and ensure a balanced learning process across classes, we propose hierarchical thresholding scheme.
We demonstrate our contributions as follows: 1) We introduce a novel conditional self- self-labeling method to incorporate labeled data into the clustering process, reducing confirmation bias and misalignment. 2) We design a hierarchical thresholding strategy that balances learning difficulties across different classes, helping unstable clusters gradually form. 3) Our theoretical analysis rigorously discusses the unbiasedness and reliability of conditional self-labeling estimator from population-level statistics. To the best of our knowledge, this is the first work proposing the expectation of chi-square statistics (ECS) to evaluate the reliability of self-label assignment estimation. 4) We conduct extensive experiments on various datasets, demonstrating the effectiveness of our approach, OwMatch, through detailed comparisons. On CIFAR-10, OwMatch significantly outperforms FixMatch [36] by up to 47.3%
in all-class accuracy, while on CIFAR- 100, it enhances TRSSL [34], the ftate-of-th-aa model in OwSSL, by up to 14.6% in novel-class and 7.2% in all-class accuracy.
2 Related work
Traditional semi-supervised learning (SSL). Traditional SSL assumes that labeled and unlabeled
data share an identical distribution. Extensive researches on SSL have spanned a considerable duration. The commonly employed strategies in SSL. consist of entropy minimization (2 14] consistency
regularization 30) and holistic methods 4 R 36
The latest progresses in SSL include adaptive
thresholding strategies
42 45 47], which enhance model performance by accounting for varying

---

Lubreled data
Aol (x))
Threduld
Acunt
On (a()))
Seen
Groun-wse
(KC,)
(CC)
9o (c)(
Nackern- Kmng
Creditenn Self Lake
fo
Cleds-W6e
<E
(22)
Pie
Novel
Figure 3: Illustration on the hier- archical thresholding scheme.
Prodictis
Lake
Rofianl Prodiction
Figure 2: Overview the OwMatch framework.
difficulties and learning conditions across classes, alongside other innovative techniques [99 41 that employ self-SL approaches to facilitate extracting the semantic information from unlabeled data. However, traditional SSL algorithms typically struggle to tackle the open-world problem in the presence of novel-class instances within unlabeled data.
Open-set semi-supervised learning (OSSL). OSSL expands the traditional SSL boundaries by allowing novel-class instances or outliers within unlabeled data. variety of OSSL approaches have emerged in recent years [46
35 & 23). A common solution among these methods is the optimization of the SSL objective exclusively for unlabeled samples deemed inliers. For instance, MTC [46] optimizes the network and estimates the anomaly score of unlabeled data alternately. OpenMatch (35] and T2T [23] train one-vs-all (OVA) classifiers for each known class to detect outliers. Subsequently, standard SSL objective (36 are applied to the remaining training data, excluding detected outliers. Furthermore, DS3L. [6 leverages a bi-level optimization technique to train weighting function, which mitigates the passive impact of out-of-distribution (OOD) samples. Nonetheless, these approaches are designed for classifying seen classes, thereby failing to learn from the novel class instances.
16
Open-world semi-supervised learning (OwSSL). OwSSL [ has been proposed to address a practical challenge: enabling the model effectively cluster novel-class instances while maintaining classification robustness on seen classes. One predominant research direction in this under-explored domain is BCE-based methods, including ORCA [5] and NACH [[5]. Additionally, there exist methods to discover novel classes by employing various other clustering techniques: OpenI.DN [33 employs bi-level optimization to train a pairwise similarity prediction network, which provides supervisory signal to the similarity of all pairs; TRSSL. [34] converts clustering into the self-laheling problem and applies Sinkhorn- Kankhorn-Knopp algorithm optimize self-label assignments. One subsequently
proposed Generalized Category Discovery (GCD) setting is similar to the OwSSL [40 49 with
detailed discussion is provided in Section 5.3
3 Methodology
Given training data consisting of labeled data Dj (x) (6)
- J and unlabeled N' + N and N > N Here x() Rd is the i-th instance with one-hot vector y E 1] as the corresponding label, where is the number all classes. We denote the set of classes in D; as Ct and the set of classes in D,, C., Previous traditional SSL studies assume Ci = C.. Here for OwSSL, we assume C & C. and C. C2 # O. Denote C, = C, a set of seen classes, Cn C. \ Ci considered classes. The desired OwSSL model is required to assign instances to either previously seen class C, novel class E C,,.
Problem setup.
data D., (x
N N=N where
set of novel classes, and C C₁ U C, set of all
For labeled dataset Di, standard supervised objective is employed shown Equation 9 Addi- tionally, OwMatch primarily incorporates two objectives: a) clustering objective, which leverages conditional self-labeling to refine the self-label assignment with the assistance supervision; b) confidence objective, which applies consistency loss with open-world hierarchical thresholding strategy to enhance predictive confidence and balance the different learning difficulties across all classes. We will elaborate on them respectively in Section 3.1 and 3.2

---

3.1 Conditional self-labeling
To effectively cluster the novel class instances, the self-labeling scheme [ has been considered in OwSSL. Formally, consider deep neural network (encoder) fa mapping imput data x to representation E IR,, the representation is followed by classification head h : RR" + K usually consisting of a single linear layer, converting the feature vectors into a vector of class scores. Denote go = a o h o so as probability function, where o refers to the SoftMax function. Moreover, denote q E IR as the ........))))...... ( E IR K as the self-label assignment for ), Asano et al. utilize a constraint of desired partition of Q to construct the transportation polytope:
soft self-label for x("), and set Q = (q q ,...,
(1) where 1 is the v-dimensional vector of all ones, P denotes the desired class distribution. On the other hand, we can obtain a probability output through p go (x((xx))) where a(.) refers to a specific weak augmentation, and denote P = [p(*), ___),_(_... p()]) as the matrix of probability
Q1 {Q E IRE R Q1v = "YP,Q"' = 1,}. =
outputs. This self-label assignment generation can be understood as solving an optimal transport problem [.. It minimizes the cross-entropy loss and aligns the training data with the desired class distribution:
max Tr(Q log()) QE
(2)
where Tr(-) is the trace of given matrix. Obviously, clustering through self-labeling primarily relies on the quality of generated self-label assignments. However, optimizing self-label assignments through unsupervised self-labeling is unreliable owing to the lack of supervision. TRSSL [34] utilizes the above-unsupervised technique optimize self-label assignment merely unlabeled data with the uniform class distribution. Despite prominent results, this unconditional self-labeling process 34 has notable flaw: it constructs transportation polytope based
inaccurate class distribution.
Moreover, we consider conducting self-labeling across all training data and constructing a trans- portation polytope with a precise class distribution. To mitigate the confirmation bias, we propose a conditional self-labeling method to refine the self-label assignment under partial supervision. Specifi- cally, we exploit the ground-truth labels from the labeled dataset and introduce another constraint Q2:
22 == {Q e R lq = y i = L''...,N'}.
Now, the conditional self-label assignment generation with the above two constraints can be formu- lated as:
Tr(Q (ooQ") + (Q).
max
QEQinQ.
where EE-) the entropy function, e a hyper-parameter controlling the smoothness of Q. We adopt fast version [10 of Sinkhorn- Knopp algorithum to optimize Equation 4 Jfffciennly and denote the optimal solution as Q = (q q(²)..... " ())) Empirically, conditional self-labeling significantly alleviates the confirmation bias, resulting expected distribution, shown in Figure [a Further theoretical analysis regarding estimators from unconditional and conditional self-labeling is provided in Section 4 Then, the clustering objective
self-label assignments that are much closer to the
has the form of: Leis = N ΣN (Q)()) (')).
3.2 Open-world hierarchical thresholding
Beyond the clustering objective, prompting the predictive confidence has proven effective for clas- sification. A similar goal arises in traditional SSL, wherein entropy minimization is employed to encourage low entropy (i.e., high confidence) in the prediction. FixMatch (36] leverages both con- sistency and pseudo-labeling to achieve exceptional performance with the following regularization: ΣNn (()) 2 (p( go (A(x()))), where p(() == arg ()) is predictive one-hot pseudo-label, with the -(h element set to 1. a and A represent weak and strong augmentation respectively. Here,
scalar hyperparameter denoting the threshold above which we retain pseudo-label. The effectiveness of the aforementioned regularization depends on accurate and suffi- cient pseudo-labels, which are directly influenced by the thresholding scheme. Under the close-word
assumption, extensive efforts 47 45 42 have been devoted to devising thresholding techniques based on the idea of balancing learning pace across classes with varying learning difficulties. How- ever, these techniques do not fit with the open- world scenario due critical challenge: the learning

---

pace of novel classes tends to be much slower [5]. The predictive confidence of these two groups does not share the same behavior, as shown in Figure lb
We introduce an open-world hierarchical thresholding scheme to balance this inconsistent learning pace the group level, leveraging these well-defined thresholds to retain high-quality and adequate pseudo-labels for learning. As shown in Figure 3 this scheme first estimates the learning conditions of the two groups and then hierarchically modulates the thresholds in class-specific fashion within each group.
First, split the dataset into seen (C&) and novel (C,.) groups based on the pseudo-label and estimate their overall learning condition by predictive confidence. Motivated by FreeMatch [42], we define the group-wish wise learning status for set of classes C₁ C. C,
1
n(c:)
Nc. Σ max(p
(T Ci), Ci = C. or Cu.
where No, Ei= T( E C.) denotes the number of samples whose predictive pseudo-labels belong to the group C. Similarly, the class-wise learning conditions can defined as
max(p
( ())( =),, c c= c..,...,KK
(6)
N
I(p c) denotes the number of samples whose predicted labels belong the e-th class. In practice, we utilize the exponential moving average (EMA) to update at each iteration. Then, merge these two learning statuses and obtain the open-world hierarchical threshold
where N₂ =
Se.
T(c) max.ec, So
-T(C),, c= c I,..,,,
where the e-th class belongs to the set C, (i.e,, E C. or C.). The learning condition vj distinguishes between seen and novel classes, while the class-wise condition adjusts for dass-wise differences. Ultimately, the confidence objective is:
N
Ccoms = N
Σ (m) T( "etpiHH((p,..AAA.. (Axx(())))))
(8)
H (1)), Together with the supervised objective Csup = 1 Ni H Ow.atch is
the overall objective for L = Leu + Lets + Leonf
Theoretical analysis of conditional self-labeling
To illustrate the superiority of conditional self-labeling over unconditional, we evaluate their estima- tors of the class distribution unlabeled data through rigorous statistical analysis. This transforma- tion justified as both self-labeling methods produce corresponding self-label assignments, each representing their estimation of the class distribution on unlabeled data.
Formulation. Assuming that the class distribution of real-world data conforms to prior informa- tion p = [pr,P2, P2, ... .P]). Suppose real-world data is composed of recognized labeled data and unrecognized unlabeled data, conforming to unknown class distribution P = [p],
pkx and p - pp",p' .kk respectively. We independently sample N = N - N instances from recognized and unrecognized data, respectively. Suppose N; N + N is composed of two random variables that denote the number of recognized and unrecognized samples belonging to the i-th class. Obviously, we have N >K ! and N = >K , No.
Objective. We hope to estimate the unknown class distribution P with ja based on prior in- formation P and observations of N/. ..., ...,...,.. .N then evaluate ja from unbiasness and ECS. Evaluation on both metrics requires estimating the number of samples in each class, denoted by
5

---

A = (A1, A.....,, Ax). Two self self-labeling approaches (unconditional and conditional) can optimize self-label assigmment, therefore obtaining two approximations of A, denoted by Auncon and Acon Denote the corresponding estimators as Muncoo and conn
Assumption 4.1. Assume that all drawn samples with a static number of samples and class distribution follow the multinomial distribution follows,
N1, N2, NK Multinomial P)
Ni. N ... No Multinomial( P()
Ni. N2.- N Multinomia( P").
Given the independency between Nf and Ny.. We basically have:
E E[N,] E[N]] + B[N]] Vi.j
(10)
Np: N/ + N"P..
Lemma 4.2. Suppose we want to test the null hypothesis (Ho) that categorical data N1, N2. Ne come from multinomial distribution with K classes and class probability Pf A chi chi-square statistic can be constructed to test the deviation between the observations , ,KK and expected outcomes for each class.
(n, Ep|N.13) Ep[N,]
X
XK-1
(11)
where Eplt] denotes the population expectation of random variable. lower chi-square value suggests that the observed data are consistent with Ho- Conversely, an exceedingly high chi-square value implies that either Hg incorrect or an event of low probability has happened.
Details of the above lemma are presented in Appendix E Then, we define the following metric to evaluate the goodness of fit of estimation based on chi-square statistics.
Definition 4.3 (Expectation of chi-square statisties (ECS)). The expectation of chi-square statis- ties (ECS) for jı are defined as the population deviation between the estimator of unlabeled class distribution fa and its true distribution P":
K Σ
(A: Ep (NT)
ECS(B) E(x)(A)) = E
(12)
Ep N
where A are estimators based on N N No. thus are still random variables.
Now, we introduce main theorems and demonstrate the superiority of our conditional self- labeling.
Theorem 4.4. Consider two estimators for class distribution on unlabeled data, Mumcon and Peear have uuncon is biased estimator and poon is unbiased estimator.
Theorem 4.5. Suppose r; == v' denote the ratio of label samples of the i-th class the whole samples, r :== E, Ts denotes the ratio of labeled samples to the whole samples. For unlabeled sample size N if N T for Vi E C2, Vj C.,, then ECS(@ccoa) S ECS( ECS( uncco))
Following rigorous statistical analysis, the generated self-label assignments from the conditional labeling method are closer to the true class distribution in the following scenarios:
Estimation based on large unlabeled sample size (N");
The difference between prior distribution P and class distribution of unlabeled data P is not negligible.
5 Experiments
This section presents a comprehensive evaluation of our approach. includes experimental results and in-depth analysis, demonstrating the effectiveness of our approach.

---

Table Average accuracy on the CIFAR-10/100 and ImageNet- with both novel class ratio and label ratio of 50%. We compare OwMatch with existing literature on OwSSL. Also compared with other related approaches of traditional SSI., OSSL, and NCD approaches following [5]. Proper modi- fications are made to make these approaches compatible with OwSSL; the details are in Appendix The results are averaged over three independent runs. The baseline figures are sourced from the
respective papers.
Method
CIFAR-
CIFAR-100
ImageNNet-100
Seen
Novel
All
Seen
Novel
Seen
Novel
FixMatch
36
71.5
50.4
49.5
39.6
23.5
20.3
65.8
36.7
34.9
L [6
77.6
45.3
40.2
55.1
23.7
24.0
71.2
32.5
30.8
CGDL 37
72.3
44.6
39.7
49.3
22.5
23.5
67.3
33.8
31.9
DTC 18
53.9
39.5
38.3
31.3
22.9
18.3
25.6
20.8
21.3
RankStats
86.6
82.9
36.4
28.7
SimCLR
58.3
63.4
51.7
28.6
21.1
22.3
39.5
35.7
36.9
UNO 12
91.6
69.3
80.5
68.3
36.5
51.5
ORCA 5
88.2
90.4
89.7
66.9
43.0
48.1
89.1
72.1
77.8
NACH
89.5
92.2
91.3
68.7
47.0
52.1
91.0
75.5
79.6
OpenLDN
33
95.7
95.1
95.4
73.5
46.8
60.1
89.6
68.6
79.1
TRSSI.
34
96.8
92.8
94.8
80.0
49.3
64.7
OpenCon Co
38
89.3
91.1
90.4
69.1
47.8
52.7
90.6
80.8
83.8
Ow.atch
93.0
95.9
94.4
74.5
55.9
65.1
91.7
72.0
81.8
OwMatch+
96.5
97.1
96.8
80.1
63.9
71.9
91.5
79.6
85.5
5.1 Experimental setup
Datasets. We evaluate our approach on CIFAR-10/100 [25]. ImageNet-100 1 and Tiny Im- ageNet [28]. A detailed description of these datasets is provided Appendix Specifically, ImageNet-100 100 dataset contains 100 classes sub-sampled from ImageNet- 1k following [39]. On all datasets, first split all classes into seen and novel classes with a novel class ratio. Subsequent experiments will adopt novel class ratio of 50% unless otherwise specified. Then, we will randomly assign labels a portion of the data from the seen classes according to the specified label ratio, while the remaining data, along with all samples from the novel classes, are assigned to the unlabeled set.
Implementation details. For fair comparison, apply ResNet-50 22 the backbone model for ImageNet- 100 and ResNet-18 for other benchmarks. We train the model with a batch size of 256 for Tiny ImageNet and 512 for other benchmarks. Following [15] experiments across all benchmarks implemented based on the pre-trained model from SimCLR (7]. We jointly optimize backbone and prototype parameters using the standard Stochastic Gradient Descent (SGD) with momentum. We apply the cosine annealing learning rate schedule for all experiments. Techniques including multi-crop and queue structure are employed to enhance the clustering objective. Additionally, RandAugment [9] serves as the strong augmentation for confidence objective. Additional implementation details are available in the Appendix B
Evaluation metric. assessing the efficacy of OwMatch, we adopt a multifaceted approach to evaluate accuracy following (5). Evaluation metrics include the standard accuracy for seen classes and the clustering accuracy for novel classes and all classes. Here, we leverage the Hungarian algorithm 26 align the predicted class assignment for novel-class instances with their ground-truth labels obtain clustering accuracy. We also report the joint clustering accuracy across all classes using the Hungarian algorithm.
5.2
Main results
We consider and evaluate two versions of our method, called OwMatch and OwMatcht. OwMatch represents the standard version as illustrated in Figure while OwMatch+ incorporates the multi- crop technique for additional augmentation. Detailed distinctions between the two versions are provided in the Appendix and 50% with the comprehensive experiment results provided in Table [ T2 and 13 Results in Table [ show that OwSSI. approaches significantly outperform current state-of-the-art methods in traditional SSL, OSSL, and NCD by considerable margin. On the other hand, OwMatch achieves
evaluate our method on all benchmarks using a label ratio of 10%

---

ttate--f-the-art across all benchmarks and evaluation metrics. It can not only classify novel classes accurately but also maintain robust performance on seen classes. On CIFAR-10, we observed OwMatch outperforms OpenI.DN on novel and all classes by 2.0% and 1.4%, respectively. It is noteworthy that the enhancement brought about by OwMatch is more pronounced on the CIFAR-100 dataset, which presents a greater challenge due to the increasing number of classes. Regarding CIFAR-100, our method surpasses TRSSL by approximately 14.6% on novel classes and 7.2% on all classes. Subsequently, we extend to evaluate ImageNet- and observe similar trend, with OwMatch+ showing significant improvement of 1.7 % on all-class accuracy compared to previous state-of-the-ar approaches.
Principle analysis of conditional self-labeling. OwMatch primarily relies on high-quality self- label assignment alleviate the model's confirmation bias. To clearly illuminate this progress during training, employ the Manhattan distance Σiek |c - c as metric to evaluate the bias between the considered class distribution {e)}==1 and the ground truth [e]";= Table demonstrates the debiasing process: the model's contirmation bias is pronounced in the early epochs, whereas the bias of optimized self-label assignment is relatively minor. As training advances, the self-label assignment continues to guide the model, effectively mitigating the confirmation bias, as reflected in the decreasing B,, and the absolute difference between Bn, and B,.
Table 2:
The Manhattan distance (MD) is used to evaluate the confirmation bias. The first row presents the bias between the model's predictive class distribution and the ground truth, denoted as Bm- The second row reflects the bias between the self-label assignment and the ground truth, denoted B,. The third row computes the absolute difference between Bn and B., highlighting the
debiasing effect of
self-label assignments.
Bias of Epoch
Epoch
Epoch
Epoch
Epoch 10
Epoch 30
Epoch
Bm
0.4463
0.2939
0.2474
0.1753
0.1505
0.0904
0.0798
B,
0.1004
0.0754
0.0893
0.0613
0.0407
0.0255
0.0219
Bm B.
0.3459
0.2185
0.1581
0.1140
0.1098
0.0649
0.0579
5.3 Ablations, analysis, and real-scenario applications
To investigate the impact of each component, we embark on comprehensive ablation studies with both novel class ratio and label ratio of 50%. The first row in Table showcases the foundational model performance, whose objective consists of only unconditional clustering objective and supervised objective, already achieving impressive performance. We then analyze the effect of integrating conditional self-labeling framework on CIFAR- 100, which boosts novel-class accuracy by 1.0% on average. Additionally, the positíve impact of consistency regularization is observed: roughly 0.9% enhancement across all evaluation metrics. Our ablation studies highlight the essential contribution of each component in OwMatch. Individually, each plays significant part in the intended functionality, and together, these elements coalesce into a cohesive and robust framework. We also ablate other factors, including the number of local views for clustering objectives and iterations for the Sinkhorn- Knopp algorithm, with detailed statements provided in Table 15 and I6.
Table 3: Ablation studies on each component with both novel class ratio and label ratio of 50%. Here, ConSL refers to conditional labelinn,,, PLCR refers to pseudo-lahel consistency regularization, and OwHT refers to an open-world hierarchical thresholding scheme.
Components
CIFAR-10
100
Tiny- Imay-ImaeeNet
ConSL
PLCR
OwlIT
Seen
Novel
All
Seen
Novel
All
Seen
Novel
x
96.5
90.2
93.3
78.8
56.7
67.7
66.5
38.1
52.0
x
x
95.4
96.4
95.9
79.2
58.5
68.7
66.0
39.4
52.4
V
V
x
96.3
97.3
96.8
80.1
59.4
69.6
68.6
42.0
54.2
X
V
V
97.1
90.4
93.8
80.7
59.7
69.9
69.7
41.4
54.6
V
V
96.5
97.1
96.8
80.1
63.9
71.9
68.8
42.4
55.0
Comparison study on varying thresholding strategies. We compare our proposed open-world hierarchical thresholding approach with two prominent techniques: static thresholding [36] and self-adaptive thresholding (2). As illustrated in Table 4 our proposal achieves superior performance
8

---

in both novel- and all-class clustering accuracy across varying thresholding techniques. While self- adaptive has proven effective under closed-world scenarios, it encounters challenges in open world settings. The pronounced disparity in overall learning conditions between seen and novel classes, as illustrated in Figure Tb] can lead to unstable global thresholds. The class wise adaptive approach based on that may exaggerate this issue, resulting in suboptimal performance. We implement hierarchical structure to mitigate the instability sourcing from distinct learning dynamics of seen and novel classes.
Table 4: Performance comparison of static, self- adaptive, and our OwHT thresholding techniques
Table 5: Comparison with GCD-related works:
average accuracy on the ImageNet-100 with both novel class ratio and label ratio of 50%.
on CIFAR-100 with both novel class ratio and label ratio of 50%.
Method
Backbone Seen
Novel
All
Thresholding Seen Acc Novel Acc
All Acc
GCD
41
ViT-B/16
91.8
63.8
72.7
Static 0.7
80.1
59.4
69.6
SimGCD
45
ViT-B/16
93.1
77.9
83,9
Static 0.8
79.8
63,9
71.7
InfoSieve
32
VIT-B/16
84.9
78.3
80.5
Static 0.9
80.2
62.8
71.3
CiPR 20
ViT-B/16
84.9
78.3
80.5
PrompiCAL
48
VIT-B/16
92.7
78.3
83.1
Self-adaptive OwHT 80.1
81.0
60.5
70.6
63.9
71.9
OwMatch++
ResNet-50
91.5
79.6
85.5
On the comparison between OwSSL and Generalized Category Discovery. The OwSSL setting resembles the subsequently proposed Generalized Category Discovery (GCD) setting [40], with both assuming the existence novel classes and that a portion of the data is labeled for seen classes. However, there are notable differences between these two groups of methods: 1) GCD-related methods leverage supervised contrastive learning (24) on labeled data and self-supervised contrastive learning (7] on all training data, whereas OwSSL typically employs pairwise similarity- -based methods for clustering samples; 2) GCD-related works typically employ a pre-trained ViT-Base/ backbone, which has significantly more parameters than the Re or ResNet-550 models commonly used in OWSSL. methods.
It is unfair to compare these two types of methods directly. Here, we still include comparison with those GCD-related works to demonstrate the effectiveness of our method. Table that method outperforms existing approaches in novel-class and all-class accuracy ImageNet-100 despite using a simpler model.
Ablation study on supervision components in the overall objective. The overall objective of OwMatch consists of standard supervised objective, clustering objective, and confidence objective. Both the supervised and clustering objectives involve the use of labeled data, raising concerns about overlap in functionality. To investigate the significance of each component, we conduct ablation study in which we modify the overall objective in two ways: 1) removing the supervised objective and 2) excluding labeled data from the online clustering process. The results are reported in Table 6
In the first case, we observe a decrease in seen-class accuracy while maintaining novel-class clustering performance, while the latter case exhibits the opposite tendency: seen-class accuracy remains high, but novel-class clustering accuracy declines. In comparison to the previous cases, our overall objective integrates both components strike balance between clustering and confidence. The supervised objective enhances seen-class accuracy through one-hot supervision, while the clustering objective with conditional self-labeling improves novel-class clustering accuracy by incorporating labeled data. This harmonious approach yields the best all-class accuracy while roughly maintaining both seen- and novel-class performance.
Table 6: Model performance under varying appli- Table 7: Estimation of the number of classes
cations of supervision in the overall objective. across benchmarks.
Objective
Seen
Novel
All
CIFARIO CIFAR100 ImageNet100
w/o supervised objective
76.8
64.4
70.6
Ground Truth
100
100
w/o supervision for clustering
80.3
61.2
70.7
Estimation
10
104
111
overall objective
80.1
63.9
71.9
Error
0%
4%
11%
The above evaluations are typically conducted under relatively ideal conditions: the datasets are and both the prior class distribution and the number of novel classes available.

---

However. in real-world scenarios, it is crucial to address the dependency these assumptions. We will elaborate on each of these aspects demonstrate the practical effectiveness of OwMatch.
Estimating the number of novel classes. OwMatch and other baselines typically assume that the number of novel classes pre-determined for clarity in evaluation. However, this prior knowledge is often unavailable in practice, necessitating a precise estimation of the number of novel classes in advance. We primarily follow the approaches of GCD [40] and TRSSL 33 to estimate the number of classes. Specifically, K-means clustering is performed on representations of the entire dataset from the pre-trained ViT-B/16 backbone. The optimal value of k is determined by evaluating the clustering accuracy the labeled samples calculated by the Hungarian algorithm. This accuracy serves scoring function, optimized using Brent's algorithm to find the that maximizes performance on the labeled data. The estimation results across generic benchmarks are shown in Table 7 which illustrates that the estimated class numbers come close to the ground truth. We also evaluate OwMatch's sensitivity varying extents of class number estimation error, with results reported in Appendix [
Table 8: Performance generic recognition benchmarks with varying imbalance factors (IF), with and without prior class distribution. These benchmarks come with both novel class ratio and label ratio of 50%.
Dataset
Prior
Uniform (IF=1)
IF=10
IF=20
Seen
Novel
All
Seen
Novel
All
Seen
Novel
All
w/
96.5
97.1
96.8
93.7
72.1
82.5
92.9
70.1
80.9
CIFAR-10
w/o
96.9
90.9
93.9
95.8
66.5
80.3
95.3
64.2
78.8
w/
80.1
63.9
71.9
76.8
42.0
57.3
76.1
35.2
51.9
CIFAR-100
w/o 82.5
57.9
69.2
74.6
39.7
54.1
73.9
33.9
49.2
42.4
55.0
61.7
25.1
41.6
62.4
21.7
38.3
Tmag-EmageNet
w/o 69.6
40.6
54.8
61.0
24.9
40.1
61.3
20.3
36.9
Data imbalance. Most generic benchmarks feature baass-balanced, whereas real-world data tend to exhibit long-tailed class distribution. Our approach accommodate to arbitrary class distribution by constraining the optimized self-label assignment to comply with the prior class distribution, thereby naturally mitigating performance degradation caused by data imbalance. We evaluate our approach on imbalanced benchmarks, constructed with varying imbalance factors following TRSSL 34 Results Table demonstrate that Ow Mathht effectively addresses the challenge of data imbalance.
Training without prior. In scenarios where prior class distribution is unavailable, propose an adaptive estimation scheme to make OwMatch still function without relying on any prior assumptions. Specifically, initially adopt bals-belanced prior if no prior information is available; then, the class distribution for conditional self-labeling is estimated and continuously updated based on model prediction. Next, standard training with estimated class distribution and distribution estimation alternately conducted, with results reported in Table g We observe that the reduction all-class accuracy achieved through the adaptive estimation scheme remains within 3% across almost all benchmarks and imbalance factors. These results reveal that the straightforward estimation technique performs robustly in the absence of prior knowledge.
Conclusion
This work integrates techniques from self-SL and SSL, refining them to present a new perspective on solving open-world SSL. We demonstrate that conditional self-labeling can achieve an unbiased estimation of the class distribution on unlabeled data with prior information, leading to high-quality sele-label assignment with reduced confirmation bias. Our future endeavors will be directed toward developing solutions that are more aligned with realistic scenarios where such prior information might not be readily available or hard to be estimated. This will involve exploring methodologies that can effectively handle uncertainty and variability inherent in real-world data distributions.
10

---

Acknowledgements
Chao Wang would like acknowledge the support from the National Natural Science Foundation of China under Grant 12201286, the Shenzhen Science and Technology Program 20231 1 15 165836001, Guangdong Basic and Applied Research Foundation 2024A151501 using the computing resources provided by the Research Center for the Mathematical Foundations of Generative Al in the Department of Applied Mathematics at The Hong Kong Polytechnic University.
This research was conducted
References
[1] Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning, In International Conference on Learning Representations (ICLR), 2020.
[2] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In Proceedings of the IEEE/CVF Interna- 2021.
tional Conference on Computer Vision (CVPR), pages
[3] David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: semi-supervised learning with distribution alignment and augmentation anchoring. In International Conference on Learning Representations (ICLR), 2020.
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[5] Kaidi Cao, Maria Brbic, and Jure Leskovee. Open-world semi-supervised learning. arXiv preprint 21Xi::2002203326., 2021.
[6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 33:9912 01499922-9944 2020.
[7] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML, pages 1597-1607. PMLR, 2020.
[8] Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. Semi-supervised learning under class distribution mismatch. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3569-3576, 2020.
[9] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 702-703, 2020.
[10] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems (NeurIPS), 26, 2013.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. Icee, 2009.
[12] Enrico Fini, Enver Sangineto, Stéphane Lathuiliere, Zhun Zhong, Moin Nabi, and Elisa Ricci.
A unified objective for novel class discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR, pages 9284-9292, 2021.
[13] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
[14] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in Neural Information Processing Systems (NeurIPS), 17, 2004.
11

---

[15] Lan-Zhe Guo, Yi-Ge Zhang, Zhi-Fan Wu, Jie-Jing Shao, and Yu-Feng Li. Robust Semi- Supervised Learning when Not All Classes have Labels. In Advances in Neural Information Processing Systems, May 2022.
[16] Lan-Zhe Guo, Zhen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe Deep Semi- Supervised Learning for Unseen-Class Unlabeled Data. In Proceedings of the 37th International Conference Machine Learning, pages 3897-3906. PMLR, November 2020.
[17] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisser- man. Automatically discovering and learning visual categories with ranking statistics. In International Conference on Learning Representations (ICLR), 2020.
[18] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 8401-8409, 2019.
[19] Tao Han, Junyu Gao, Yuan Yuan, and Qi Wang. Unsupervised semantic aggregation and deformable template matching for semi-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 33:9972-9982, 2020.
[20] Shaozhe Hao, Kai Han, and Kwan- Yee K Wong. Cipr: An efficient framework with cross- instance positive relations for generalized category discovery. arXiv preprint arXiv:2304.06928, 2023.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[23] Junkai Huang. Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. Trash to treasure: Harvesting ood data with cross-modal matching for open-set semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 8310-8319, 2021.
[24] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 13:18661-18673 2020.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Harold W Kuhn. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97 1955.
[27] P. Langley. Crafting papers on machine learning. In Proceedings of the 17th International Conference on Machine Learning (ICML), pages 1207-1216, 2000.
[28] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[29] Dong-Hyun Lee. Pseudo-label: : The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPL), 07 2013.
[30] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and supervised learning. IEEE Transactions on Pattern Analysis and machine intelligence, 1979-19999, 2018.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
12

---

[32] Sarah Rastegar, Hazel Doughty, and Cees Snoek. Learn to categorize or categorize to learn? self-coding for generalized category discovery. Advances in Neural Information Processing Systems, 36, 2024.
[33] Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. OpenLDN: learning to discover novel classes for open-world semi-supervised learning. In European Conference on Computer Vision (ECCV), pages 382-401. Springer, 2022.
[34] Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. Towards realistic supersieeee learning. In European Conference on Computer Vision (ECCV), pages 437-455. Springer, 2022.
[35] Kuniaki Saito, Donghyun Kim, and Kate Saenko. Openmatch: Open-set semi-supervised learning with open-set consistency regularization. Advances in Neural Information Processing Systems, 34:25956-25966 2021.
[36] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf- fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: simplifying semi- supervised learning with consistency and confidence. Advances in Neural Information Process- ing Systems (NeurIPS), 33:596-608. 2020.
[37] Xin Sun, Zhenning Yang. Chi Zhang, Keck-Voon Ling, and Guohao Peng. Conditional Gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 13480-13489, 2020.
[38] Yiyou Sun and Yixuan Li. OpenCon: open-world contrastive learning. Transactions on Machine Learning Research, 2023.
[39] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conference on Computer Vision (ECCV), pages 268-285, Springer, 2020.
[40] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7492-7501, 2022.
[41] Xudong Wang, Long Lian, and Stella X Yu. Unsupervised selective labeling for more effective semi-supervised learning. In European Conference on Computer Vision (ECCV, pages 427-445. Springer, 2022.
[42] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, et al. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint 27X5v22005.07246, 2022.
[43] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. Parametric classification for generalized category discovery: A bascline study. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16590-16600, 2023.
[44] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmen- tation for consistency training. Advances in Neural Information Processing Systems (NeurIPS), (6662666668,,,, 2020.
[45] Yi Xu, Lei Shang, Jinxing Ye. Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning, pages 1 11525-1155 11566. PMLR, 2021.
[46] Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. Multi-task curriculum framework for open-set semi-supervised learning. In European Conference on Computer Vision (ECCV), pages 438 454. Springer, 2020.
[47] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems (NeurIPS), 14:18408-18419 2021.
13

---

[48] Sheng Zhang, Salman Khan, Zhiqiang Shen. Muzammal Naseer, Guangyi Chen, and Fa- had Shahbaz Khan. Promptcal: Contrastive affinity learning via auxiliary prompts for gener- alized novel category discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1479-3488, 2023.
[49] Bingchen Zhao, Xin Wen, and Kai Han. Learning sumi-super gaussian mixture models for generalized category discovery. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 16623-16633, 2023.
14

---

Technical Appendices
Roadmap of technical appendices. These appendices structured follows: Appendix A introduces dataset details utilized in our experiments. Appendix B outlines the implementation specifies, including data augmentations and hyperparameters. Modification details for utilized baselines are provided in Appendix [C] Additional experiment results consisting of supplementary main results, in-depth analysis, and ablation study of hyperparameters are reported in Appendix [D] Complete and rigorous proof of theoretical results is represented in Appendix Appendix diiccusses the social impacts of our work, and the limitations are considered in Appendix G
A Datasets
The details the datasets utilized our experiments are provided in Table 2 which includes dataset statements, well as the corresponding backbones and batch sizes for training. The choice of backbone and batch size matches previous works [5] [5] for fair comparison. For CIFAR- 10/100 datasets, employ simple prooccsessinng encompassing random crop with padding and horizontal flip. To make ResNet-18 18 compatible with CIFAR input data with a small resolution of 32 x 32, we refine the CNN by setting the kernel size to 3 3 and applying stride of 1. We train the model with a batch size of 512 over 300 epochs. For ImageNet- 100 and Tiny-ImageNet datasets, raw images pre-processed with random resized crop and horizontal flip [2]. We use the standard version of ResNet-50 and R 18 as the backbone, respectively. We leverage the standard SGD method with momentum and weight decay optimize the network parameters; hyperparameters in Table I1
Table 9: Details of evaluation benchmarks, we show the number of classes, dataset statistics, selected backbone, and batch size for training.
Dataset
Num. Class
Train Samples
Test Samples
Backbone
Batch size
CIFAR-10 25
10
50,000
10,000
ResNet-18
512
CIFAR- 100 25
100
50,000
10,000
ResNet-18
512
ImageNet- 100
100
128,545
5,000
ResNet-50
512
Tiny-ImageNet
28
200
100,000
10,000
ResNet-18
256
B Implementation details
Computational resources. The foundational algorithm of our study is constructed utilizing Python 3.8 and Py 2.1 [31] All experiments are carried out on NVIDIA's Tesla A100 GPU with 40G memory. All benchmarks are public and can be easily downloaded.
Strong augmentation. Furthermore, we apply the strong augmentation to input data for all experi- ments following FixMatch (36), including random resized crop, horizontal flip, and RandAugment 9.. It should be noted that the only additional enhancement in strong augmentation is RandAugment compared to basic pre-po processing. Specifically, RandAugment randomly selects transformations from a collection of options for each sample in a mini-batch. We employed the same sets of image transformations those used in RandAugment. A complete list of these transformations can be found in the original work 99.
MwMetth v.s. OwMatch+. We evaluate two versions of our approach in the main results, as illustrated in Table [ 12 and 13 In short, OwMatch+ further incorporates the multi-crop technique as an additional augmentation to boost clustering capacity and, hence, improve model performance. The multi-crop strategy was proposed by Sw [6] to additionally augment images by covering only small sections. The resulting low-resolution images, referred to as local views, allow more augmentations marginally increased computational cost. Compared to simple preprocessing (global views), local views generated through multi-crop strategy involve resized cropping at smaller scales but in greater numbers, as illustrated in Table T0 And they experience additional color distortion 7 consisting of random color jitters, solarizing, and equalization in pursuit of model robustness. Here we apply the multi-crop technique to produce many low-resolution images (local views) and set the hyperparameters following PAWS 22, as detailed in Table I0
15

---

Table 10: The details of crop augmentation hyperparameters.
Dataset
Global views
Local views
Crop scale
Resize
Crop scale
Resize
Numbers
Distortion
CIFAR-I0
(0.75.1)
(0.3,0.75)
18
0.5
CIFAR-100
(0.75,1)
32
(0.3,0.75)
18
4
0.5
ImageNet- 100
(0.2,1)
224
(0.14,0.3)
18
Tiny-ImageNett (0.75,1)
64
(0.3,0.75)
36
4
1
Except for additional multi-crop augmentations, OwMatch and OwMatch+ differ slightly in the form of clustering objective. Specifically, we augment the input image by taking 2 full-resolution crops (normally and strongly augmented global views) and V low-resolution crops (local views). Note that we optimize the self-label with only global views, since local views can only capture localized semantic information and are unable to provide comprehensive overview of the entire image. We promote the model consistency by encouraging the prediction of different local views to be close to the optimized self-labels. Specifically, the clustering objective of OwMatche is formulated by
C. (V+1)N
Σ
Σ (())) ===1
go (ax + (() p(.)))
(13)
where @ correspond the optimized self-label of global view, and g
go (av (x(())
stand for predictions of V local views. Increasing the number of random low-resolution crops encourages the model learn global-to-local information (2) which reflects in performance gain
across all benchmarks; see main results in Table [ 12 and 13 The utilization of low-resolution
images boosts the model's efficiency with only marginal rise computational costs.
The effectiveness of conditional self-labeling is significantly compromised when the ratio of batch size (B) class numbers (K) is relatively small. In scenario where class is disproportionately sampled in the labeled data of a batch, the conditional self-labeling mechanism might be unable to reassign the unlabeled data that particular class. Notably, when this ratio falls below assigning B samples to all K classes becomes unfeasible. Therefore, we leverage the queue structure to store data from previous batches by following SwAV (6. In practice, a queue of 1024 logits are stored for the implementation of the Sinkhorn- Sinkhorn-Knopp algorithm, which is utilized to derive the self- assignment. We then retain the logits from the last batch the optimized self-labels to construct the clustering objective. Such queue length proves effective in our experiments with large batch size (e.g., 512) and relatively small number of categories (e.g., 100 classes for CIFAR- 100). However, when dealing with high-resolution images that encompass a greater variety of categories, storing much more previous data information is essential.
Hyperparameters Here, we provide comprehensive list of hyperparameters in Table For hyperparameters related to the SGD optimizer, we adhere to the settings used in the previous works [4] to ensure a fair comparison. Regarding hyperparameters introduced by our proposed method, we perform ablation studies to determine the most appropriate values, specifically for SK-iteration and the number local views, as detailed in Appendix [ These hyperparameters are selected based on a balanced consideration of computational costs and model performance.
Table 11: List hyperparameters for CIFAR-10/100, ImageNet-100, and Tiny- ImtgeNe.. Hyper-parameter
CIFAR-10 CIFAR-100
ImageNet-100
ImageNtt
SGDD-moeentum SGDDllerning rate
0.9
0.1
SK-c
10
SK-iteration of local views
10
SGD-weight decay
0.0005
0.0005
0.0001
0,0001
16

---

C Baseline implementation details
We compare our proposal with baselines from other settings: traditional SSL, open-set semi- supervised learning (OSSL), and novel class discovery (NCD). We will elaborate the modifications to these settings separately. Traditional SSL methods cannot deal with the novel-class instances and extend it in the following manner: samples firstly divided into seen-class and novel-class instances based on out-of-distribun (OOD) criteria, then we report the standard classification accuracy seen-class instances and apply K-means algorithms to achieve clustering accuracy the novel-class instances. Hungarian algorithm (26] is utilized to match the clustering result and their ground-truth labels, this result is reported novel-class accuracy. For the traditional SSL. method, FixMatch [36], we separate the OOD samples based on confidence scores produced by the Softmax function. Many OSSL approaches like CGDL [37] are naturally capable of detecting outliers, thus we directly cluster the considered outliers by the K-means algorithm and report the clustering accuracy without manually inspecting OOD samples. Since DS3L [16] applies the re-weighting technique to downsize the passive effect of OOD samples, we consider the samples with the lowest weight as outliers. Both traditional SSL and weighting-base score to partition the inliers and outliers. Here we follow ORCA [S] to determine the threshold for OOD samples by using ground truth class information.
OSSL approaches depend on the OOD likelihood
NCD methods are trained to discover novel classes in unlabeled data with totally novel-class instances, thus failing to recognize seen-class instances. For NCD approaches without seen-class classification heads, like DTC [18] and RankStats [17.. we report the performance on novel classes and extend them to classify seen classes by assuming the seen-class instances in unlabeled data as novel. Then we extend the unlabeled classification head to include logits for seen classes by following [5] and leverage the Hungarian algorithm match the discovered classes with ground-truth labels within labeled data, the best assignment is reported as ceensclass accuracy. And for recent UNO [12] with explicitly labeled classification heads, we generate pseudo-labed labels for both seen- and novel-class instances based on model predictions from concatenated labeled and unlabeled classification heads. Therefore, both seen and novel class classification accuracy can be directly computed and reported. We apply the same pre-trained model on NCD methods to demonstrate that the enhanced performance not attributable to the the application of pre-training. Additionally, we present the clustering outcomes based on representations from the pre-trained model.
D Additional results
(a) Epoch 1
(b) Epoch 10
(c) Epoch 100
Figure 4: Confusion matrices on CIFAR-10 with both novel class ratio and label ratio 50%. The model needs classify the initial five seen classes accurately (as reflected in the diagonal elements). While for the novel classes (6-10), the classes clustering are required to align with the ground-truth label (dark blue in one cell).
Training process. We plot the confusion matrices CIFAR-10 with both novel class ratio and label ratio of 50% in Figure This collection of images compellingly demonstrates that the bias derived from novel classes is progressively mitigated as the experiment advances, leading to continuous improvement the model's prediction accuracy.
As depicted in Figure 4a| at the beginning of training, the model struggles to effectively distinguish between novel and seen class instances, although can classify seen class instances normally. As training progresses, increasing number of samples assigned to the novel classes and prediction
17

---

accuracy for the seen-class instances advances, illustrated in Figure 4b In the later stages of training as shown in Figure 4c the model becomes capable of accurately classifying the seen-class instances and clustering novel-class instances simultaneously.
Table 12: Average accuracy the CIFAR-10/100 and ImageNet-100 with novel class ratio of 50% and labeled ratio of 10%.
Method
CIFAR-10
CIFAR-100
ImageNet-100
Seen
Novel
All
Seen
Novel
All
Seen
Novel
All
FixMatch
86
64.3
49.4
47.3
30.9
18.5
15.3
DS 16
70.5
46.6
43.5
33.7
15.8
DTC 18
42.7
31.8
32.4
22.1
10.5
RankStats
177
71.4
63.9
66.7
20.4
16.7
17.8
UNO 12
86.5
71.2
78.9
53.7
33.6
42.7
66.0
42.2
53.3
ORCA
82.8
85.5
84.1
52.5
31.8
38.6
83.9
60.5
69.7
NACH
91.8
89.4
90.6
65.8
49.2
OpenLDN
33
92.4
93.2
92.8
55.0
40.0
47.7
TRSSL
34
94.9
89.6
92.2
68.5
52.1
60.3
82.6
67.8
75.4
OpenCon
38
62.5
44.4
48.2
OwMatch
89.3
92.2
90.7
59.5
43.7
51.2
86.4
69.2
77.8
OwMatch+
94.4
96.2
95.3
69.9
51.5
60.3
87.8
72.7
80.2
Main results with label ratio of 10%. We evaluate approach CIFAR-10/100 and ImageNet- 100, closely similar to Table [ but with the label ratio adjusted to 10%. The results are detailed in Table 12 OwMatch+ continues to maintain state-of-the-an performance across most benchmarks and evaluation metrics.
Performance sensitivity to varying extents of class number estimation error. As illustrated in Figure OwMatch maintains robust performance over a range of errors.
Accuracy (96)
Novel
Class Estimation Error (%)
Figure 5: Accuracy function of class number estimation error on CIFAR-100 dataset.
Main results on Imag-Im addition to the results on CIFAR and ImageNet datasets, we also evaluate OwMatch and OwMatch+ on Imny-Im with 50% novel classes in Table [3] We found that both OwMate and OwMatth+ surpass previous state-of-the-art methods across benchmarks and evaluation metrics.
Different novel class ratio. Previously, we assessed model's performance with a constant novel class ratio of 50%, which is also variable in real-world scenarios. Here, we alter this value and fix the label ratio within seen classes to 50%; the results are reported in Table [4] The model's performance generally exhibits a declining trend across all benchmarks and evaluation metrics as the novel class ratio increases. important to note that as the number of novel classes increases, the total amount of labeled data decreases.
18

---

Table 13: Average aceuracy on Tiny-ImageNer datasets with label ratio of 10% and 50%.
Method
50% label ratio
10% label ratio
Seen
Novel
All
Seen
Novel
All
DTC 18
28.8
16.3
19.9
13.5
12.7
11.5
RankStats
17
5.7
5.4
3,4
9.6
8.9
6,4
UNO
12
46.5
15.7
30.3
28.4
14.4
20.4
OpenLDN
25.5
41.9
TRSSI
59.1
24.2
41.7
39.5
20.5
30.3
OwMatch
62.9
38.9
50.6
44.8
25.4
34.7
OwMatch+
68.8
42.4
55.0
55.8
33.3
43.3
Table 14: Model performance across benchmarks with varying novel class ratios.
Novel class ratio
CIFARIO
CIFAR100
Tiny-ImageNet
Seen
Novel
All
Seen
Novel
All
Seen
Novel
All
50%
96.5
97.1
96.8
80.1
63.9
71.9
68.8
42.4
55.0
60%
96.5
92.1
93.9
80.3
60.4
68.1
68.5
41.1
51.7
70%
97.5
91.0
81.3
58.6
71.3
34.7
45.6
80%
98.4
92.9
94.0
78.9
58.0
61.8
69.5
32,9
40.0
90%
97.8
93.6
94.0
82.0
50.7
53.5
69.4
26.4
30.3
Efficient labeling strategy under a fixed budget. In the previously conducted experiments, we assessed the model's performance by altering the novel class ratio and label ratio, respectively. Merely altering single factor does not yield highly convincing inferences, it is impractical to fix the novel class ratio or the label ratio for some classes real-world scenarios. Here, we consider fixed budget, specifically the total number of labeled data, as illustrated in Figure 6 This comparison aims to shed light on how the balance between labeled data in seen classes and the proportion of novel classes influences model performance under fixed level of supervision.
When the supervisory information is held constant, configuration with a smaller portion of labeled data spread across a greater number of different classes results in higher accuracy for both all classes and novel classes. Additionally, it is observed that as the number of novel classes decreases, the accuracy of the seen classes improves. This improvement is attributed to the reduced complexity of the classification task when there are fewer categories to classified. From these observations, can be inferred that within a limited labeling budget, it is more effective to label broader category of samples, thereby capturing as many representative points as possible within the feature space. This strategy appears optimize the model's performance across both known and novel classes.
Hierarchical thresholding scheme with scarce supervision information. Previously, we observed that in the scenario where the label ratio seen classes is 50%, the performance difference between adopting the hierarchical thresholding strategy and setting a high static threshold hyperparameter not significant. Here, maintain the novel class ratio 50% while decreasing the label ratio to 10%. Figure 7a and Figure/7bJdemont [7]demmnsttrtee that a hierarchical thresholding scheme not only retains more laeudo-labels but also preserves the predictive accuracy of selected pseudo-labels.
Table 15: Model performance on CIFAR-10/100 and ImageNtt with different number of local views to contrastive learning.
Crops
CIFAR10
CIFAR100
Tiny- ImtgeNet
Seen
Novel
Seen
Novel
All
Seen
Novel
All
2
95.9
97.7
96.8
78.8
61.6
70.2
66.4
43.6
54.2
96.5
97.1
96.8
80.1
63.9
71.9
68.8
42.4
55.0
6
96.8
97.2
97.0
81.1
59.4
69.8
70.1
42.1
55.6
Different number of local views in contrastive learning. Contrastive learning boosts clustering by acquiring different views of the data and promoting consistency across these views the represen- tation level. The manner of data augmentation and the number of augmented views largely determine
19

---

90
80
Novel Class All Class
50
40
30
10
20
30
40
50
Labeled
Figure 6: Model performance on CIFAR-100 with varying numbers of seen classes and a fixed amount of labeled data (5% of total data). Models trained with more seen classes generally perform better on both novel and all classes, despite having fewer labeled data per seen class.
62001
0.65
20000
Predicaiee
His ancheoall Suck
0.3
Meurhha Stakk
2
100
390
250
900
359
302
(a) Number of selected pscudo-labels
(b) Pscudo-labels accuracy
Figure 7: Open-world hierarchical thresholding scheme generally selects instances pseudo- labels (a), and the quality of pseudo-labels is also enhanced (b).
the model performance. Here we take the combination of color distortion and random resized crop 7 as augmentation and apply a multi-crop technique [6 to produce many low oowrrssouution images (local views). We compare the performance with varying number of local views, the results reported in
Table 15
The incorporation of local views significantly enhances model performance. However, it's observed that as the number of local views increases, the incremental benefits to the model come to plateau, while the training and computational time considerably escalate. This aligns with the notion that local views assist capturing local patterns, aiding in the development of robust representations. However, there is threshold beyond which the addition of more local views contributes less to learning, overshadowed by the rise in computational demands.
Different number of iterations in the Sinkhorn-Knopp algorithm. Conditional self-labeling is proposed to optimize high-quality self-label assignments, which depends on fast version of the Bookhorn-Knopp algorithm [00 to solve this complex linear programs efficiently. Resolving this involves iterative processes to converge on the optimal solution. We evaluate the model performance across different iterations, with the results presented in Table 16
Generally, model performance tends increase with more iterations of the Sinkhorn-Know algorithm. Despite the positive correlation tendency, we observe that certain iterations (e.g., 6 for CIFAR-100 with both novel class ratio and label ratio of 50%) already achieve satisfactory outcomes, with only marginal gains from further iterations.
E
Proof details
Here, we derive ECS for both unconditional and conditional self-labeling in the following two lemmas.
20

---

Table 16: Model performance on CIFAR-10/100 and Imy--mageeNet with different iteration numbers in Sinkhorn Sinkhorn-Knopp algorithm.
SK-iters
CIFARIO
CIFARI00
Tiny- ImageNet
Scen
Novel
Seen
Novel
All
Seen
Novel
All
3
96.8
91.6
94.2
81.5
55.0
67.6
68.6
41.7
54.1
96.7
91.0
93.9
81.4
62.4
71.3
68.0
42.8
54.5
10
96.5
97.1
96.8
80.1
63.9
71.9
68.8
42.4
55.0
Lemma E.1. Estimators of A from unconditional self-labeling has the form of
Auncon NN No <xK,
(14)
and ECS for uunco
No uncon can be derived as
(" p
ECS( uncon)
(15)
PN
Lemma E.2. Estimators of A from conditional self- labeling has the form of
Acon [NP1 - N1, NPk - -N
(16)
and ECS for Moon No Acon can be derived as
N N'p pt)
ECS(
(17)
N"p,"
E.1 Proof of Lemma
Proof. Under the null hypothesis Ho, the sample size of the i-th class follows the Binary distribution with parameters of and Therefore, we have the expectation and standard error of N; as (P(N)) Np; and SDN,,) Np.(1 p.). respectively, where the standard deviation of N; measures the average deviation of random variable N₂ from its expected value. And for observation n, for each class, the discrepancy can be denoted as n4 EEp (NN). To ensure discrepancies for each class are evaluated on a consistent basis, dividing them by their standard errors under Ho, which enables to focus the standardized variable n; -:-R<(N.)
(Np:(1-p.))772-
Note that Σx Ep(N)) Np; N, therefore the discrepancies across the K classes cannot simultaneously be all positive or all negative. To measure the discrepancies of all K classes, we sum the squares of the discrepancies of each class. This formulation ensures that discrepancies independent of sign, merely reflecting the deviation between observed and expected values under Ho:
(n1 Np1)² Np - PI)
(n2 Np2) + Np2(1 - P2)
(ną NNk)2
(18)
Np.(1 - PK)
We then prefer to exclude the factors (1 - p:)) from the denominators of the sum for two primary reasons. Firstly, if numerous classes exist and none of them has significant large probabilities, then (1 - p:) almost negligible. Moreover, when the expectation (NN)) is substantial, the approximation of each discrepancy adheres to standard normal distribution, which leads to the construct of chi-square statistics with the degree freedom of 1. Then the summary statistic can be obtained as follows
nn:
2p.) p;
K
(n: Ep [N:13)
X2 X
(19)
Ep N.)
21

---

E.2 Proof of Lemma E.1
Proof. TRSSL 34 posits that unlabeled and real data share the same class distribution, and optimize the self-label assignment unlabeled data solely based prior information P. Owing to the
alignment constraint between the generative self-label assignment and prior information, as shown in
(T. Then the estimated number of samples in each class from conditional self-labeling is
A uncon p1, PI, N2, P2, .. N pk]:
(20)
where both N and P1:-.. PK are known, therefore Auncon is constant vectors. Then the estimator Funcon No A uncon is also constant vectors. Thus, the ECS for Puncon can be calculated directly,
K
NN P. Epl N
ECS
(21)
Ep
[NT]
(N "DD
(22)
N"p
N" 221)
(23)
E.3 Proof of Lemma D.2
Proof. Different from an unconditional setting, conditional self-labeling considers partial supervision. Specifically, except for the constraint brought from prior information, additional constraint is constructed to realize the alignment between clustering results and existing labels labeled data, as shown in 3). Then the estimated number of samples in each class from conditional self-labeling is
Acon
[N No.
NpK N)
(24)
where N. P1, ... PP are static values, and ,.. No are set of random variables. Note that in specific experiment, we can get set of observations of N ... N thus estimation based on conditional self-labeling is feasible. Then consider the estimator Mcon = 1 of the ECS for Pcon are as follow,
No Acon. derivation steps
K
(Np:
N
Ep N
K
NNp. N N Nup
(C
E
E
(25)
N
According to 10 we have
K (Np: N NN
K
(N: N2Y² N"p
(N - - [N
=E
(26)
N"p
N"P
Since ECS is also defined the population level, thus have
K
(N
2
K E(N Ep 2
Var(N!
E
(27)
Nup
Nop
N
Since N/. N7.
No
Multinomial Pt). we have
K (ar(NN)
NO (1 pt) N"
(28)
Nup
22

---

E,4 Proof of Theorem 4.4
Proof. From Theorem
E.2 and Theorem
E.3
we have that
P Puncon
P1.P2-
.PK] P.
(29)
Np
N
Npx - N K
P
Peon
(30)
N
N
Note that Munegn is exactly the prior information, Poon are function of set of random variables N N2, :.. N K.
then we have
Npi N
NPK
N
E
EE
(31)
N
No N'Pk
Npi N'p!
Npi
(32)
Na
Nu
According to (TO), we have
Npi P
Np1 N'P
N
No
(33)
N
N
N
N
I P2.
.Pk
(34)
Thus, Pcon a unbiased estimator of C".
E.5 Proof of Theorem 4.5
Proof. Note that == x is non-negative and obtain zero if and only if the i-th class denote novel class.
When the i-th class refers seen classes, have p P- then
N" - pt)² 2 N (r. - r p. 2 1.
When the i-th class refers an novel class, we have pp
-
p then
2 No.
2
(p: - 2 No
Pi
=
TP:
2 Nr 1.
Therefore, get that
K
Nº p2)² NO P1 (1 - -p!)
(
F.C (pcom))))
(35)
PN
K
Nº 22
2 0.
(36)
F
Broader impacts
This research delves into the issue of sumi-supervieed learning (SSL) in situations where not all classes possess labeled instances, an aspect that has received limited attention within the realm of SSL. We aim draw increased focus towards examining the resilience SSL in diverse real- wolld scenarios, thereby fostering a broader application of SSL in various contexts. However, the current accuracy is not very high for some challenging datasets. Therefore, the predictive results should be best used references rather than treated as ground truth.
23

---

G Limitations
OwMatch, similar existing methods in OwSSL, faces a significant challenge when applied to imbalanced datasets or unknown prior class distribution. Existing OwSSL methods are typically applied on bass-balanced datasets where instances of each class share nearly the same frequency; the model performance would deteriorate when encountering imbalanced datasets. On the other hand, prior class distributions are not available in real-world applications. Addressing the dependency on prior class distribution and effectively handling datasets of arbitrary composition remain challenging for existing OwSSI. algorithms, including OwMatch.
Recognizing this. we propose an adaptive estimation scheme for the OwMatch framework and show its feasibility in the experiments, with results reported in Table Although a performance decline within 3% may seem acceptable, it is worth further consideration and exploration to determine whether further optimizations can enhance model performance without any prior. At the same time, we aim to prove the convergence of this adaptive algorithm in our future work.
24

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We claim our scope and contributions clearly in abstract and Section |
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work and possible refinements in Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

We provide all theoretical results with detailed assumptions in Section 4 and complete proof in Appendix E from the Appendix. Guidelines:
Justification:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide all training implementation details in Section 5.1 and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: All the datasets are public as illustrated in Appendix B We will release the full code and commands once the manuscript gets accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wttgg/udd/o/oouuubbeeeee ssinoolicyy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We claim the experiment setting and details in Section 5.1 and Appendix Choosing syper-parameters is based on the observations of the ablation study, as illustrated in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The reported results are averaged after running experiments multiple times with the same random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information regarding compute resources in Appendix B from the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
cc http///www.....po........ccaaaaa
Answer: [Yes]
Justification: We have meticulously reviewed the NeurIPS Code of Ethics to ensure compli- ance with the requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impact in Appendix E
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work focuses on classification algorithms using public datasets and therefore poses no risks concerning safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all the original papers involved. All utilized dataset and models are public, as detailed in Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The paper introduces newly developed code, which is well documented. The documentation includes detailed descriptions of the code's purpose, functionality, usage examples, and license details. The documentation will be publicly available.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "3QGgEJXm8a", "reviewer": "Reviewer_PbWT", "summary": "The paper has a novel idea, clear problems, and rich experimental results. It is a good article. However, there are some problems that need to be further optimized.", "questions": "1.\tReferences should not be used as sentence components, such as reference [31].\n2.\tI would like to know how the proposed method reduces the bias, not just the final result. Can the author provide some principle experimental analysis?\n3.\tMaybe I overlooked it. I did not find any experimental results on Tiny ImageNet that are consistent with Table 1.\n4.\tFrom the confusion matrix, at 100 iterations, for unseen data, only the sixth category can be predicted correctly, while the other categories are all wrongly predicted. This is different from what the author described.\n5.\tWhy does the prediction effect of seen classes become worse after adding the ConSL module?\n6.\tThe authors should try to align the experiments. The experimental data for each experiment is incomplete and is based on three of the datasets.", "limitations": "As above", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The idea is clear and the problem is prominent.\nThe experimental results prove the superior performance of the proposed method in open world scenarios.", "weaknesses": "Some results need to be supplemented and explained\nRequiring some proof of principle, theory or experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your comments are appreciated, and they have been beneficial in enhancing our work.\n\n**[Q1;3;4]** Based on your suggestion, we will revise our manuscript to fit the appropriate writing requirements. First, we will correct the reference citations in detail to avoid them being sentence components. Secondly, we will move manuscript's Table 8, which is the results on Tiny ImageNet, from the appendix to the experimental section. Regarding the confusion matrix, we utilize the Hungarian algorithm to calculate clustering accuracy for novel classes, which matches discovered novel clusters to ground truth labels through optimal assignment. Therefore, the samples don't need to align along the diagonal; the presence of five distinct clusters already indicates effective clustering on novel-class samples. To avoid confusion, we will reorder the confusion matrix to match the manuscript's description.\n\n**[Q2]** The main challenge in OwSSL involves the tendency to predict samples in unseen classes as seen classes. Conditional self-labeling incorporate label data into online clustering as prior to correct the the unseen class distribution. Based on this, the posterior prediction exhibits less bias tendency. To demonstrate its effectiveness, we compare the number of samples predicted to one class with ground truth. We expected a smaller gap between them under an effective method. Figure 1(a) in the manuscript shows that with the conditional component key, the number of samples predicted to each class is drawn closer to its ground truth for both seen and unseen classes.\n\n**[Q5]** The last two columns of manuscript's Table 2 may seem to imply a decline in seen-class accuracy. However, this resulted in significant improvements in both unseen-class and all-class accuracy. Besides, seen-class accuracy is calculated by applying the formula on samples whose ground truth belongs to seen classes rather than all classes. However, that fails to measure the false assignment of novel classes. Consider an extreme model only assigns samples to seen classes, while it minimizes the mistake of losing any possible seen instances, it completely ignores the unseen set. This is not a preferred model/representation. An unbiased and generalized model should balance the prediction of all classes. As for the comparison of these two models, performance does not degrade, in fact gets more accurate. While the recall slightly decreases, the precision is improved.\n\n**[Q6]** The “-” in the tables indicates that the authors of these works have not publicly provided the relevant results. While we are confident in the performance of our method, we are concerned that our simple reproduction of their work might not fully represent the potential of their methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kXvjNq2UtF", "reviewer": "Reviewer_dC6G", "summary": "This paper proposes OwMatch, a new approach for open-world semi-supervised learning (OwSSL). The key contributions are: (1) A conditional self-labeling method that incorporates labeled data into the clustering process to reduce confirmation bias and misalignment. (2) A hierarchical thresholding strategy to balance learning difficulties across different classes. (3) Theoretical analysis of the unbiasedness and reliability of the conditional self-labeling estimator. (4) Extensive experiments demonstrating state-of-the-art performance on multiple benchmarks. The method combines ideas from self-supervised and semi-supervised learning, using consistency regularization and self-labeling tailored for the open-world setting where unlabeled data may contain novel classes.", "questions": "Please refer to questions in the weakness section.", "limitations": "Limitations and potential impact are well discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a novel combination of conditional self-labeling and open-world hierarchical thresholding to address the challenges in OwSSL. Hierarchical thresholding is proposed to address the issue of different learning pace of seen and novel classes and seems to work well. \n2. The paper is well-written and clearly structured, with detailed explanations of the methodology, theoretical analysis, and experimental results. The figures and tables effectively illustrate the key points and support the claims made by the authors.\n3. The theoretical analysis provides a solid foundation for the proposed method, demonstrating its unbiasedness and reliability in label assignment. The empirical results show significant performance improvements across various datasets, indicating the method's robustness and effectiveness.", "weaknesses": "1. The Open-world Semi-supervised Learning setting is very similar to or the same as Generalized Category Discovery, both assuming novel classes exist and part of the data has labels in seen classes. The paper lacks discussion and comparison with closely-related work in generalized category discovery, such as GCD[1], SimGCD[2], and BaCon[3].\n\n\n2. A critical ablation study is lacking/needed to verify the effectiveness of the proposed open-world hierarchical thresholding (OwAT). OwAT is an incremental adaptation to existing self-adaptive thresholding in FreeMatch[4] by considering the different group learning speeds in novel and seen classes. In the existing ablation study, it is shown that OwAT boosts model performance, but I think the previous self-adaptive thresholding contributes most of the improvement and it is unclear how much the proposed hierarchical design really helps. Besides, since the self-adaptive thresholding adjusts thresholds based on the learning status of different classes, I think it can automatically consider the different learning paces of novel and seen classes and assign different thresholds, which also questions the effectiveness of the proposed hierarchical design.\n\n\n3. The proposed conditional self-labeling aims to incorporate labeled data into the online clustering process. But its goal and function seem to overlap with the supervised loss, which is used in this work. How is the performance (1) without the supervised loss, (2) without the conditional part of self-labeling, (3) without both?\n\n4. Currently, the code is not available to verify the performance and ablation study.\n\n[1] Vaze, Sagar, Kai Han, Andrea Vedaldi, and Andrew Zisserman. \"Generalized category discovery.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7492-7501. 2022.\n\n\n[2] Wen, Xin, Bingchen Zhao, and Xiaojuan Qi. \"Parametric classification for generalized category discovery: A baseline study.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16590-16600. 2023.\n\n\n[3] Bai, Jianhong, Zuozhu Liu, Hualiang Wang, Ruizhe Chen, Lianrui Mu, Xiaomeng Li, Joey Tianyi Zhou, Yang Feng, Jian Wu, and Haoji Hu. \"Towards distribution-agnostic generalized category discovery.\" Advances in Neural Information Processing Systems 36 (2023): 58625-58647.\n\n\n[4] Wang, Yidong, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang et al. \"Freematch: Self-adaptive thresholding for semi-supervised learning.\" arXiv preprint arXiv:2205.07246 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful analysis, it has been instrumental in refining our research. We hope that the following responses can make our manuscript clearer and more persuasive.\n\n**[W1]**  GCD and its related works indeed focus on a setting similar to OwSSL. However, it's worth noting that GCD-related works typically employ pre-trained ViT-Base/16 as the backbone, which has over three times more parameters than our ResNet-50. It is unfair to compare these two types of methods. Here, we still compare those GCD-related works. **Table 1** of Reference PDF  shows that our method outperforms existing approaches in novel-class and all-class accuracy on ImageNet100, despite using more compact model. We do not compare with imbalanced GCD methods (BaCon [6]) since its experimental settings are different from ours.\n\n**Table 1: Comparion on GCD-related work: average accuracy on the ImageNet100 with 50% novel classes and 50% labeled data within seen classes.**\n| Method     | Ref           | Backbone  | Seen Acc  | Novel Acc | All Acc  |\n|------------|---------------|-----------|-------|-------|------|\n| GCD [1]        | (CVPR'22)     | ViT-B/16  | 91.8  | 63.8  | 72.7 |\n| SimGCD [2]    | (ICCV'23)     | ViT-B/16  | 93.1  | 77.9  | 83.9 |\n| InfoSieve [3] | (NeurIPS'23)  | ViT-B/16  | 84.9  | 78.3  | 80.5 |\n| CiPR [4]       | (TMLR'24)     | ViT-B/16  | 84.9  | 78.3  | 80.5 |\n| PromptCAL [5]  | (ICCV'23)     | ViT-B/16  | 92.7  | 78.3  | 83.1 |\n| OwMatch+   | OURS          | ResNet-50 | 91.5  | **79.6**  | **85.5** |\n\n**[W2]**\nThanks for pointing out our shortcomings in the ablation experiment. We conduct additional ablation studies comparing static, class-wise adaptive thresholding which is used in FreeMatch [7], and our OwAT thresholding techniques as shown in **Table 2** of Reference PDF. While FreeMatch has proven effective in closed-world Semi-SL, it faces challenges in open-world settings. The significant disparity in overall learning conditions between seen and novel classes, as illustrated in Figure 1(b) of our manuscript, can lead to unstable global thresholds. The class-wise adaptive approach based on that may exaggerate this issue, resulting in suboptimal performance. We implement hierarchical structure to mitigate the instability sourcing from distinct learning dynamics of seen and novel classes. \n\n**Table 2: Comparison of static, FreeMatch's and our OwAT thresholding techniques on CIFAR100 with novel class ratio of 50\\%**\n| Method        | Seen Acc | Novel Acc | All Acc |\n|---------------|----------|-----------|---------|\n| Stat thre - 0.7| 80.1     | 59.4      | 69.6    |\n| Stat thre - 0.8| 79.8     | 63.9      | 71.7    |\n| Stat thre - 0.9| 80.2     | 62.8      | 71.3    |\n| FreeMatch     | **81.0** | 60.5      | 70.6    |\n| OwMatch+      | 80.1     | **63.9**  | **71.9**|\n\n**[W3]** Appreciate your insightful question, our complementary results, presented in **Table 3** of Reference PDF, can demonstrate the following:\n* Without supervised loss: we observe a decrease in seen-class accuracy but maintain novel-class clustering performance.\n* Without incorporation of labeled data into clustering: seen-class accuracy remains high, but novel-class clustering accuracy declines.\n* Without both: the model collapses in seen-class accuracy due to the absence of label information in the objective function.\n\nOur objective integrates both components to achieve a balance between clustering and confidence. The supervised loss enhances seen-class accuracy through one-hot supervision, while the clustering objective with conditional self-labeling improves novel-class clustering accuracy by incorporating labeled data. This harmonious approach yields the best all-classes accuracy while roughly maintaining both seen-class accuracy and novel-class clustering accuracy. Additionally, the parameters that balance the significance of each loss term are missing in equation (10). We will rectify this accordingly.\n\n**Table 3: Comparison of different settings.**\n| Method                                  | Seen Acc | Novel Acc | All Acc |\n|-----------------------------------------|----------|-----------|---------|\n| OwMatch+                                | 80.1     | 63.9      | 71.9    |\n| w/o supervised loss                     | 76.8     | 64.4      | 70.6    |\n| w/o supervision within clustering loss  | 80.3     | 61.2      | 70.7    |\n| w/o both                                | 0.62     | 45.1      | 40.0    |\n\n**[W4]** \n Under the NeurIPS's regulation, we are unable to share the code at this stage. However, we will make the code publicly available upon acceptance of the paper.\n\n[1] Vaze, Sagar, et al. \"Generalized category discovery.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Wen, Xin, Bingchen Zhao, and Xiaojuan Qi. \"Parametric classification for generalized category discovery: A baseline study.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Rastegar, Sarah, Hazel Doughty, and Cees Snoek. \"Learn to categorize or categorize to learn? self-coding for generalized category discovery.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Hao, Shaozhe, Kai Han, and Kwan-Yee K. Wong. \"Cipr: An efficient framework with cross-instance positive relations for generalized category discovery.\" arXiv preprint arXiv:2304.06928 (2023).\n\n[5] Zhang, Sheng, et al. \"Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[6] Wen, Xin, Bingchen Zhao, and Xiaojuan Qi. \"Parametric classification for generalized category discovery: A baseline study.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16590-16600. 2023.\n\n[7] Wang, Yidong, et al. \"Freematch: Self-adaptive thresholding for semi-supervised learning.\" arXiv preprint arXiv:2205.07246 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xF4CT4jRvd", "reviewer": "Reviewer_9vSt", "summary": "This paper proposes OxMatch, a semi-supervised learning (SSL) algorithm for an open-world setup where unlabelled data might come from outside of the labeled class distribution. The authors combine consistency regularization and self-labeling techniques to address the main challenges in open-world SSL (OwSSL). In particular, they use a conditional self-labeling approach to improve the label assignment stage (i.e. computation of pseudo-labels) and a hierarchical thresholding scheme to weight the contribution of different samples depending on the state of the model (i.e. the rate at which each class is being learned). Additionally, the authors provide a theoretical analysis of the conditional self-labeling estimator.", "questions": "None", "limitations": "The authors already mention the main limitations fo the proposed approach: the method assumes that the class distribution is known a priori.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "- The paper is theoretically sound and the authors properly justify the proposed elements.\n- The experimental section is complete and provides insights into the contribution of the different elements of the model.", "weaknesses": "- The presentation and phrasing of some paper parts should be addressed. The explanation of the method is slightly confusing and difficult to follow at some parts. \n- The paper addresses confirmation bias but this is not introduced at any point. Similarly, the text refers to Figure 1 as an example of the mitigation of confirmation bias (line 48) but this is not explained or elaborated. I would suggest addressing this in the main text and writing a more comprehensive caption for Figure 1.\n- Line 138 refers to Figure 3 for an illustration of issues with self-labeling and confirmation bias. This is not clear. Whether this refers to Figure 1 or Figure 3, it needs some more explanation. \n- In general, it is a good practice to write self-contained captions for figures. These should explain the elements in the figure and highlight their relationship with the main text. \n- Captions of tables should also be reviewed to include details of the metrics being displayed in the table. for better readability I would suggest highlighting the relevant results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. With your help, our revised manuscript is now clearer and more readable.\n\n**[W1;4;5]** We accept your suggestions regarding improving the alignment between the chart explanations and the text and providing more detailed legends and scales in our figures. We will make the necessary revisions to address these concerns.\n\n**[W2]** We regret that the absence of a clear definition for \"confirmation bias\" may have confused readers. We will include a detailed explanation at the start of our revised document. In our context, confirmation bias means that the model is biased because it has been exposed only to instances from seen classes. During the prediction phase, a model with confirmation bias may predict an ambiguous sample as belonging to a seen class, even if it is likely from a novel class. This results in high recall but low precision for seen classes.\n\n**[W3]** We apologize for the error; the correct reference should be Figure 1(a). We will add further illustrations to clarify this figure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LpxmjvLsWt", "reviewer": "Reviewer_cbPu", "summary": "The paper introduces OwMatch, a novel method for open-world semi-supervised learning (SSL). This approach incorporates conditional self-labeling and open-world hierarchical thresholding. Additionally, the paper provides theoretical analyses that demonstrate the unbiasedness and reliability of the label assignment estimator. Rigorous experimentation confirms that the proposed method achieves a significant improvement in performance.", "questions": "See weakness", "limitations": "No", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper provides adequate theoretical analyses for the proposed novel method.\n\n2. The paper is well-presented with clear organization throughout.\n\n3. The experiments are adequate and achieve better performance compared to existing methods.", "weaknesses": "1. The paper exhibits limited innovation. The two strategies proposed—Conditional Self-labeling and Open-world Hierarchical Thresholding—are commonly employed in existing OwSSL (Open-world Semi-Supervised Learning) and GCD (Generalized Category Discovery) methodologies. Specifically, the Conditional Self-labeling strategy mirrors the optimal transportation label assignment method detailed in references [1-4], and the Hierarchical Thresholding closely resembles the Momentum Prior Update described in references [3-4].\n\n2. The method assumes that the number of unknown classes is pre-determined and fails to address the identification of unknown classes. The estimation of the number of unknown classes is a critical issue in both OwSSL and GCD fields.\n\n3. The comparison with other algorithms is insufficient. Recent publications such as OpenLDN, TRSSL, and OpenCon from 2022 are included; however, more recent methods, particularly those utilizing pretrained ViT (Vision Transformer) architectures, should also be compared. Additionally, comparisons with methods addressing imbalance GCD are necessary.\n\n\n[1] Towards Realistic Semi-Supervised Learning\n\n[2]  OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning\n\n[3]  ImbaGCD: Imbalanced Generalized Category Discovery\n\n[4]  Bootstrap Your Own Prior: Towards Distribution-Agnostic Novel Class Discovery", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and valuable suggestions. They play a crucial role in improving our manuscript. \n\n**[W1]** We acknowledge the ubiquity of self-labeling (SL) and adaptive thresholding (AT) techniques, and we drew inspiration from the seminal contributions of TRSSL [6], OpenLDN [7], and BYOP [8]. We propose simple, intuitive, and effective modifications - transforming SL into **conditional** SL and AT into **hierarchical** AT - to ensure better compatibility with considered settings, thereby enhancing state-of-the-art performance. Our concepts of **conditional** and **hierarchical** are different from existing works in OwSSL. Specifically, by incorporating label data into the online clustering process, we can theoretically ensure optimal-transport-based optimized label assignment unbiasedness and lower variance in test statistics. On the other hand, the hierarchical design stems from the empirical observation of significant disparity in overall learning conditions between seen and novel classes. Although simple, this design is effective in mitigating the instability arising from the distinct learning dynamics of seen and novel classes. Extensive experiments validate the effectiveness of every modification. Moreover, to the best of our knowledge, this is the first work proposing the expectation of chi-square statistics (ECS) to evaluate the reliability of label assignment estimation.\n\nThanks for your professional concerns. We will put more emphasis on how our approach differs from foundational techniques in subsequent manuscripts.\n\n**[W2]** Thanks for highlighting this important aspect. We conducted additional experiments mainly following the approaches of GCD and TRSSL. Specifically, to estimate the number of classes, $k$-means clustering is performed on representations of the entire dataset from pre-trained ViT-B/16. The optimal value of $k$ is determined by evaluating the clustering accuracy on the labeled samples calculated by the Hungarian algorithm. This accuracy serves as a scoring function, optimized using Brent's algorithm to find the $k$ that maximizes performance on the labeled data. The estimation result is shown in **Table 4** of Reference PDF, which illustrates that estimation comes close to the ground truth. We also evaluate the method's sensitivity to the class estimation error. As illustrated in **Figure 1** of Reference PDF, our method still achieves reasonable performance over a larger range of errors.\n\n**Table 4: Estimation of the number of novel classes.**\n|              | CIFAR10 | CIFAR100 | ImageNet-100 |\n|--------------|---------|----------|--------------|\n| Ground Truth | 10      | 100      | 100          |\n| Estimation   | 10      | 104      | 112          |\n| Error        | 0%      | 4%       | 12%          |\n\n**[W3]** Here, we compare recent GCD-related works. **Table 1** of Reference PDF shows that our method outperforms existing approaches in novel-class and all-class accuracy on ImageNet100.  It's worth noting that GCD-related works typically employ pre-trained ViT-Base/16 as the backbone, which has over three times more parameters than our ResNet-50.  Despite a more compact model, our approach achieves superior performance. We do not compare with imbalanced GCD methods since their experimental settings are different from ours. \n\n**Table 1: Comparison on GCD-related work: average accuracy on the ImageNet100 with 50% novel classes and 50% labeled data within seen classes.**\n| Method     | Ref           | Backbone  | Seen Acc  | Novel Acc | All Acc  |\n|------------|---------------|-----------|-------|-------|------|\n| GCD [1]       | (CVPR'22)     | ViT-B/16  | 91.8  | 63.8  | 72.7 |\n| SimGCD [2]    | (ICCV'23)     | ViT-B/16  | 93.1  | 77.9  | 83.9 |\n| InfoSieve [3]  | (NeurIPS'23)  | ViT-B/16  | 84.9  | 78.3  | 80.5 |\n| CiPR [4]      | (TMLR'24)     | ViT-B/16  | 84.9  | 78.3  | 80.5 |\n| PromptCAL [5]  | (ICCV'23)     | ViT-B/16  | 92.7  | 78.3  | 83.1 |\n| OwMatch+   | OURS          | ResNet-50 | 91.5  | **79.6**  | **85.5** |\n\n[1] Vaze, Sagar, et al. \"Generalized category discovery.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Wen, Xin, Bingchen Zhao, and Xiaojuan Qi. \"Parametric classification for generalized category discovery: A baseline study.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Rastegar, Sarah, Hazel Doughty, and Cees Snoek. \"Learn to categorize or categorize to learn? self-coding for generalized category discovery.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[4] Hao, Shaozhe, Kai Han, and Kwan-Yee K. Wong. \"Cipr: An efficient framework with cross-instance positive relations for generalized category discovery.\" arXiv preprint arXiv:2304.06928 (2023).\n\n[5] Zhang, Sheng, et al. \"Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[6] Rizve, Mamshad Nayeem, Navid Kardan, and Mubarak Shah. \"Towards realistic semi-supervised learning.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2022.\n\n[7] Rizve, Mamshad Nayeem, et al. \"Openldn: Learning to discover novel classes for open-world semi-supervised learning.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2022.\n\n[8] Yang, Muli, et al. \"Bootstrap your own prior: Towards distribution-agnostic novel class discovery.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers' valuable feedback, which has significantly helped us improve our paper. Below, we provide detailed responses to each reviewer. Additionally, we've included all required tables and figures in a one-page Reference PDF.

Meta-review: This paper investigates the open-world semi-supervised learning problem (OwSSL). The authors propose a novel approach that combines self-labeling and open-world hierarchical thresholding to address the challenges in OwSSL. Both the empirical and theoretical results demonstrate the effectiveness of the proposed method, making this paper technically solid. Major concerns of reviewers were addressed well during the rebuttal, and thereby all reviewers agree to accept it for publication. The authors should add the additional modifications and results to the final version of the paper.
Decision: Accept (poster)