Supervised Knowledge May Hurt Novel Class Discovery Performance
Authors: ZIYUN LI
Jona Otholt
Ben Dai
Di Hu
Christoph Meinel
Haojin Yang
Venue: Accepted by TMLR ()

Abstract: Novel class discovery (NCD) aims to infer novel categories in an unlabeled dataset by leveraging prior knowledge of a labeled set comprising disjoint but related classes. Given that most existing literature focuses primarily on utilizing supervised knowledge from a labeled set at the methodology level, this paper considers the question: Is supervised knowledge always helpful at different levels of semantic relevance? To proceed, we first establish a novel metric, so-called transfer leakage, to measure the semantic similarity between labeled/unlabeled datasets. To show the validity of the proposed metric, we build up a large-scale benchmark with various degrees of semantic similarities between labeled/unlabeled datasets on ImageNet by leveraging its hierarachical class structure. The results based on the proposed benchmark show that the proposed transfer leakage is in line with the hierarachical class structure; and that NCD performance is consistent with the semantic similarities (measured by the proposed metric). Next, by using the proposed transfer leakage, we conduct various empirical experiments with different levels of semantic similarity, yielding that supervised knowledge may hurt NCD performance. Specifically, using supervised information from a low-similarity labeled set may lead to a suboptimal result as compared to using pure self-supervised knowledge. These results reveal the inadequacy of the existing NCD literature which usually assumes that supervised knowledge is beneficial. Finally, we develop a pseudo-version of the transfer leakage as a practical reference to decide if supervised knowledge should be used in NCD. Its effectiveness is supported by our empirical studies, which show that the pseudo transfer leakage (with or without supervised knowledge) is consistent with the corresponding accuracy based on various datasets.

Published in Transactions on Machine Learning Research (04/2023)
Supervised Knowledge May Hurt Novel Class Discovery Per- formance
Ziyun Li
ziyun. de
Hosso Plattner Institute University of Polsdam
Jona Otholt
jona.Othols Otholt Chpii de
Hasso Plattner Institute University of Potsdam
Ben D
bendai@cuhk.. edu.hk
Department of Statistics Chinese University of Hong Kong
Di Hu
dihu@ruc.ed edu.co
Gaoling School of Artificial Intellggecce Renmin University of China
Christoph Meinel
christoph.mmcinel@hpi.dd
Hosso Plattner Institute Universily of Polsdam
Haojin Yang"
haojinnyang pang@&hpp..de de
Hosso Plattner Institute
University of Poisdam
Reviewed on OpenReview: https: // openreview. net/ forum? id- ogOBTo5ukD
Abstract
Novel class discovery (NCD) aims infer novel categories in an unlabeled dataset by leveraging prior knowledge of a labeled set comprising disjoint but related classes. Given that most existing literature focuses primarily on utilizing supervised knowledge from labeled set at the methodology level, this paper considers the question: Is supervised knowledge always helpful al different levels of semanlic relevance? To proceed, we first establish novel metric, so-called trunsfer flow, to measure the semantic similarity between labeled/ labeled/unlabell /unllaeledd datasots. To show the validity of the proposed metric, we build up large-scale benchmark with various degrees of semantic similarities between labeled/ cabeled/unlateee uulabbledd datasets on ImageNet by leveraging its hierarchical class structure. The results based the proposed benchmark show that the proposed transfer flow is in line with the hierarchical class structure; and that NCD performance is consistent with the semantic similarities (measured by the proposed metric). Next, by using the proposed transfer flow, we conduct various empirical experiments with different levels of semantic similarity, yiclding that supervised knowledge may hurt NCD performance. Specifically, using supervised information from low-similarity labeled set may lead to suboptimal result as compared using pure self-supervised knowledge. These results reveal the inadequacy of the existing NCD literature which usually assumes that supervised knowledge is beneficial. Finally, we develop a pseudo-version of the transfer flow as practical reference to decide if supervised knowledge should be used in NCD. Its
*Corresponding author

---

Published in Transactions on Machine Learning Research (04/2023)
effectiveness is supported by our empirical studies, which show that the pseudo transfer flow (with or without supervised knowledge) is consistent with the corresponding accuracy based on various datasets. Code is released at htttp:///github..co//--//// -urrtKK-HHHNI
1 Introduction
The combination of data, algorithms, and computing power has resulted in a boom in the field of artificial intelligence, particularly supervised learning with its large number of powerful deep models. These deep models are capable of properly identifying and clustering classes that are present in the training set (i.e., known/seen classes), matching or surpassing human performance. However, they lack reliable extrapolation capacity when confronted with novel classes (i.e., unseen classes) while humans can easily recognize the novel categories. A classic illustration is how effortlessly a person can readily discriminate (cluster) unseen but similar vehicles (e.g,, trains and cars) based on prior experience. This motivated researchers to develop a challenge termed novel class discovery (NCD) (Han et al.) 2019 Chi et al. 2022 Han et al. 2021 Zhong ct al. 2021a), with the goal of discovering novel classes in an unlabeled dataset. by leveraging knowledge from a labeled set, which contains related but disjoint classes.
Recently, the majority of works NCD implicitly assumes that more data is better, and devotes to designing and developing neural networks to better utilize the supervised knowledge contained in the labeled set. For example, DTC Han et al. (011)), RS Han et al. (2021 2020) and RSMKD Zhao & Han (2021) ) transfer supervised knowledge by pretraining the model on the labeled data. OpenMix Zhoog et al. (2021b) mixes supervised knowledge from the labeled set and unsupervised knowledge from the unlabeled set to learn a joint
label distribution. UNO Fini et al. (2021) | combines supervised knowledge in unified objective function.
Despite their remarkable performance, there is less in-depth analysis of the supervised knowledge from the labeled set itself. Therefore, in this work, we investigate fundamental question of NCD: is supervised knowledge always helpful? Clearly, this question is naturally associated with the discrepancy between labeled and unlabeled sets. As mentioned by Chi et al. (2022). NCD is theoretically solvable when labeled and unlabeled sets share high-level semantic features. However, no quantitative analysis of semantic similarity was proposed. In this paper, we hypothesize that supervised knowledge would be beneficial when labeled and unlabeled sets share high degree of semantic similarity but may be less beneficial or even harmful when semantic similarity is low. To examine the hypothesis, we first propose a quantitative metrie, transfer flow, to mcasure the semantic label similarity of the two datasets. Specifically, it quantifies the discriminative information in the labeled set leaked in the unlabeled set, i.e., how much information we can leverage from the labeled dataset to help improve its performance. More details are provided in Section 3
To demonstrate the validity of transfer flow, we establish a new NCD benchmark with multiple semantic similarity levels. Specifically, the new benchmark is constructed based on a large-scale dataset, ImageNet (Dngg et al. 2009), by leveraging its hierarchical semantic information. It includes three difficulty levels, high, medium, and low semantic similarity, where each difficulty level includes two data scttings. Based on the benchmark, our experiments confirm that semantic similarity is positively related with NCD performance multiple pairs of labeled and unlabeled sets with varying semantic similarity under multiple baselines (Han et al. 2019 2021 Fini et al.) [021] MacQueen et al.] [166).. Also, a mutual validation between the proposed metric and the benchmark is conducted, which reveals that the transfer flow corresponds strongly with NCD performance. Detailed information on the benchmark can be found in Section
Based on the proposed metric and benchmark, we then analyze our core research question. Our experiments are conducted on the current state-of-the-art (SOTA) approaches Finn et al. 2021 Zhong et al. 2021a] Han et al. 2021), comparing the NCD performance with (ie., standard NCD) without supervised information. Unexpectedly but reasonably, we observe that the latter can outperform the former in cases where the sermantic similarity between the labeled and unlabeled sets is low, in contrast, to the commonly held assumption that supervised knowledge (or more data) can improve NCD performance, As a by-product, this also raises a question of whether to use supervised or just self-supervised knowledge for a given dataset. To address this issue, we provide two practical solutions. (i) A data selection solution. We develop a pseudo-version of the proposed metric, namely pseudo transfer flow, as a practical metric to infer the flow of supervised knowledge
2

---

Published in Transactions on Machine Learning Research (04/2023)
and/or self-supervised knowledge. Thus, it serves as instructive reference to decide what sort of data we intend to employ. (ii) A data combining solution. We develop new model which smoothly combines supervised and self-supervised knowledge from the labeled set and achieves 3% and 5% improvement in both CIFAR100 Krizhovsky 2009) and ImageNet compared to SOTA. More information can be found in Section
We summarize our contributions as follows:
We find that using supervised knowledge from the labeled set may lead to suboptimal performance low semantic NCD datasets. Based on this finding, we propose two practical methods and achieve ~3% and ~5% improvement in both CIFAR100 and ImageNet compared to SOTA.
We introduce theoretically reliable metric to measure the semantic similarity between labeled and unlabeled sets. A mutual validation is conducted between the proposed metric and a benchmark, which suggests that the proposed metric strongly agrees with NCD performance.
We establish a comprehensive benchmark with varying degrees of difficulty based on ImageNet by leveraging its hierarchical semantic similarity. Besides, we empirically confirm that semantic similarity indeed a significant factor influencing NCD performance.
2
Related Work
Novel class discovery (NCD) is relatively new problem proposed in recent years, aiming to discover novel classes (i.e., assign them to several clusters) by making use of similar but different known classes. Unlike unsupervised learning, NCD also requires labeled known-class data to help cluster novel-class data. NCD is first formalized in DTC (Han et al. 2019), but the study of NCD can be datod back to carlier works, such as KCL (Hsu et al., 2018) and MCL
(Hsu el al. 2019). Both of these methods are designed for general lask transfer learning and connect two models trained with labeled data and unlabeled data, respectively. In contrast, DTC first learns a data embodding on the labeled data with metric learning, then employs deep embedded clustering method based on Xie et al.
2016) to cluster the novel-class data.
More recent works, such as RS (Han et al. 2021 2020 and RSMKD (Zhao & Han 2021), , use self-supervised learning to boost feature extraction and use the learned features to obtain pairwise similarity estimates. Additionally, Zhao & Han [2021] improve RS by using information from both local and global views, as well as mutual knowledge distillation to promote information exchange and agreement. NCL. (Zhong et al., 2021ah extracts and aggregates the pairwise pseudo-labels for the unlabeled data via contrastive learning and generates hard negatives by mixing the labeled and unlabeled data in feature space. This idea of mixing data is also used in OpenMix (Zhong et al. 2021b), which mixes known-class and novel-class data to learn a joint label distribution. The current state-of-the-art, UNO (Fini et al. 2021 combines pseudo-labels with ground-truth labels in a unified objective function that enables better use of synergies between labeled and unlabeled data without requiring self-supervised pretraining. The most related one to our work is Meta discovery (Chi et al., 2022), which demonstrated the solvability of NCD by showing high-level semantic similarities between known and unknown classes. However, they lacked quantitative analysis, which we addressed by introducing metric for quantifying semantic similarity. In addition, Chi et al. (2022) concentrate developing method for scenarios with limited novel class data, while our study focuses on the standard NCD setting.
3 Quantifying Semantic Similarity
In this section, we present a novel metric for measuring the semantic similarity between labeled and unlabeled sets.
3.1 NCD Framework
We denote (X,, Yi) and (X,, Y.) as random samples under the and Qx,y, respectively. X; EC IRd and X,, Xx, Rdd are the labeled /unlaeledd feature vectors, Vi E Ci and
probability measures Fx,y
3

---

Published in Transactions on Machine Learning Research (04/2023)
Yu Cu are the true labels of labeled/unlabeler data, where C and C. are the label sets under the labeled and unlabeled probability measures x and Qx,y, respectively. Given a labeled set L. = (Xt,s, Y1,i) independently drawn from the labeled probability measure Ix,,, and an unlabeled dataset Um = (X 1=1 independently drawn from the unlabeled probability measure Lxx.. our primary goal to predict Ya, given Xu, where uu, the label of the i-th unlabeled sample x.,-- We now give a general definition of NCD.
Definition 1 (Novel Class Discovery) Let Px,Y, v. be a labeled probability measure on Xr x Ct, and Qx be on unlabeled probability measure on Xy x Cu. with C. nc =0. Given a labeled dataset Cn sampled from , and an unlabeled dalasel Um sampled from Qx novel class discovery aims to predict the label Y, of each unlabeled instance Xa given Cn and Um.
3.2 Transfer Flow
For further investigating the effectiveness of supervised knowledge NCD, we propose a quantitative metric, namely transfer flow, to assess semantic similarity between labeled labeled/unlabeled /unlabeled datasets. To the best of our knowledge, the question of how to measure the semantic similarity between the labeled and unlabeled sets in NCD remains unsolved.
To proceed, we begin with introducing Maximum Mean Discrepancy (MMD; Gretton et al. 2012 which is used to measure the discrepancy of two distributions. For example, MMD of two random variables ~ Pz and ~ PPzz defined as:
MMD* AMDx(PZ,, (Pz, Fz:): == sup
(E(M(Z)) - -B(ZZ)))
where H. is class of functions h Ny R, which is specified as reproducing kernel Hilbert Space (RKHS) associated with contimuous kernel function K(.,-). From (1 when MMD is large, the distributions between Z and Z' appear dissimilar.
In NCD, the unlabeled dataset predicted by taking the information from the conditional probability Pyix, (usually presented by a pretrained neural network) of labelod dataset. For example, if the distributions of Pyxx==x, under Y. = e and Yu = are significantly different, then its overall distribution discrepancy is large, yielding that more information can be leveraged in NCD. On this ground, the discrepancy of the labeled probability measure Py,xx on X,, under the unlabeled probability measure Q namely transfer flow.
MMD to quantify
Definition 2 (Transfer Flow) The transfer flow of NCD prediction under Q based on the labeled conditional probability Py/px: is
T2F Ba (MMD) (Qp(x.)).. (MMD) Ro(xw)r?)
where (X,: Yu), (X,,, Y.) ~ Q are independent copies, the expectation Eg is taken with respect to Y, and Y! under Q, and p(x) the conditional probability under Pyzx, on an unlabeled data X., = x, defined as
= p(x) = c X1 = x))
To summarize, transfer flow measures the overall discrepancy of pXx)) under different new classes of the unlabeled measure Q, which indicates the informative flow from P to Q. Note that the metric intrinsically quantifies the information based on data, and independent with NCD methods. Lemma α shows the lower and upper bounds of transfer flow, and provides theoretical justification of its effectiveness in measuring the similarity between labeled and unlabeled datasets.
Lemma 1 = maxcec. Eo( p(X.X = < xx, then 0 S T- T-Flow QQ,P) S 4x². Moreover, T- T-Flow(Q P) = 0 if and only if Yn is independent with p(X.) that is, for any E C.:
Q(Y,= = p(X))) = Q(Y, = =),
(3)
yielding that p(..) is useless in NCD Q.
4

---

Published in Transactions on Machine Learning Research (04/2023)
Note that can be explicitly computed for many common used kernels, for example, for Gaussian Laplacian kernel.
From Lemma D T-Flow(Q, P) = 0 is equivalent to Yu is independent with p(X.), which matches our intuition of no flow. Alternatively, Yu dependent with p(X.), we justifiably believe that the information of X can be used to facilitate NCD, Lemma [ tells that T-Flow(Q, > in this case. Therefore, Lemma reasonably suggests that the proposed transfer flow is an effective metric to detect if the supervised information in IP is useful to NCD on Q
Next, we give finite sample estimate of transfer flow. To proceed, we first rewrite transfer flow follows.
T-Flow(Q,P)
Σ (20%= =c)
(Qpox.IM ( (o(x)113=o)). -
(4)
e,c'Euuiett
where the equality follows from the fact that MMD3 (Qp(x.)). (p(X.)M = 0.
Given an estimated probability Prix, and an evaluation dataset (Xu,is : under Q, we assess XN,i on Prxx, as p(x,,)) = (P(Y)- =||x, = x).e)) then the empirical transfer flow is computed as:
[Z m(m 11
T-Flow(Q2 P)=
E
MMDH ( (p(x)Mx=d))
(5)
c, Ecuuece
2 where In,c = (1 (1sism m: Vu, [ c) is the index set of unlabeled data with yo.i = and MMDx is defined as:
MMD (2- (pxx.))) =c ((x))) ((=x));=o))
2
Σ
((f(x.,)
Iunel - 1)
(JEIa.citf
Σ
K ((((xn)); p(xus))
Σ
((p(x.,),,((x
Iueer ([Iu,e 1)
I IEIn.e JEI
IJEI
Remark 1 Note that the definition of the proposed transfer flow in Definition is based on the conditional probabilily pxxu) from P. Yel, can be extended to more general represenlation s(x.) estimated based on supervised or self-suu supervised information from labeled dataset. See the definition of pseudo transfer flow as follows.
Furthermore, we define pseudo transfer flow, which is computed by a pretrained representation S(x) and pseudo-label obtained from clustering method applied to the representations (e.g., K-means, GMM, agglomerative, etc.). It is worth mentioning that pseudo transfer flow is more practical than transfer flow because that transfer flow requires the estimated probabilitics and the true label Y,, while pseudo transfer flow can apply to any pretrained representation and any pseudo label obtained from a clustering method.
Definition 3 (Pseudo Transfer Flow) The pseudo transfer flow of NCD prediction under Q based on pretrained representation ((x) is
Z...llZuncl m(m - -1) MMDx (QA Po(k:))
Pseudo- Sheudo-T-Flone F-Flow P)
Σ
(6)
c,e Elusctte
where In.c = (1 Vo.i cf is the indez set of unlabeled data with Tu,i c, Vu.s is provided based clustering method their representations S(xu.), and 8(xu.)) is the representation estimated from supercised model or a self-supervised model. MMD defoned as in Section 3.2.
4 Benchmark
To examine the validity of the proposed metric, we first construct benchmark with various degrees of semantic similarity by using the hierarchical structure in ImageNet. We design two groups of numerical
5

---

Published in Transactions on Machine Learning Research (04/2023)
Fruit
Flower
Pear
Pineupple
Stawherry
Carnation
Orchid
Sunflower
Jackfrrut
Granny smith
Acom
Lotos
Unlabeled set
Labeled set
Labeled set 1.5
Labeled
Unlabeled
High similarity
Medium similarity
Low similarity
Figure 1: Illustration of how we construct the benchmark with varying levels of semantic similarity. Unlabeled set Un and labeled set L, are from the same superclass (fruit), whereas unlabeled set U2 and labeled set L2 belong another superclass (flower). Labeled set L1S composed of half of L1 and half of Lg. If both the labeled and unlabeled classes are derived from the same superclass, i.e., (U1, I1) and (U2, T2); we consider them a high semantic similarity split. In contrast, (U1, L2) and (U2, L1) are low semantic similarity splits, since the labeled and unlabeled classes are derived from distinct superclasses. In addition, we consider (U1, L1.5) and (U2, L1.5) to have medium semantic similarity because half of L1.5 share the same superclass as U1.
experiments: the consistency between NCD difficulty and degrees of semantic similarity (implemented by the hicrarchical label structure) in Section 4.2.1 and the consistency between degrees of semantic similarity and the proposed (pseudo) transfer flow in Section 4.2.2
4.1 Construction Principle
Unlike existing benchmarks, which only take into account the difficulty of NCD based on the labeled set in terms of the number of categories (e.g., Fini et al. (2021)) or the number of images in each category (e.g., Chi et al. (2022) ) we argue that semantic similarity is also a significant factor inffuencing NCD performance.
Specifically, our proposed benchmark based on the ENTITY-30 task (Santurkar et al. 2020). which contains 240 ImageNet classes in total, with 30 superclasses and 8 subclasses for each superclass. Three different semantic similarity levels (high, modium and low) are produced by leveraging ImageNet's underlying hierarchy. For example, shown in Figure 1, despite the fact that the labeled (e.g., pineapple, strawberry) and unlabeled (e.g., pear, jackfruit) classes are disjoint, if they derive from the same superclass (i.e., fruit), they have a higher degree of semantic similarity. Conversely, when labeled (e.g,, rose, lotus) and unlabeled (e.g., pear, jackfruit) classes are derived from distinct superclasses (i.e., labeled classes from flower while unlabeled classes from fruit), they are further apart semantically.
As consequence, we define three labeled sets L1, L155, La and two unlabeled sets U1, U2. The sets Lı and Un are selected from the first 15 superclasses, where 6 subclasses of each superclass assigned to L1, and the other 2 are assigned to U1. The sets L2 and U2 are created from the second 15 superclasses in similar fashion. Finally, LLS is created by taking classes half from L₁ and half from L2. As a consequence, (U1, L1)/UU,, L2) are closely related semantically (high), whereas (U1, L2)/(U,, L1) are far apart (low), with (U1, (U2, L1.s) in-between (medium). Additionally, we also provide four data settings on CIFAR100, two high-similarity settings and two low-similarity settings, by leveraging the hicrarchical class structure of CIFAR100 similarly. Each case has labeled classes and 10 unlabeled classes. A full list of the labeled and unlabeled sets can be found in Appendix F

---

Published in Transactions on Machine Learning Research (04/2023)
Table 1: Comparison of different combinations of labeled sets and unlabeled sets consisting subsets of CIFAR100. The unlabeled sets are denoted U, and U2 while the labeled sets are called L and L2- Un and L1 share the same set of superclasses, similar for U2 and L2- Thus, the pairs (U1, L1) and (U2, L2) are close semantically, but (U1, L2] and (U2, I1) are far apart. We report the mean and standard deviation of the clustering accuracy across 10 runs for multiple NCD methods. The higher mean is bolded.
Unlabeled set U₁
Unlabeled set U2
Methods
L high
L2 - low
L low
L2 high
K-means (MacQueen et al. 1967
61.0 + 1.1
37.7 = 0.6
33.9 + 0.5
55.4 + 0.6
DTC
(Han al. 2019
64.9 + 0.3
62.1 + 0.3
53.6 + 0.3
66.5 + 0.4
RS
(Han et al.) 2020
78.3 + 0.5
73.7 = 1.4
74.9 + 0.5
77.9 + 2.8
NCL
Zhong et al. 2021a
85.0 + 0.6
83.0 + 0.3
72.5 + 1.6
85.6 + 0.3
UNO
Fini et al. 2021
92.5 + 0.2
91.3 + 0.8
90.5
0.7
91.7 2.2
Table 2: Comparison of different combinations of labeled sets and unlabeled sets on our proposed benchmark.
Similar to the CIFAR-based experiments, L1 is closely related to U1 and L2 is highly related to U2. The third labeled set L1.5 is constructed from half of L1 and half of L2, so in terms of similarity it is between L1 and L22 For all splits we report the mean and the standard deviation of the clustering accuracy across 10 runs for multiple NCD methods. The higher mean is bolded.
Unlabeled set U1
Unlabeled set U2
Methods
L1 high
L1s medium
L2 low
L1 low
L1.5 medium
L2 high
K-means
41.1 0.4
30.2 + 0.4
23.3 + 0.2
21.2 + 0.2
29.8 0.4
45.0 + 0.4
DTC
43.3
1.2
35.6 + 1.3
32.2 + 0.8
21.3 + 1.2
15.3 1.5
29.0 + 0.8
RS
55.3 0.4
50.3 0.9
53.6 0.6
48.1 + 0.4
50.9 0.6
55.8 + 0.7
NCL
75.1
0.8
74.3 + 0.4
71.6 +
61.3 + 0.1
70.5 + 0.8
75.1 + 1.2
UNO
83.9 + 0.6
81.0 0.6
77.2 0.8
77.5 + 0.7
82.0 + 1.7
88.4 + 1.1
4.2
Experiments
4.2.1
Validating the Benchmark
Experimental Scttings To suggest the offectiveness the benchmark, we conduct experiments on 5 baselines, including K-means (MacQueen et al. 1967). DTC (Han et al. 2019), RS (Han et al., 2021), NCL
[Zhong et al., 2021a and UNO (Fini al. 2021 We follow the baselines regarding hyperparameters and implementation details.
Experimental Results In Table (CIFAR100), the gap between the high-similarity and the low-similarity settings is larger than 20% for K-means and reaches up to 12% for more advanced methods. Similarly, in Table 2 (ImageNet), the high-similarity settings generally obtain the best performance, followed by the medium and low settings. Under the unlabeled set U1. L1 achieves the highest accuracy, with around 17% improvement. compared to T2, and around - 11% improvement compared to L..5- For the unlabeled set U2, Le the most similar set and obtains 8 14% improvement compared to L1, and around 5 14% improvement compared to LI.5-
Conclusion Consistency between Semantic Similarity and Accuracy. The above numerical results suggest positive correlation between semantic similarity and NCD performance, which suggests that semantic similarity is significant factor influencing NCD performance.

---

Published in Transactions on Machine Learning Research (04/2023)
Table 3: Experiments on transfer flow. To obtain the standard deviation we recompute the transfer flow 10 times using bootstrap sampling. The results show that transfer flow is consistent with semantic similarity. The maximum transfer flow for the same unlabeled set highlighted in bolded.
Dataset
Unlabeled Set
Labeled Set
Transfer flow
L1 high
0.62 + 0.01
U1
L2 low
0.28 + 0.01
CIFAR100
L low
0.33 + 0.01
U2
L2 high
0.77 = 0.02
L1 high
0.71 + 0.01
U1
L1.5 medium
0.54 0.01
L2 low
0.36 + 0.01
ImageNet
L low
0.33 + 0.00
U2
L medium
0.50 + 0.01
L2 low
0.72 0.01
Table 4: Experiments on pseudo transfer flow under three clustering methods, i.e., K-means, GMM and agglomerative, each setting is repeated for 10 times. The maximum pseudo transfer flow is highlighted in bold for each baseline.
Unlabeled set U1
Unlabeled set U2
Method
L1 high
L1.5 medium
L2 low
L1 low
L1.5 medium
L2 high
K-means
1.23 + 0.03
1.02 + 0.03
0.99 + 0.02
0.96 + 0.01
0.99 + 0.03
1.24 + 0.02
GMM
0.79 + 0.01
0.69 + 0.02
0.56 + 0.02
0.58 + 0.02
0.68 + 0.04
0.91 + 0.02
Agglomerative
1.17 = 0.00
0.96 = 0.00
0.87 0.00
0.83 = 0.00
0.89 + 0.00
1.15 = 0.00
4.2.2 Validating (Pseudo) Transfer Flow
Experimental Settings We evaluate transfer flow and pseudo transfer flow on both CIFAR100 and our proposed ImageNet-based benchmark under different semanttcc similarity cases. We employ ResNet18 (He) et al. 2016) as the backbone for both datasets following Han et al. (2019 2021 Fini al. (2021). Known-class data and unknown-class data are selected based on semantic similarity, apply fully supervised learning to the labeled data for each dataset to obtain the pretrained model. Then, we feed the unlabeled data to the pretrained model to obtain its representation. Lastly, we calculate the transfer flow flow/pseudo pseudo transfer flow based on the pretrained model and the unlabeled samples' representation. Specifically, for pseudo transfer flow, we apply clustering mothods to generate the pscudo labels. For the first step, batch size is set to 512 for both datasels. We use an SGD optimizer with momentum 0.9, and weight, decay 1e-4. The learning rate is governed by a cosine annealing learning rate schedule with base learning rate of 0.1, a linear warmup of 10 epochs, and minimum learning rate of 0.001. We pretrain the backbone for 200/ 100 epochs for CIFAR-100/ImageNett
mentioned in Section 4 We first
Experimental Results and Conclusions (i) Consistency between (pseudo) transfer flow and semantic similarity, Table 3 demonstrates the transfer flow under different data settings. In CIFAR100, settings with high semantic similarity tend to have higher transfer flow than those with low semantic similarity. Similarly, in ImageNet, setlings with high semantic similarity have the highest transfer flow, followed by those with medium semantic similarity, while those with low semantic similarity have the lowest transfer flow. [Table 4 shows the pseudo transfer flow under various basclines and datasets. This result supports the conclusion above that, for given baseline, settings with higher semantic similarity tend to have higher pseudo transfer flow.
8

---

Published in Transactions on Machine Learning Research (04/2023)
Low similarity
Medium similarity
High similarity
UNO
NCL
K K-means
90
90
80
80
Accuracy in %
Accuracy in %
70
70
60
50
40
40
30
30
0.3
0.5
0.7
0.6
0.7
0.9
Transfer-flow
Pseudo Transfer-flow
Figure 2: The experiments investigate the correlation between accuracy and transfer pseudo transfer flow. We show three baselines, including UNO, NCT. and K-means. On the left, we measure the transfer flow and the clustering accuracy. On the right, we replace tramsfer flow with pseudo transfer flow calculated from GMM clustering. As expected, there is a positive correlation between accuracy and transfer flow as well as pseudo transfer flow, which shows that transfer flow and pseudo transfer flow can demonstrate the difficutyy of NCD tasks.
(ii) Consistency between (pseudo) transfer flow and accuracy. Figure 2 illustrates the relationship between semantic similarity, Imnsfer flow/ pscudo transfer flow, and NCD performance ImageNet, where the color corresponds to the same baseline. As expected, there is consistent positive correlation between transfer flow/ useudo transfer flow and NCD accuracy, supporting the validity of transfer flow ppeudoo transfer flow as a metric for quantifying semantic similarity and the difficulty of a particular NCD problem.
5 Supervised Knowledge May Hurt Performance
NCD is based on the idea that supervised knowledge from labeled data can be used to improve the clustering of unlabeled data. Yet, our empirical studies raise a possibility that supervised information from a labeled set may result in suboptimal outcomes compared to using exclusively self-supervised knowledge.
5.1 Empirical Experiments
We first conduct experiments in the following settings:
(a) X, + X1, using the images unlabeled set and the labeled set but without labols.
(b) X, + (X,, Yi,, using the unlabeled set and the whole labeled set, (i.e., standard NCD).
(c) X,, using the unlabeled set.
Particularly, for (a), even though we do not the labels, still extract the knowledge of the labeled set via self-supervised learning. By comparing (a) and (b), can analyze the performance gain from including supervised knowledge in the form of labels. For a better comparison, we also include part of experiments on (c) for estimating the total performance gain caused by adding the labeled set in Appendix Table 9). In (c), NCD is degenerated to unsupervised learning (i.e., clustering on X.).
Experimental Settings We conduct experiments based on recent NCD methods, including RS (Han et al., 2021), NCL (Zhong et al., (021aa) and UNO (Fini et al.) 2021) (the current state-of-tho-art method in NCD). To perform settings (a), we make adjustments to the framework of each method with as minimal modifications possible, enabling to run fully self-supervised. For UNO, we replace ground truth labels Ber in the
9

---

Published in Transactions on Machine Learning Research (04/2023)
labeled set with self-supervised pseudo labels Birr which are obtained by applying the Sinkhorn-Knopp algorithm (Cuturi 2013). For NCL. and RS, we replace the ground truth labels with labels obtained using K-means (MacQueen et al. 1967). More details on these modifications as well as the used hyperparameters can be found in Appendix B.2 We conduct experiments on our Imagenet-based benchmark, as describe in Section 4 where we define high, medium, and low similarity seltings based on the hierarchical structure of the dataset and we evaluate these scttings by transfer flow. Additionally, we conducted experiments on the CIFAR100 50-50, established by Fini et al. (2021), which randomly splits the CIFARI00 dataset (Krizhevsky 2009 into 50 labeled and 50 unlabeled classes without considering semantic similarity.
Experimental Results As shown in Table 5 by comparing (a) and (b) under different baselines, we have the following numerical results:
For RS, (a) outperforms (b) on all datasets, with 3% improvement in CIFAR100 and ~10 15% improvement in ImageNet.
For NCL, (b) exceeds (a) by 22.% in CIFAR1000 whereas in ImageNet, (a) surpasses (b) by ~2 8% in all semantic similarity settings.
For UNO, we discuss 3 scttings,
In high- similarity settings, (i.e., L1 U1), (a) performs 44% worse than (b).
In low similarity settings, (i.e., L2 U1 and Li U2), (a) outperforms (b) by ~3% 8%.
In medium- similarity settings, (i.e., L1.5 - U1 and L1.5 U2), neither (a) nor (b) has an absolute
advantage.
Conclusion Supervision information with low semantic relevance may hurt NCD performance. Based on the analysis of numerical results, we find that using self-supervised knowledge is significantly more advantageous than using supervised knowledge in both RS and NCL, while in UNO, supervised knowledge is beneficial when the labeled and unlabeled sets have a high degree of semantic similarity, but harmful when the semantic similarity is low.
Table 5: Comparison of using supervised knowledge or using exclusively self-supervised knowledge on CIFARI00 and our proposed benchmark. We present clustering mean and standard error on three recent methods, including RS, NCL and UNO (SOTA). Unexpectedly, X, + X, outperforms X, + (X,, Yi) in both RS and NCL on ImageNet. For UNO, in CIFAR100-50 and low similarity case of our benchmark, X,, + X; can also get greater performance than X,, + +X,, The empirical results indicate that supervised knowledge may have a negative impact on NCD performance. The higher mean is bolded.
Unlabeled set U1
Unlabeled set U2
Setting
CIFAR100-50
L1 high
LI.5 medium
L2 low
L1 low
LI.5 medium
L2 high
X. + X1
42.8 + 0.4
64.9 0.2
64.5 +
67.3 + 0.9
62.9
1.2
65.3
0.5
67.4 0.8
RS
X. + (X1, Yi)
39.2 + 1.0
55.3 + 0.4
50.3 + 0.9
53.6 = 0.6
48.1 + 0.4
50.9 0.6
55.8 + 0.7
X, + X,
50.9 + 0.4
77.3 + 0.4
75.2 + 0.5
75.9 + 0.6
77.3
+ 0.6
77.5 + 0.7
83.2 = 0.8
NCL
X. + (X1, Yi)
53.4 + 0.3
75.1 + 0.8
74.3 + 0.4
71.6 + 0.4 61.3 + 0.1
70.5 = 0.8
75.1 + 1.2
X, 4.+40 X1
64.1 + 0.4
79.6 + 1.1
79.7 + 1.0
80.3 + 0.3
85.3
+ 0.5
85.2 + 1.0
89.2 + 0.3
UNO
X. (X,, Yi)
62.2 + 0.2
83.9 + 0.6
81.0 + 0.6
77.2 + 0.8 77.5 + 0.7
82.0 1.7
88.4 + 1.1
5.2 Practical Applications
Table 5 indicates that supervised knowledge from the labeled set may cause harm rather than gain, is nature ask whether to utilize supervised knowledge with labeled data or pure self-supervised knowledge without labels. Therefore, we provide two instructive solutions, including a practical metric (i.e., pseudo transfer flow) and new method tuning the weighting between supervised and/or self-supervised knowledge.
10

---

Published in Transactions on Machine Learning Research (04/2023)
Table 6: Results showing the link between pseudo transfer flow (PTF) and accuracy on novel classes (ACC). The pseudo transfer flow is computed based either on a supervised (SL) or self-supervised model (SSL), using ResNet18 in both cases. The accuracy is obtained using the standard NCD setting (X,, (X,, Yi)) for supervised learning, and self-supervised NCD setting (X, X2) for self-supervised model. The higher mean value is presented in bold, while the results within standard deviation of the average accuracy are not bolded.
High similarity
Medium similarity
Low similarity
Model
L - U,
L - U2
L1.5 - -U
L - U2
L2 - U1
L - U2
SSL
0.96 + 0.01
0.96 + 0.02
1.14 + 0.02
1.19 + 0.01
1.05 = 0.03
1.25 + 0.03
PTF
SL
1.21 + 0.02
1.21 + 0.01
1.03 + 0.02
0.98 + 0.03
0.99 = 0.02
0.96 + 0.01
SSL
79.6 + 1.1
89.2 + 0.3
79.7 + 1.0
85.2 + 1.0
80.3 + 0.3
85.3 + 0.5
ACC
SL
83.9 + 0.6
88.4 + 1.1
81.0 0.6
82.0 + 1.7
77.2 0.8
77.5 + 0.7
5.2.1
Data Selection: Supervised or Self-supervised Knowledge?
We conduct the experiment to investigate the relationship between pseudo transfer flow and NCD's accuracy under various semantic similarity settings.
Experimental Scttings We first perform supervised lcarning and sclf-supervised learning to achiove two pretrained models under varying data sets based on UNO (the best performing method in Section respectively. Then, pseudo tronsfer flow is computed based on the two pretrained models.
Experimental Results In Table 6. PTF denotes pseudo transfer flow and ACC represents accuracy. We find that pseudo transfer flow is consistent with the accuracy under various datasets. For example, in L - -1, the pseudo transfer flow computed on the supervised model is larger than the one computed in the self-supervised model, which is consistent with the accuracy, where the supervised method outperforms the self-super one. Reversely, for L2 - U1, L1 U2 and L1.5 U1, the pseudo transfer flow computed on the self-supervised model is larger than the one computed the supervised model, which is again consistent with their relative performance. L2 U2 and L1.5 U are omitted as their performance falls within the standard deviation of the average accuracy.
The proposed pscudo transfer flow can be used as practical reference to infer what sort of dala we want to use in NCD, image-only informalion, X, + X, or image-label pairs, X, + (X,, Yi) of the labeled set.
Conclusion
5.2.2 Data Combining:
Weighting Supervised Knowledge
Rather than adopting binary approach of either fully utilizing or disregarding supervised knowledge, it is essential to determine the optimal amount of supervised knowledge to incorporate.
Method and Experimental Settings Specifically, we first utilize self-supervised learning for pretraining rather than using supervised pretraning as UNO. Then, similar to UNO, we improve image representations by contrastive learning framework (i.e., SwAV (Caron et al., 2020)). Different from UNO, we generate pscudo labels Bi., for labeled data utilizing Sinkhorn- Knopp algorithm (Cuturi 2013) and combine them with ils corresponding ground truth labels Mer. Then, the overall classification larget of the labeled data is Mt after + (1 - a)Birzz where E [0, 1] represents the weight of the supervised component. Specifically, when a = 1, our proposed method has the same target as UNO FFini et al. 2021), but the pretraining is different.
Experimental Results
From Figure by comparing supervised (dotted lines) and self-supervised pretraining (dashed lines), that in the high similarity setting, supervised pretraining is slightly better than self-
11

---

Published in Transactions on Machine Learning Research (04/2023)
SOTA (supervised pretraining)
SOTA w. self-supervised pretraining
85
85
85
Accuracy in %
80
80
80
75
0
0.25
0.5
0.75
0
0.25
0.5
0.75
0.25
0.5
0.75
1
a
a
(a) High similarity
(b) Medium similarity
(c) Low similarity
Figure 3: Experiments combining supervised and self-supervise objectives, where shows the weight of the supervised component. The experiments are carried out on our ImageNet-based benchmark with high, medium and low similarity settings, respectively. Dotted lines show the accuracy of the SOTA (UNO, using supervised pretraining) and dashed lines show the accuracy of SOTA when replacing the pretraining with self-supervised learning. In the low-similarity setting, a mix of supervised and self-supervieed objectives oulperforms either alone. Self-supervised pretraining purperfosss supervised pretraining low and medium similarity settings and is comparable in high similarity settings.
supervised pretraining. Insterestingly, in the medium similarity setting, self-supervised pretraining outperform supervised pretraining, and the advantage of self-supervisdd pretraining becomes more significant in the low similarity setting, with an improvement of approximately 2%.
As observed in the low similarity setting, NCD accuracy demonstrates increasing trend followed by a decreasing trend the utilization of supervised knowledge (a) rises, with an approximate enhancement of 2 - 3% when a is set to 0.25 in comparison to the fully supervised (a = 1) and exclusively self-supervised (a = 0) training, and ~ 5% improvement compared to SOTA. Specifically, pure self-supervised training (a = 0) surpasses fully supervised training (a = 1) with ~ 1% improvement. In contrast, in the high-similarity setting, NCD accuracy demonstrates an upward trend with the increase the level of supervised knowledge. However, in the medium similarity setting, the improvement is not substantial with an increase in supervised knowledge.
Conclusion
Supervised knowledge may lead inferior performance during both pretraining and training.
The effectiveness of incorporating supervised knowledge in the training process is closely related to the degree of similarity between the labeled and unlobeled datasets. Therefore, in order to mitigate the occurrence of negative transfer, the use of supervised knowledge in NCD should be considered with regard to the appropriate amount rather than being employed in naive manner.
6 Conclusion
We thoroughly investigate the effectiveness of supervised knowledge in the NCD task. We first introduce transfer pseudo transfer flow, metric for measuring the semantic similarity between two data sets. Then, we proposc comprchensive benchmark with varying levels of semantic similarity based on ImageNet for validating the proposed metrie and verify that semantic similarity has a significant impact. on NCD performance. By leveraging the metric and benchmark, we observe that supervised knowledge may lead to inferior performance in circumstances with low semantic similarity. Furthermore, we propose two practical applications: (i) pseudo transfer flow as reference on what sort of data we aim to use. (ii) weighting supervised knowledge, which obtains 55% improvement. under low similarity settings for ImageNet.
12

---

Published in Transactions on Machine Learning Research (04/2023)
7 Acknowledgement
This work was supported in part by RGC-ECS 24302422, and CUHK Faulty of Science direct grant. The authors are grateful to reviewers and the Action Editor for their insightful comments and suggestions which have improved the manuscript significantly.
References
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised lcarning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020.
Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Bo Han, Gang Niu, Mingyuan Zhou, and Masashi Sugiyama. Meta discovery: Learning to discover novel classes given very limited data. In International Conference on Learning Representations, 2022.
Marco Cuturi. Sinkhorn distances: lightspeed computation of optimal transport. In Proceedings of the 26th International Conference Neural Information Processing Systems, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition. IEEE, 2009.
Eurico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 2012.
Kai Han, Andrea Vodaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering, In Proceedings of the IEEE Conference Computer Vision and Pallera Recognition, 2019.
Kai Han, Sylvestre-Alvies Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. Tn International Conference Learning Representations, 2020.
Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In International Conference on Learning Representations, 2018.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without. multi-class labels. In International Conference on Learning Representations, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley Symposium on Mathematical Statistics and Probability. Oakland, CA, USA, 1967.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. International Conference on Learning Representations, 2020.
13

---

Published in Transactions on Machine Learning Research (04/2023)
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 2010.
Naftali Tishby, Fernando Pereira, and William Bialek. The information bottleneck method. arXiv preprint 2000.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning. PMLR, 2016.
Bingehen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. Conference Neurul Information Processing Systems, 2021.
Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021a.
Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, 2021b.
14

---

Published in Transactions on Machine Learning Research (04/2023)
A Details of Section 3
A.1 Technical Proofs
To proceed, we summarize all notations used in the paper in Table 7
Table 7: Notation used in the paper.
Notation
Description
X Xu.
labeled data / unlabeled data label of labeled data unlabeled data domain of labeled data / unlabeled data label set of labelod data unlabeled data probability measure of labeled data / unlabeled data labeled dataset unlabeled dataset reproducing kernel Hilbert space (RKHS) kernel function
Me. Bl
Xg X
C2 Cu
P. Q
L. (Xi,s, 11,1)s=1,. n
Um = )..,)).=,....
H
k(,-)
(X'Y")
independent copy of (X
P
estimated probability measure of labeled data expectation with respect to the probability measure Q the i-th unlabeled data
BQ
Xu,i Ma.b
Lu.c
index set of unlabeled samples labeled as yu,,
Proof of Lemma 1. We first show the upper bound of the transfer flow. According to Lemma 3 in Gretton
el al. 2012 we have
T-Flow(Q. (Q. P) (E(MMD"(Qpxx..11 (1oxxxXXIIXxxxxxxxx)))) Ee (Iueroxii "eprxtiing l12)
S max c..'EC. || Vuze - "C 11 max || vu=e 11
eEC,
4 max rEC.
Bc -) = c))
= 4 E2 (K((p(X.)) K )) = =c G. cc CEC,
S 4 mmxx Eq CECo
|| (I)
lr, = = -c)
= 4 max Re cECu
P(X (FXX) = =c) S An?
where MO ,. := (K((KK)) )(Y,) is the kernel mean embedding of the measure Qp(X.))I. Gretton et al. 2012), the second inequality follows from the triangle inequality in the Hilbert space, the fourth equality
follows from the fact that Eeg is a linear operator, the second last inequality follows from the Cauchy-Schwary incquality, and the last equality follows the reproducing property of K(..-).
Next, we show the if and only condition for T-Flow (Q. PP) = Assume that Q(Y, = c) > for all e E Cu. According to Theorem 5 in Gretton et al. (2012), we have
T-Flow(Q,PP) =0
Q(p(x) = =c)= for c E Cu, x E X
Note that
Q(p())X, = =)Q =
(x(x) =)
(x)
1=
Σ ccC.
Q(v = clp(x)) = Σ
Σ
Q(p(x))
Q(p(x))
((((x)))
cEC.
DEC.
15

---

Published in Transactions on Machine Learning Research (04/2023)
yielding that Q(p(x))Y, = c) = Q(p(x)), for CE Cu, Lu. This is equivalent to,
Q(v==cl((x)) = = = = =Q(Y, =
@(p(x))
This completes the proof.
A.2 Additional Experiments on the Robustness of Transfer Flow
To investigate the consistency of transfer flow across different kernels and bandwidths, we compare the transfer flow values of the Gaussian and Laplacian kernels with varying bandwidths (h). In this study, we consider Gaussian and Laplacian kernels, each with 5 different bandwidths h. We compute the bandwidth as:
R - -xx1-x11 n(n 1) 1)
where n is the number of samples and x; and Ij represent the i-th and j-th samples, respectively. The physical meaning of bandwidth the average distance between all possible pairs of data points in the whole datasct.
Our analysis reveals that the transfer flow values differ across the different kernels and bandwidths (as shown in Table S). However, consistently observe that the high similarity settings result in higher transfer leakage compared to the low similarity settings.
Table 8: Comparison of transfer flow values for Gaussian and Laplacian kernel functions and vary bandwidths (h) CIFAR100. The high similarity settings consistently result in greater transfer flow compared to the low similarity settings. The highest value for each unlabeled set and kernel function is bolded.
Kernel
Similarity
h
2h
2
Sum
High
0.13 + 0.00
0.15 + 0.00
0.15 + 0.00
0.11
+ 0.00
0.08
0.00
0.62 0.01
Gaussian
Low
0.06 = 0.00
0.07
= 0.00
0.07
= 0.00
0.05
0.00
0.04
0.00
0.28 0.01
High
0.06 + 0.00
0.11
+ 0.00
0.12
= 0.00
0.09
0.00
0.06
0.00
0.43
0.00
Laplacian
Low
0.03 = 0.00
0.05 + 0.00
0.06 = 0.00
0.05 0.00
0.03 = 0.00
0.21 + 0.00
B Details of Section 5
B.1 Additional Experiments
Table 9: Comparison of different data settings on CIFAR100 and our proposed benchmark. We present clus- tering mean and standard error for each setting. (c) uses only the unlabeled set,, whereas (a) uses both the unlabeled set and the labeled set's images without labels. (b) represents the standard NCD setting, ie., using the unlabeled set and the whole labeled set. However, in CIFAR100-50 and low similarity case of our benchnark, (a) can get greater performance than (b). The higher mean is bolded.
Unlabeled set U1
Unlabeled set U2
Setling
CIFAR100-50
L, high
I1.5 medium
L2 low
L low
LI.5 - medium L2 - high
(c)
X X
54.9 + 0.4
70.5 1.2
70.5 + 1.2
70.5 + 1.2 71.9
+ 0.3
71.9 + 0.3
71.9 + 0.3
aa)
X
64.1 + 0.4
79.6 = 1.1
79.7 = 1.0
80.3 + 0.3 85.3
+ 0.5
85.2 + 1.0
89.2 + 0.3
X. + (X,, 17)
62.2 = 0.2
83.9 + 0.6
81.0 0.6
77.2 + 0.8 77.5 = 0.7 82.0
1.7
88.4 + 1.1
We also conduct full experiment on UNO with settings (a) - (c) to further examine the benefit of the self-supervised from the labeled set. In Table 9. by comparing (a) and (c), we find that NCD performance is consistently improved by incorporating more images (without labels) from labeled set, with around 10% improvement in accuracy CIFAR100. For our benchmark, setting (a) obtains improvement.
16

---

Published in Transactions on Machine Learning Research (04/2023)
about 6% - 10% over (c) and the increase is more obvious in the low similarity settings. This numerical results demonstrates that self-supereised knowledge from the labeled set is beneficial for NCD performance under verying semantic similarity.
B.2 Experimental Setup and Hyperparameters
In general, we follow the basclines regarding hyperparameters and implementation details unless stated otherwise. We repeat all experiments on CIFARI00 10 times and the ones on our proposed ImageNet-based benchmark 5 times, and report mean and standard deviation. All experiments are conducted using PyTorch and run on NVIDIA V100 GPUs
Adapting Baselines to the Self-supervised Setting Since experiment settings (1) and (2) do not make use of any labels, we need to adapt UNO (Fini et al. 2021 to work without labeled data. The standard UNO method conducts NCD in two-step approach. In the first step, it applies supervised pretraining on the labeled data only. The pretrained model is then used as an initialization for the second step, in which the model is trained jointly on both labeled and unlabeled data using one labeled head and multiple unlabeled heads. To achieve this, the logits of known and novel classes are concatenated and the model is trained using a single cross-entropy loss. Here, the targets for the unlabeled samples are taken from pseudo-labels, which are generated from the logits of the unlabeled head using the Sinkhorn-Knopp algorithm (Cuturi, 2013
To adapt UNO to the fully unsupervised sctting in (1), we need to remove all parts that utilize the labeled data. Therefore, in the first step, replace the supervised pretraining by self-supervised one, which is trained only on the unlabeled data. For the second step, we simply remove the labeled head, thus the method is degenerated to a clustering approach based solely on the pseudo-labels generated by the Sinkhorn-Kuopp algorithm. For setting (2), we apply the self-supervised pretraining based on both unlabeled and labeled images obtain the pretrained model in the first step. In the second step, we replace the ground-truth labels for the known classes with pscudo-labels generated by the Sinkhorn-Knopp algorithm based on the logits of these classes. Taken logether, the updated setup utilizes the labeled images, but not their labels.
For NCL and RS, the modifications are similar to the ones done UNO. Both methods consist of three steps, solf-supervised pretraining step, followed by another supervised pretraining and lastly the novel class discovery step. To adapt them to setting (2). we replace the first two steps with a single self-supervised pretraining step based on SwAV (Caron et al. 2020p. For the last step, we simply replace the ground truth labels with labels obtained using k-means (MacQucen et al. 1967). while koeping the framework itself the same.
We conduct our experiments on CIFAR100 well as our proposed ImageNet-based benchmark. All settings and hyperparameters are kept close as possible as to the original baselines, including the choice of ResNet18 as the model architecture. We use SwAV as self-supervised pretraining for all experiments. The pretraining is done using the small batch size configuration of the method, which uses a batch size of 256 and a queue size of 3840. The training is run for 800 epochs, with the queue being enabled at 60 epochs for our TmageNet- comparison with the standard NCD setting, the same data augmentations were used. In the second step of UNO, we train the methods for 500 epochs on CIFAR100 and 100 epochs for each setting on our benchmark. The experiments are replicated 10 times on CIFAR100 and 5 times the developed benchmark, and the averaged performances and their corresponding standard errors are summarized in Table 5
Hyperparameters
benchmark and 100 epochs for CTFAR100. To ensure a fair
C Computation Cost
Overall, the methods we assossed have relatively low computational costs, primarily because of the utilization of a wiiht 18 backbone across all methods. For instance, training a single split of our benchmark using ImageNet took approximately 8 hours for supervised pretraining and about 13 hours for the class discovery phase, all performed on a single Nvidia V100 GPU. During the inference phase, the methods manifest. higher similarity since computation steps exclusive to the training process, such as the Baokhorn-Knopp algorithm for UNO, or the computation of ranking statistics for RS, are excluded.
17

---

Published in Transactions on Machine Learning Research (04/2023)
The complexity of (pseudo) transfer flow is O((Cu)² m²), where C, denotes the number of classes in the unlabeled dataset and denotes the number of unlabeled samples.
D
Discussion
The nature of the dark knowledge transferred from the labeled set is still mystery and provide intuitive understandings on why supervised knowledge can have a negative impact on NCD performance.
Bias / Conflicting information: In cases where there is significant bias and conflicting information between the supervised knowledge (Y) in the labeled data and the predictive information (Yu[Xu) in the unlabeled data, the utilization of supervised knowledge may lead to negative effects. Intuitively, supervisod knowledge obtained from X,, V1 provides two pieces of information, including classification rule and improved representation, while self-supervised information from X, primarily enhances representation. However, in scenarios with low semantic similarity or differing classification rules, the conflicting information present can pose challenges for the model to reconcile effectively.
Limited gencralization: From the information bottleneck Saxe et al. (019 Tishby et al. 2000 Shamir
et al. 2010)) perspective, feature space X has a larger dimension and contains richer information content. However, the incorporation of category information Y may lead to the removal of information that is not related to category Y, which result in reduction in the dimension of the feature space. Furthermore, combined with the first point, when the bias is high, the removed feature space may overlap with the unlabelled features. This may be one of the reasons why incorporating self-supervised has shown performance improvement, while incorporating supervised information has led to a reduction.
18

---

Published in Transactions on Machine Learning Research (04/2023)
E
Detailed Benchmark Splits
Table ImageNet class list of labelod split L and unlabeled split U1 of our proposed benchmark. As they share the same superclasses, they are highly related semantically. For each superclass, six classes are assigned to the labeled set and two to the unlabeled set. The labeled classes marked by the red box also included in Lis, which shares half of its classes with L1 and half with L2.
Superclass
Labeled Subclasses
Unlabeled Subclasses
garment
vestment, jean, academic gown, sarong, fur coat, apron
swimming trunks, miniskirt
tableware
wine bottle, goblet, mixing bowl, coffee mug, water bottle, water jug
plate, beer glass
insect
leafhopper, long-horned beetle, lacewing, dung bee- tle, sulphur buttcrfly, fly
admiral, grasshopper
vessel
wreck, liner, container ship, catamaran, trimaran, lifeboat
yawl, aircraft carrier
building
toyshop, grocery store, bookshop, palace, butcher shop, castle
beacon, mosque
headdress
cowboy hat, bathing cap, pickelhaube, bearskin, bon- net, hair slide
crash helmet, shower cap
kitchen utensil
cocktail shaker, frying pan, measuring cup, tray, spatula, cleaver
caldron, coffeepot
footwear
knee pad, sandal, clog, cowboy boot, running shoe,
Christmas stocking, maillot.
Loafer
neckwear
stole, necklace, feather boa, bow tie, Windsor tie, neck brace
bolo tie, bib
bony fish
puffer, sturgeon, coho, cel, rock beauty, tench
gar, lionfish
tool
screwdriver, fountain pen, quill, shovel, screw, com- bination lock
torch, padlock
vegetable
spaghetti squash, cauliftower, zucchini, acorn squash, artichoke, cucumber
cardoon, butternut squash
motor vehicle
beach wagon, trailer truck, limousine, police van, convertible, school bus
garbage truck, moped
sports oquipment
balance beam, rugby ball, ski, horizontal bar, racket, dumbbell
tennis ball, croquet ball
carnivore
otterhound, flat-coated retriever, Italian greyhound, Boston bull, Bedlington ter- Shih-Tzu, basenji, black-footed ferret.
rier
19

---

Published in Transactions on Machine Learning Research (04/2023)
Table 11: ImageNet class list of labeled split L2 and unlabeled split U2 of our proposed benchmark. As they share the same superclasses, they are highly related semantically. For each superelass, six classes are assigned the labeled set and two the unlabeled set. The labeled classes marked by the red box are also included in L.s, which shares half of its classes with L1 and half with L2.
Superclass
Labeled Subclasses
Unlabeled Subclasses
fruit
corn, buckeye, strawberry, pear, Granny Smith, pineapple
acorn, jackfruit
saurian
African chameleon, Komodo dragon, alligator lizard, agama, green lizard, Gila monster
banded gecko, chameleon
American
barrier
stone wall, chainlink fence, breakwater, dam, ban- nister, picket fence
worm fence, turnstile
electronic
equip-
casselle player, modem, printer, monitor, computer keyboard, pay-phone
dial telephone, microphone
ment
serpentes
green snake, boa constrictor, green mamba, ringneck snake, thunder snake, king snake
rock python, garter snake
dish
hot pot, burrito, potpie, meat loaf, cheeseburger, mashed potato
hotdog, pizza
home appliance
espresso maker, toaster, washer, space heater, vac- uum, microwave
dishwasher, Crock Pot.
measuring instru- ment
wall clock, barometer, digital watch hourglass, mag- netic compass, analog clock
digital clock, parking meter
primate
indri, siamang, baboon, capuchin, chimpanzee, howler monkey
patas, Madagascar cat
crustacean
rock crab, king crab, crayfish, American lobster, Dungeness crab, spiny lobster
fiddler crab, hermit crab
musical instrument
organ, acoustic guitar, French horn, electric guitar, upright, maraca.
violin, grand piano
arachnid
black and gold garden spider, wolf spider, harvest- tarantula, scorpion man, lick, black widow, barn spider
aquatic bird ungulate
dowitcher, goose, albatross, limpkin, white stork, red-backed sandpiper
drake, crane
hippopotamus, hog, llama, hartebeest, ox, gazelle
warthog, zebra
passerine
house finch, magpie, goldfinch, indigo bunting, chick- adee, brambling
bulbul, water ouzel
20

---

Published in Transactions on Machine Learning Research (04/2023)
Table 12: Labeled Split of CIFAR100 used in Section We construct data settings based on its hierarchical class structure. U1-L1/U2-I2 are share the same superclasses.
Superclass
Labeled Subclasses (I1)
Unlabeled
Subelasses
(U1)
aquatic mammals
dolphin, otter, seal, whale
beaver
fish
flatfish, ray, shark, trout
aquarium fish
flower
poppy, rose, sunflower, tulip
orchids
food containers
bowl, can, cup, plate
bottles
fruit and vegetables
mushroom, orange, pear, sweet. pepper
apples
houschold electrical devices
keyboard, lamp, telephone, television
clock
houschold furniture
chair, couch, table, wardrobe
bed
insects
beetle, butterfly, caterpillar, cockroach bee
large carnivores
leopard, lion, tiger, wolf
bear
large man-made outdoor things
castle, house, road, skyscraper
bridge
Superclass
Labeled Subclasses (L2)
Unlabeled Subclasses
(U2)
large natural outdoor scenes
forest, mountain, plain, sea
cloud
large omnivores and herbivores
cattle, chimpanzee, elephant, kangaroo
camel
modium-sized mammals
porcupine, possum, raccoon, skunk fox
non-insect invertebrates
lobster, snail, spider, worm
crab
people
boy, girl, man, woman dinosaur, lizard, snake, turtle
baby
reptiles
crocodile
small mammals
mouse, rabbit, shrew, squirrel
hamster
trees
oak tree, palm (ree, pine tree, willow tree
maple
vehicles bus, motorcycle, pickup truck, train
bicycle
vehicles
rocket, streetcar, tank, tractor
lawn-mower
21


Reviews:
[{"review_id": "5cAXloqrX7", "reviewer": "Reviewer_xz1z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n- The viewpoint that supervised knowledge may hurt novel class discovery performance is pioneering and instructive for novel class discovery tasks. \n- The proposed transfer leakage/pseudo transfer leakage can reflect the semantic similarity between labeled and unlabeled datasets, which can provide guidance for the usage of supervised data in novel class discovery problems. \n- The approach the paper proposed obtains good results and outperforms the state-of-the-art method UNO. \n\nWeaknesses: \n- Although the proposed transfer leakage can provide guidance for handling novel class discovery problems, I cannot find an essential role of transfer leakage in the experiment. I mean that if we delete the contents about transfer leakage in Section 5, we can still obtain the conclusion that supervised knowledge may hurt novel class discovery performance from the experiments and provide solution in Section 5.2.2. \n- From the experimental results in Section 5.2.2, we can see that from ~3%-~5% improvement in the experiment, ~2% improvement should be owed to self-supervised pretraining. Why is self-supervised pretraining not introduced in detail in the paper? Are there any existing works that use self-supervised pretraining to improve novel class discovery performance? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper mainly proposes a new viewpoint for novel class discovery problem: supervised knowledge may hurt novel class discovery performance. To verify the viewpoint, the paper first proposes a novel metric, called transfer leakage, to measure the semantic similarity between datasets. Then, the paper conducts experiments to demonstrate that supervised knowledge weakens novel class discovery performance when the semantic similarity between labeled and unlabeled datasets is low. Finally, the paper proposes two solutions to improve existing novel class discovery methods.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions/suggestions: \n- The paper said that “supervised knowledge may hurt novel class discovery performance”, and are there any potential reason why supervised knowledge hurts novel class discovery performance?\n- In the problem definition, the paper sets up that samples in labeled and unlabeled datasets do not have same labels. However, in related works such as UNO, samples in labeled and unlabeled datasets share some labels. Why different papers choose different settings on labels in labeled and unlabeled datasets?\n- In the definition 1 in Section 3.1, it is said “novel class discovery aims to predict the labels of the unlabeled dataset”. It seems that we only need to find the new classes in the unlabeled dataset, but do not need to classify the samples from the definition. However, in experiments, accuracy is used as evaluation metric, which shows that the category judgment of each sample is needed. Is the description in the definition proper? \n- In the paper, the computation costs of different methods and transfer leakage/pseudo transfer leakage are not discussed. \n- In section 5.1, there are 3 settings. But why there are the experiment results of (c) only appear in appendix?\n- The experiment results in Section 5.2.1 mention error margins, is there any quantitative definition of error margin in the paper?\n- Some figures/tables are not put at the right place. For example, Figure 1 is in page 3, but it is first mentioned in page 6.\n- In the last sentence in conclusion, “a straightforward method” should be briefly introduced.\n- Why the semantic similarity quantification is named as “transfer leakage”? The “leakage” shows some information that should not be spread is transferred. \n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0bVncVWghX", "reviewer": "Reviewer_LiEt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\nThis paper considers a significant problem, NCD, and provides a more detailed analysis of NCD.\n\nA new metric is proposed to show the similarity of labeled and unlabeled data.\n\nExtensive experiments are provided to validate the performance of the proposed method.\n\nA new benchmark is proposed for NCD.\n\nCons:\n\nAs suggested by recent literature on MMD, kernel selection is important for it. How does the kernel selection influence your methods? Like changing the bandwidth of the Gaussian kernel.\n\nIn Chi et. al. (2022), they have an analysis regarding the solvability of NCD. Can the authors provide more comparison between your work and Chi et. al.?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the NCD problem and thinks about it more deeply. In Chi et. al. (2022), although they show the solvability of NCD, quantitive analysis is not explored. This paper fills this gap and contributes to the NCD field a lot.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "No concern regarding the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3PYX8hcPhh", "reviewer": "Reviewer_wTCF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "For strengths,\n-\tThe paper identified and studied a meaningful question: Is supervised knowledge always helpful at different levels of semantic relevance? The studied problem is well-motivated. The effect of different levels of semantic relevance is rarely studied before.\n-\tThe proposed metrics transfer leakage/pseudo transfer leakage are interesting and can be applied to the analysis of other problems as well.  \n-\tThe conclusion of the paper. supervised knowledge can harm the performance of NCD and the combination of supervised and self-supervised knowledge works better, will contribute to the community, and may open up new research directions.\n-\tThe experiments are conducted in various setups. The experimental design is comprehensive and clear.\n\nFor weaknesses,\n-\tThe article has some corresponding analysis for transfer leakage. For the more practical pseudo-transfer leakage, can there be some analysis and guarantee as well?\n-\tThe description of transfer leakage is not very clear. Especially it took me a while to understand the confusing notations such as p(x), p(X_u), X_u=x, X_l=x. Would like to see this section written in more detail.\n-\tFor the experiments, some results seem inconsistent. For example, the performance of UNO under L1.5-U1 is 81.0±0.5 in table 2, but the performance becomes 80.9±1.2 in table 5. Same for L2-U2.\n-\tSome findings seem problematic. For table 6, the higher pseudo-transfer leakage doesn’t always imply better performance (L2-U2 and L1.5-U1). Why does the paper simply say “pseudo transfer leakage is consistent with the accuracy under various datasets”?\n-\tFor figure 3, why setting alpha to zero or one can already outperform SOTA accuracy? In fact, no matter what the value of alpha is, the performance is always better than SOTA. The figure requires further justifications.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Novel Class Discovery (NCD) aims to infer novel categories in an unlabeled dataset by leveraging prior knowledge from a labeled set of related but disjoint classes. This paper examines the question of whether supervised knowledge is always helpful for NCD at different levels of semantic relevance. The inadequacy of existing NCD literature is revealed, which usually assumes that supervised knowledge is beneficial. A novel metric, transfer leakage, is proposed to measure the semantic similarity between labeled and unlabeled datasets. A new benchmark based on ImageNet class hierarchies is also built to show the validity of the proposed metric. A pseudo-version of the transfer leakage is developed as a practical reference to decide if supervised knowledge should be used in NCD. The effectiveness of the proposed modules is verified by extensive empirical studies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Thanks to the authors for the work. I hope the authors answer the questions mentioned in the above weaknesses. Although I like the idea of the paper, I’m unsure about the method clarity and especially the experimental correctness. ", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: See comments above.
Decision: Accept (poster)