Discover and Align Taxonomic Context Priors  for Open-world Semi-Supervised Learning
Authors: Yu Wang
Zhun Zhong
Pengchong Qiao
Xuxin Cheng
Xiawu Zheng
Chang Liu
Nicu Sebe
Rongrong Ji
Jie Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: open-world semi-supervised learning; novel class discovery;
TL;DR: We propose a uniformed framework to explore taxonomic context as prior for open-world semi-supervised learning.

Abstract: Open-world Semi-Supervised Learning (OSSL) is a realistic and challenging task, aiming to classify unlabeled samples from both seen and novel classes using partially labeled samples from the seen classes. 
Previous works typically explore the relationship of samples as priors on the pre-defined single-granularity labels to help novel class recognition. In fact, classes follow a taxonomy and samples can be classified at multiple levels of granularity, which contains more underlying relationships for supervision. We thus argue that learning with single-granularity labels results in sub-optimal representation learning and inaccurate pseudo labels, especially with unknown classes. In this paper, we take the initiative to explore and propose a uniformed framework, called Taxonomic context prIors Discovering and Aligning (TIDA), which exploits the relationship of samples under various granularity. It allows us to discover multi-granularity semantic concepts as taxonomic context priors (i.e., sub-class, target-class, and super-class), and then collaboratively leverage them to enhance representation learning and improve the quality of pseudo labels.
Specifically, TIDA comprises two components: i) A taxonomic context discovery module that constructs a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors; ii) A taxonomic context-based prediction alignment module that enforces consistency across hierarchical predictions to build the reliable relationship between classes among various granularity and provide additions supervision. We demonstrate that these two components are mutually beneficial for an effective OSSL framework, which is theoretically explained from the perspective of the EM algorithm. Extensive experiments on seven commonly used datasets show that TIDA can significantly improve the performance and achieve a new state of the art. The source codes are publicly available at https://github.com/rain305f/TIDA.

Discover and Align Taxonomic Context Priors for Open-world Semi-Supervised Learning
Yu Wang
Zhun Zhong
Pengchong Qiao
Xuxin Cheng
Xiawu Zheng
Chang Liu
Nicu Sebe"
Rongrong Ji Jiee Chen
School of Electronic and Computer Engineering, Peking University, Shenzhen, China
School of Computer Sceince, University of Nottingham, United Kingdom
Peng Cheng Laboratory, Shenzhen, China Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University
5 Department of Automation, Tsinghua University, Beijing, China Program, Peking University Shenzhen Graduate School, China Department of Information Engineering and Computer Science, University of Trento, Italy
AI for Scienee (AI4S)-Preeeee)
(rain_wang, peqiao, chengxx J @ stu, stu.pku.edu.cu (zhengxw01, chenj J @ pcl.accccn zhunzhong007 @gmail.com liuchang @ @wwiiggua.cdd.... edu.en niculac.scbe@uui unit..i
riji xmu.cdu.enn
Abstract
Open-world Semi-Supervised Learning (OSSL) is realistic and challenging task, aiming to classify unlabeled samples from both seen and novel classes using par- tially labeled samples from the seen classes. Previous works typically explore the relationship of samples as priors on the pre-defined single-granularity labels help novel class recognition. In fact, classes follow a taxonomy and samples can be classified at multiple levels of granularity, which contains more underlying relationships for supervision. We thus argue that learning with single-gran granuulrrtl labels results in sub-optimal representation learning and inaccurate pseudo labels, especially with unknown classes. In this paper, we take the initiative to explore and propose uniformed framework, called Taxonomic context prlors Discovering and Aligning (TIDA), which exploits the relationship of samples under various granu- larity. It allows us discover multi-grande malti-granullrityy semantic concepts taxonomic context priors (i.e., sub-class, target-class, and super-class), and then collabora- tively leverage them to enhance representation learning and improve the quality pseudo labels. Specifically, TIDA comprises two components: i) A taxonomic context discovery module that constructs a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors; ii) A taxonomic costext-based prediction alignment module that enforces consistency across hi- erarchical predictions to build the reliable relationship between classes among various granularity and provide additions supervision. We demonstrate that these two components are mutually beneficial for an effective OSSL framework, which is theoretically explained from the perspective of the EM algorithm. Extensive experiments on seven commonly used datasets show that TIDA can significantly improve the performance and achieve a new state of the The source codes are publicly available at
Introduction
Deep neural networks have obtained impressive performance on a variety of visual and language tasks [42, 48, 61, 15, 11, 12, 13, 14, 35, 36, 37, 45, 46]. However, their success is largely dependent
*Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Pseudo-L.abel Ace.
58.15%
Chair
Single
Acc
Granularity Priors
Target
Inputs
(a) Previous Single Taxonomy Methods
CIFARIOU
120 Epoch
Furmiture
Super
x Preude-Label Ace.
Various Tnxonomic
Context Priors
65.67%
Align
Acc
Chair
Seper
Funkkar
Targer
Align
Kaach
Chair
Bel
Scen/Novel Inputs
Model
Wooden Chaire
Basches
In
Sub
Weodan Cluiss
CIFAR100
Epoch
(b) Contextual Taxonomy Method (Ours)
Figure I: (a) Previous works (e.g. TRSSL [65]) typically focus on constraining models on the pre- rre-nefined single granularity, resulting in inferior pseudo labels. (b) Our method explicitly explores the multi- cualiiigranulartyy priors for model training, producing more accurate pseudo labels.
on abundant labeled data, where the acquisition is expensive and time-consuming [75]. One popular solution to this problem is Semi-Supervised Learning (SSL) [69, 34, 63, 54, 62, 44, 56), which aims to reduce annotation costs by supplementing a small amount of labeled training data with large number of unlabeled data.
Despite their significant success in various tasks [68, 77, 21, 75, 69, 53, 78, 38, 73, 59]. most existing SSL methods are designed under the closed-set assumption, i.e., the training and test data share the same class set. Recently, several works [64, 26, 54, 31, 74] extend the standard SSL to an open-world setting, where the unlabeled are not present in the labeled set. In Open-world SSL (OSSL), the model is required to identify samples from both known and novel classes with partial labeled sample from seen classes. Due the lack of labeled samples for novel classes, it is vital to exploit some priors auxiliary supervision learn discriminative representations and classifiers for all classes. Existing works either exploit pairwise similarity prior [64, 5, 25] or class distribution prior [65] to achieve this. However, as single granularity, which suffer from sub-optimization and inaccurate pseudo labels, due to the limited supervision.
contains samples from novel (unknown) classes that
shown in Fig. (a), these methods only explore priors
In this paper, we argue that leveraging multiple levels of granularity as semantic priors (e.g., sub- classes, classes, and super super-classes, etc.) is a more preferable solution for OSSL [43]. which helps us utilize more underlying relationship to supplement simplex single supervision. As illustrated in Fig. 1(b), in the super-class granularity, we can treat Couch (seen) and Chair (novel) as belonging to the same super-class Furniture. This builds a relationship between seen and novel classes and constraints samples from Chair and Couch be closer [43]. Meanwhile, in the sub-class granularity, Chairs are clusteedd into different sub-classe according the materials, helping us distinguish some hard negative samples [24]. These motivate us to inject such taxonomic context into OSSL for producing more discriminative representation learning and accurate pseudo-labeling. To this end, we propose a novel OSSL framework, named Taxonomic context prlors Discovering and Aligning (TIDA), to explore the relationship of samples (or classes) under various granularity. Without auxiliary supervision, TIDA can automatically discover are then collaboratively leveraged to improve representation and classifier learning. Meanwhile, the alignment across granularity ensures consistent taxonomic context and additional supervision.
taxonomic context priors, which
Specifically, we develop two modules: i) Taxonomic Context Discovery (TCD) module, which discovers the underlying taxonomic context priors by constructing set of hierarchical prototypes to cluster samples; ii) Taxonomic Context-basse based prediction Alignment (TCA) module, which enforces consistency across hierarchical predictions to build reliable relationships between classes at various levels of granularity and provide additional supervision. These two modules are mutually beneficial for learning discriminative representation and accurate pseudo-labeling for OSSL. As shown in

---

Fig. 1(b), our method can largely improve the quality of the pseudo label over the baseline.
To sum up, the main contributions are as follows:
We identify the importance of multi guanti-gaanulrrity priors in the context of OSSL and introduce new type of prior knowledge, i.e., taxonomic context priors, for solving the OSSL problem. We introduce a uniformed OSSI framework, which can discover taxonomic context priors without any extra supervision. With the proposed cross-hierarchical prediction alignment, our framework can effectively enhance the performance of the model.
Experiments conducted on seven visual benchmarks show that our TIDA achieves new state-of- the-art results. Additionally, we provide theoretical analysis with the EM algorithm to better understand the underlying mechanism of our approach.
2 Related work
Open-world Semi-Supervised Learning (OSSL). Semi- Semi-Suuervised Learning (SSL) aims to learn informative semantics from unlabeled data to reduce the dependence on human annotations [10, 2, 1, 39, 7, 60, 33, 9, 30, 49, 67, 16, 6, 76, 4]. However, these methods assume that the unlabeled and labeled samples come from the same distribution, which is impractical in real-world applications. To address this limitation, ORCA [5] extends SSL to the more realistic and challenging open world, which assumes the unlabeled data are from both novel and labeled (seen) classes. NCLPS [25] proposes exploit pairwise similarity priors with distribution alignment and applies an adaptive threshold to synchronize the learning pace between seen/novel classes, OpenLDN [64] optimizes pairwise similarity loss by bi-level way, which exploits the information available in the labeled set as priors to implicitly cluster samples from novel class. Later, TRSSL [65] proposes a class- distribution-awee -warre pseudo-label method for OSSL, which utilizes sample uncertainty and class distribution as priors to generate lseels-labels. However, they still only focus on priors learned on single label-hierarchy, which lacks accurate modeling of multi-level semantic relations between seen/novel category samples. For new perspective, TIDA applies more preferable solution, which subdivide semantic concepts from coarse to fine as taxonomic context (i.e., sub-class, target-class, and super-class). With taxonomic context to describe seen/novel samples, it enjoys a number of desirable properties, including flexible encoding of label relations, predictions consistent with label relations [19]. Our model can significantly improve performance by exploiting the label relations.
Novel Class Discovery (NCD). NCD is a challenging open-world problem, which most closely relates to OSSL and aims at clustering unlabeled samples of novel classes with the guidance of knowledge from seen categories [70, 32, 80, 79, 50, 66, 51, 57]. RankStats [28] utilizes rank statisties to transfer the knowledge of the labeled classes to the problem of clustering the unlabelled images. UNO [23] first formulates the NCD problem into a transportation problem by Shinkhorn Knopp algorithm [17]. Different from OSSL, NCD has a strong assumption that all the unlabeled samples belong to novel classes, thus NCD methods are not applicable to OSSL problems. In this paper, we adopt some NCD methods as comparisons, but the results show that they fail to jointly distinguish seen/novel samples.
3 Methodology
Problem Setup. Similar to standard closed-worle SSL, the training data for OSSL consist of labeled
data D' (a/, Ni E X' x y and unlabeled data n {-²", y ),N" E X' y", where 2! is the i-th labeled image with the label y x is the i-th unlabeled image belonging to the class y²" that is not available during training, and N₁ < No. Different from standard SSL. OSSL assumes that 0d C We aims to train encoder and a |)* classifiers to classify all classes. We denote ) as seen classes and " \ yd as novel classes y", i.e., ) )" y.. Following [5], we assume the number novel classes [ is known.
Bascline. Different from standard SSL, the pseudo-labeling process is inherently ill-posed in OSSL since there are no labeled samples for novel classes. To solve this problem, TRSSL [65] and UNO [23] design a unified self- lelf-labeling classification objective based on Shinkhorn Shinkhorn-Knopp algorithm [17]. They view pscudo- labeling as an optimal transport problem that can generate reliable lacuds for unlabeled data (more details refers to [65, 23]. The objective can be formulated as:
1 +|>-1|
exp(S(2i, (21,,;)) 0;))
Loe -
Σ
v log Vl.
o/ p(isxz)
(I)
+
exp (2,, Cm.)
3

---

Discovery (TCD Sec.3.1)
Align (TCA in Sec.3.2)
Samples
goarse-grained
ansupervised cluster
Super prioes
Furmiture
Seen
TCD
splix
Prototy
Image Encoder
D
target-grained
Classficial Boundary
Secall supervised cluster
O
TCA Loss
TCD Loss
TCA Lose
Plastic/l Letther Chair
A Estimated
Novel
Weoden Chair
Chair
Prototypes TC priors
fine-grained
uncupervised cluster
Sub priors
TCD Loss
merge
4
Extimated
Boundary TC priors
The overall architecture of TIDA. First, we extract features using an image encoder. Then, we use the proposed Taxonomic Context Discovery (TCD) build hierarchical prototypes to cluster features under various granularity by optimizing TCD losses Lted. Given the discovered taxonomic context (i.e. hierarchical prototypes), the proposed Taxonomic Context-based prediction Alignment (TCA) estimates taxonomic context-based predictions on target hierarchy by splitting/merging cluster results on coarse/fine- -raineedd into the target. Last, TCA constrains the estimated predictions to be consistent with the original one on the target hierarchy with TCA losses Ltaa. During inference, the encoder and target-grained prototypes are applied to classify seen/novel class samples. Colors indicate different classes for samples or different granularity for prototypes.
Figure 2:
where E I I +13"I denotes the ground-truth pseudo-label for labeled / unlabeled sample my 2 is the output predicted by the model, representing the probability of x; belongs to j-th class. z represents the features of sample z; and c, represents the prototype j-th class. S(:,)) is cosine similarity function. Note that to mitigate over- fett-fitting noisy pseudo-labels, both methods first obtain pseudo labels for two views of each unlabeled sample by applying various perturbations. Then, the classification objective in Eq. for unlabeled data is accomplished by optimizing each view using the pseudo-labels from the other view. For simplify, we do not explicitly display this this self-labeling strategy the
pseudo-label exchange process in Eq. In this paper, baseline. Differently, we propose to explore underlying taxonomic context priors as a remedy for single-granularity supervision. As shown in Fig. 2, TIDA consists of two mutually benefited modules: Taxonomic Context Discovery (TCD) and Taxonomic Context-based prediction Alignment (TCA). Next, we describe TIDA in detail.
3.1 Taxonomic Context Discovery
The lack of supervision for novel categorics in OSSL poses a challenge for models to learn accurate and discriminative features for classification. Inspired by [58, 27, 43], argue that the underlying taxonomic context of data facilitate novel class discovery, which can provide rich relationships between samples under different category granularity for supervision. To achieve this goal, build several learnable hierarchical prototypes C =
upon a shared feature encoder So : X + Z, which are normalized and can be regarded semantic priors under various category granularity. L is the number of hierarchies and n4 is the number of prototypes of the I-th layer and M1 < 112 < ... Ti,, Note that the number of prototypes ni controls the granularity of classification, where larger/smaller ny means more tin-/ccarse-grrinee semantic concepts at hierarchy I.
Specifically. TIDA first obtains normalized feature representation 2 R fe( for the i-th sample x; by image encoder fo- Then, we use z; and prototypes on each hierarchy to perform clustering based on the baseline. The combination of loss functions each hierarchy is written as:
L
L
exp(S(z,,))/1
Lind
log ofd
log E exp(S(z), (/T)
1=1
1=1
where v {0, 1)11"42+ is the pseudo-label of x; on 1-th hierarchy generated by Sinkhorn- algorithm [17], of denotes the prediction of model for v; and 0.1 is the temperature. In this

---

Spait
Sipert
(lspert
Ferniture Suppei
Animal
Supaz
reprenciation
Aligo
DE
Coach
Cluir
Ting
Couch
Chair
Dog
10 AutoMONNIC
Go hg
Sihi
Suk2
Worden Chair
Sahi
Saha
10 Anggeggrsine
AP
y/2
Mazis muliipiiationn
The tirgetgraatiod porskb-iabrrlune
peotonpe
Figure 3: (a) The left part denotes inconsistent laxonomic context, where the relationships between labels across different hierarchies are unclear and inaccurate. TCA can address this problem by explicitly building affinity relationships and aligning hierarchical predictions, resulting taxonomic context. (b) The illustration of TCA. Given the taxonomic context and the -rgget-griieed prototype, we first calculate their affinity matrix. Then, we split/merged the prediction on other- grained into target-grained as Eq. 4. Finally, we apply the loss Leea constrains them to be consistent.
consistent
paper, we apply TIDA with three prototype hierarchies, i.e., C = {fc, 1e (4 (4122) n1 = a n22 n22 | + [V* n2 B |C3| and 0 < a < 1 > 1, which can achieve consistent well performance on all settings. More analyses about L are provided in the Appendix.
Discussion. In fact, the objective of Eq. is a hierarchical classification task, where the model requires distinguishing samples at different granularity. As shown in Fig. 2, the first hierarchy can be regarded as the super-class or coarse-grained class level; the second hierarchy refers to the target-class/ / target-grained level (i.e.. D + +"""Iwwy classification); and the third can be regarded as the sub-class or fine-grained class level. We call the learned hierarchical clusters as taxonomic context priors because they reflect the inter-class or intra-class relationships among various granularity. However. with unknown class and limited labeled data, training each hierarchy individually cannot guarantee the faxonomic context consistency among hierarchies (see left part of Fig. 3()). That is, the relationships between labels among hierarchies are unclear and inaccurate, leading to limited advantages to target-classifier learning. With the consistent taxonomic context priors, we can build a clear subsumption relationship among classes of adjacent hierarchies and thus infer the predictions of a sample on a certain hierarchy based on its predictions on other hierarchies. For example, as shown in the right part of Fig. 3(a), given the affinities between the classes of different hierarchies, if we know the coarse-grained label of a sample is Furniture, we can infer that its target-grained label is most likely Chairs or Coach and definitely not Dog. Similarly, if we know the fine-grained label of a sample is Wooden Chairs, we can infer that its target- target-grained label is most likely Chairs and definitely not Coach Dog. This indicates that the key to taxonomic context consistency is having reliable affinity relationships between labels of various granularity.
3.2 Taxonomic Context-based Prediction Alignment
To achieve taxonomic context consistency, we propose the Taxonomic Context-based prediction Alignment (TCA), as shown in Fig. 3(b), which aims to establish reliable affinity relationships across hierarchies. TCA is inspired by the statement of [19, 8]: given a sample and its label on a certain hierarchy/ 1, we can treat its unobserved labels on adjacent hierarchies as latent variables and use them to infer the probability belonging to each class on the hierarchy L. Then, the tree hierarchy constraint can be achieved by minimizing the cross-entropy loss between the label/pseudo-lat label and the inferred prediction of the sample. To this end, we regard coarse/fine- -gainedd predictions as latent variables and use them to infer target-graine predictions. This allows us to explicitly build affinity relationships across hierarchics and supervise them with partially labeled samples from seen classes. In addition ensuring taxonomic context consistency, TCA also offers two more benefits: i) coarse-/fine- -raieeerrndd classifications can aid help predict target-grained classes [19, 8, 43]; ii) the supervision of labeled samples of seen classes can improve discrimination coarse/fine- grained. -araeniinnngg.ddd.....
Specifically, given parse//ine-grainee prototypes C" and target-grained prototypes C², we first obtain the affinity matrix Mr2 E Ru, where for each r.? E Mr22, m = S( eg) of is the i-th prototype in C", of is the j-th one in C², and r = 1, Second, we estimate the taxonomic bontext-based similarity between the feature of each sample and the target-grained prototypes 2
5

---

with C" as latent variables, as follows:
S(zi, 2 Σ S(z,, (
n:2 maj
Σ
S(a,, ④ S(ck et).
(3)
k=1
Then, we can obtain the taxonomic context-based prediction at the target grained level by:
exp(E S(z,, -(C,
2 2
EC'+0"
exp the S(z,, c.)/////// - 2.))
where indicates the hierarchy of C Finally, can enforce the taxonomic context consis- teney/alignment learning based on the cross- cross-entropy loss and the pseudo-labels (or ground truth) on the target level, formulated as:
1**+1>" Σ
Lica
y
log 1..21
(5)
Finally, we jointly perform taxonomic context discovery and taxonomic context-based prediction alignment during the training process, i.e, optimizing the model with the objective of Lted + Lica-
4 Theoretical Analysis
Assuming that the observed data D' U D are related to some latent variables sets which refer to the
hierarchical prototypes C' = {(c [1 1il above, where n, is the number of latent variables in set C², 1,... L and π < 112 < ... < nr. In our TIDA, L = 3, and n2 | + 1D*
LL
From the EM [18] perspective, TIDA aims to maximize the likelihood of the observed N, + N, samples based on these hierarchical prototypes C. According Jensen's inequality [47]. we write
the surrogate objective function as follows:
No+N, 0* 0" arg max D.C.
Σ
[((c) logg),ogp with Q(d) = pp(cppps. 21, (6) C'ECC,EEC
EEA
To sum up. TIDA aims to estimate the posterior class probability Q(c)) = p(c;; mi f) on each hierarchy at the E-steps. Then, TIDA draws each sample to the prototype of its assigned cluster on each hierarchy at the M-step by optimizing Eq. with known Q(c,))
Theorem By executing EM algorithm iteratively, samples from the same class in the feature space will be mapped into d-variate von Mises-Fisher (vMF) distribution whose probabilistic density given by g( n;; 8) = ca(n))ee' pe fo(m), where |c!| = 1 and d represents the mean direction, 1/T is the concentration parameter, and ca(K) is the normalization factor.
E-step. In fact, the posterior class probability Q(c)) p(d; 0) = T([;; c4), which is equivalent to y in Eq. 2. In this paper, TIDA the Sinkhorn-Knopp algorithm each hierarchy to assign the samples to the cluster center, as mentioned in Sec. 3.
M-step. Given Q(c,)= = p(c): x4, 0), TIDA aims maximize the surrogate function in Eq. 6 in M- step. Finally, as Theorem. 1 shown, samples will be mapped into different d-variate von Mises-Fisher (vMF) distributions under different hierarchies [72].
(a) Separate Objective Functions for Each Hierarchy. Due to Eq. is hard to optimize, follow- ing [47], we assume that the distribution around each prototype of satisfy isotropic Gaussian, iee. p(a; c/, 0)
exp (-(2: c)2 /2)²
//E (-(x (=()=)))//205 /200) Then Eq. 6 is written as:
exp
No+NN
exp (z C
0" C arg max
Σ
Σ
log E/ cxp (2s c2/T)²
0,C*
of which (b) Constrain the Consistency across Hierarchies in Feature Space. Since only optimizing Eq. results in inconsistencies of these learned vMF distributions across hicrarchies, propose to obtain
where cid denotes 2, 's assigned prototype (p(c)/: x1, 1 and ded E C and plays the role of temperature parameter. As we can see, Eq. is equivalent Eq. 2.

---

Table 1: Comparison with statc-of-the- methods on generic datasets.
CIFARIO
CIFAR100
ImageNet-100
Tiny ImageNet
Methods
Seen
Novel
All
Seen
Novel
All
Seen
Novel
All
Seen
Novel
All
DTC [29]
42.7
31.8
32.4
22.1
10.5
13.7
24.5
17.8
19.3
13.5
12.7
11.5
RankStats [28]
71.4
63.9
66.7
20.4
16.7
17.8
41.2
26.8
37.4
9.6
8.9
6.4
UNO [23]
86.5
71.2
78.9
53.7
33.6
42.7
66.0
42.2
53.3
28.4
14.4
20.4
ORCA [5]
82.8
85.5
84.1
52.5
31.8
38.6
83.9
60.5
OpenNCD [50]
83.5
86.7
85.3
53.6
33.0
41.2
84.0
65.8
73.2
TRSSL [65]
94.9
89.6
92.2
68.5
52.1
60.3
82.6
67.8
75.4
39.5
20.5
30.3
OpenI.DN [64]
92.4
93.2
92.8
55.0
40.0
47.7
TIDA (Ours)
94.2
93.4
93.8
73.3
56.6
65.3
83.4
71.2
77.6
45.7
28.4
37.2
Table 2: Comparison with state-of-the-art methods on fine-grained datasets.
Oxford-IIl Pet
FGVC Aircraft
Stanford Cars
Methods
Seen
Novel
All
Seen
Novel
Seen
Novel
DTC [29]
20.7
16.0
13.5
16.3
16.5
11.8
12.3
10.0
RankStats [28]
12.6
11.9
11.1
13.4
13.6
11.1
10.4
9.1
UNO [23]
49.8
22.7
34.9
44.4
24.7
31.8
49.0
15.7
30.7
TRSSL [65]
70.9
36.1
53.9
69.5
41.2
55.4
83.5
37.1
60.4
OpenLDN [64]
66.8
33.1
50.4
45.7
38.7
TIDA (Ours)
75.7
39.2
59.9
71.1
43.7
57.4
85.9
43.5
66.0
the aggregated latent variables ( with variables set C", then constrain the distribution around (2) be consistent with the original one around C², where r == 1.3.
First, establish the similarity relationship across adjacent hierarchy, i.e., the mapping ma- trix M S(CrT ,C²), where for m² k] E Mrr2, m²²"jj, ki] S(C), G.) Then we ob-
2|r tained its aggregated variables set ČzIr with latent variables O as Čzer ( J "M=, Ck
Ee [ Note, the variables set Č dynamically aggregates semantics from r-th hi- erarchy and builds communication among clusters across hierarchies, which facilitates aligning hierarchical semantics with target tasks.
Given aggregated variables, then apply consistent constraints on the sample's distribution around Čalr and the original ones in the target hierarchy, where 1,3. Specifically, we use aggregated variables ( to calculate the objective function in Eq. as follows:
Nr+NN
exp
24 21. /T)
0-,C* C
arg max O.C
Σ
Σ
log
(8)
Σ
k=
exp
2r 24 Ck
/T)
where the index s in 2/r corresponds the assigned target- prototype dt2 for feature 2 (p(c2: 2, = 1), As we can see, Eq. 8 is equivalent to Eq. 5. More details are in Appendix.
Experiment
5.1 Experiment Setup
Datasets. We evaluate TIDA on four commonly used generic image classification datasets (i.e. CIFARIO [41], CIFARIO0 [41]. TinyImageNet [20] and ImageNet-100 [20]) and three fine-grained
datasets (i.c. Oxford-IIT Pet [52], Standford-Catr [40] and Aircraft [55]). Following [5, 65], use the first half of classes as seen classes and the remaining as novel. More details are in the Appendix.
Implementation Details. Following [65, 5, 64]. we use ResNet-50 [22] for ImageNet- 100 and ResNet-18 [22] for the other datasets. For all datasets, we set the length of prototypes set L 3, in which C = {C', _, C3 and C ! U ) For the number of prototypes C and C we set / 2.5 for generic / fine-grained learning rate scheduler accompanied by a linear warmup, where we set the base learning rate 0.5 1.5 for generic/ fine-grained datasets. The warmup length is set to 10 epochs and the weight decay is set Ic-4. More details are in the Appendix.
|C C2| and |C 3 |C We set a = 0.2 / 0.4 and
datasets. We use cosine annealing-basee
7

---

Table 3: The ablation study. C-TCP: Coarse-grained Taxonomic Context Priors; F-TCP: Fine-
grained Taxonomic Context Priors; TCA: Taxonomic Context-based prediction Alignment. When using TCA only, the model is equipped with three target-grained classifiers that are aligned by TCA.
CIFAR100
Tiny ImageNet
Stanford Carr
# C-TCP F-TCP
TCA
Seen
Novel
All
Seen
Novel
All
Seen
Novel All
a)
67.0
48.9
57.9
39.7
21.8
31.1
84,8
38.3
61.3
b)
63.7
47.0
55.5
36.4
19.2
28.0
81.2
41.2
61.1
C)
71.3
51.6
61.6
43.7
27.1
35.8
84.9
40.8
62.9
d)
65.3
45.6
55.3
36.3
20.4
28.8
78.1
31.2
54.8
c)
71.5
54.6
63.1
44.6
27.3
36.8
85.3
42.3
64.1
f)
71.8
49.1
60.5
43.1
21.1
32.5
82.8
39.5
61.9
g
69.9
45.4
57.7
36.3
19.8
29.0
76.4
30.2
53.8
h)
73.3
56.6
65.3
45.7
28.4
37.2
85.9
43.5
66.0
5.2 Comparison with Staee-offThe-Art
We first compare TIDA with state-of-the-art methods [5, 23, 28, 29, 64, 65] four generic datasets and three fine-grained datasets, where the results are reported Tab. and Tab. 2, respec- tively. It is clear that TIDA consistently outperforms all saate-o-thh--ar methods across all datasets and metrics. Specifically, TIDA surpasses the current best competitor TRSSL [65] by 6.9% on TinyImageNet for All classes. Importantly, TIDA outperforms previous methods most datasets for Novel classes by a large margin, e.g., 7.9% on Tinylmage and 6.4% on Stanford-Cars. These results experimentally demonstrate that exploring taxonomic context priors auxiliary supervision a beneficial way for discriminating seen and novel classes under OSSL..
5.3 Ablation Study
Effect of Coarse-grained Taxonomic Context Priors. To verify the effectiveness of Coarse-grained Taxonomic Context Priors (C-TCP), we conduct a comparison with three variants: a) the baseline that only clusters samples on target-grained; b) the model that clusters samples on both searse-grained and target-graines without using Taxonomic Context-based prediction Alignment (TCA); c) the model which simultaneously clusters samples both on coarse- baars--grained and can see in Tab. 3, c) outperforms a) and b) but b) worsened a), showing that generic semantics will improve the performance only when the poarse-grained clustering is consistent with the target task.
with TCA. As we
Effect of Fine-Grained Taxonomic Context. Similarly, we also conduct a comparison with three variants to verify the effectiveness of Fine- sene-grained Taxonomic Context Priors (F-TCP), i.e., a) the baseline, d) the model additionally includes F-TCP learning, and e) the model additionally includes F-TCP learning and TCA. Results show that F-TCP learning facilitates the target task only when using ATC. Moreover, the fine- fine-grained semantic works relatively better than the generic semantic (see the comparison between e) and e)), especially the performance on Novel classes. In addition, jointly considering C-TCP and T-T could further improve the performance, as shown in variant h).
Effect of Taxonomic Context-based Prediction Alignment. As discussed in the previous two ablation studies, the proposed TCA is indispensable to our TIDA. Specifically, by comparing b) v.s. c), d) v.s. e), and g) v.s. h), we observe that only using coarse-grained taxonomic context learning or fine-grained taxonomic context learning, or both of them fail to achieve improvement over the baseline. However, after injecting the proposed taxonomic context-based prediction alignment module, the performance is significantly improved. Because the inconsistency among multi-granulan classification leads hard-optimization problem when using TCD alone. Actually, TCD applies multiple classifiers to cluster samples under different granularity. It can be regarded as multi-task learning with shared backbone. However, due lack of supervision (e.g.. semantic names and attributes), when using TCD alone, the training objective among classifiers is inconsistent and the optimization on the sub- and sup-classifiers is completely unsupervised. This may have a negative impact model optimization and consequently result in performance degradation. While, the TCA learn to align the training objectives among classifiers and enable the sub- and sup-classifiers to benefit from the knowledge of labeled data on the target classifier.
8

---

Tiny ImageNet
Stundard Car
Tiny ImageNet
Standard Car
39-
37-
03
31-
29-
27
25
0.5
1.0 0.1
0.4 0.6
0.8
1.0
1.0
2.5
(a) The analysis of
(b)The analysis of B
Figure 4: The hyper-parameters analysis. (a) a: The weight that decides the number super-classes. (b) 3: The weight that decides the number of sub-classes.
Coach
#subl
Bed
with open
8sub2
Wardrobe
(a) TRSSL
(b) Ours
(c) Fined-Grained TC
Wardrobe
(d) Coarse-Grained TC
Furniture
Figure 5: The t-SNE visualization of features from TRSSL and TIDA on CIFAR100, For clarity, we
randomly select 20 and 20 novel classes examples from CIFAR100.
Effect of Alignment across Multiple Granularity. It is interesting see whether the performance improvement comes from the alignment across multiple granularity or alignment across a number of classifiers. evaluate this, we introduce a variant with three target-grained clusters learning, i.., |C*| |C2| CSI [J*| + [)*]]_ Compared with f) and h), we obverse that the alignment across multiple target-gren segett-raaullarity is beneficial for Seen classes but there is no improvement or even reduction for Novel classes. This indicates that the improvement mainly comes from interaction among different levels granularity instead of different classifiers the same granularity.
5.4 Hyper-Parameter Analysis
Impact of the Number of Super-Classes. As shown in Fig. 4(a), is more suitable to assign smaller value for a, which leads to small number of coarse-grained classes. When approaches to 1, TIDA degenerates to the model that ignores the boarse-grained granularity. The best result is achieved by 0.2 / 0.4 for TinylmageNet/ Standard Car.
Impact of the Number of Sub-Classes. In contrast, as shown in Fig. 4(b), assigning a larger value for 3 leads higher performance, which enables to obtain fine-grained prior. Again, when nears 1, TIDA will largely overlook the fined-grained granularity and produce lower results. The best performance is obtained by
2.5 for Tiny ImageNet Standard Car.
5.5 Qualitative Analysis
T-SNE Visualization. We use t-SNE [71] to visualize the features learned by TIDA and the base- line [65]. As shown in Fig. 5()) and Fig. 5(b). TIDA produces more discriminative features than TRSSL [65], where the samples are generally better clustered. To better understand our method, we also visualize the fine- fine-graine mna-gra and coarse- coarse-grained features learned by TIDA, shown in Fig. 5(c) and Fig. 5(d). At the fine- tineggainee level, samples from the same categories are divided into different clusters; while, samples belonging to the same super-class are clustered together the coarse- coarse-grained grained level. This further verifies that TIDA can learn more semantic priors which are complementary.
Visualization of Affinity Matrix. In Fig.
calculate the affinity matrix by counting the number of samples belonging to each target class that are classified into each super-class. Results show that samples of the same target class are commonly classified into the same super-class for TIDA w/ TCA. In addition, we find that similar target classes are generally mapped into the same super-class. For example, "bowl, cup, lamp, plate, clock" assigned with the 18th super-class. In contrast, for TIDA w/o TCA, samples of the same target class tend to be classified into different super-classes, resulting in unclear and inconsistent affinity relationships among classes across hierarchies. These results validate the effectiveness of our TCA in establishing clear and consistent affinity relationship.

---

super-
21
26
36
Target clusses
16
56
(a) w/o Leon
61
71
76
51
91
96
super--lasses
Lure
21
26 31 36
66
classes
71
81 86 91 96
(b) w/ Lion
Figure 6: Affinity matrix that calculates the number of samples of each target class classified into each super-class CIFAR100. For clarity, rearrange the target classes, which belong to the same super-class being grouped together (see Appendix). Here, (a) and (b) follow the same order.
6 Contribution
In this paper, we propose novel framework known TIDA. It can effectively learn discriminative visual representations and improve pseudo-label accuracy by exploring the underlying taxonomic context priors. Specifically, TIDA consists of two core components: i) clustering samples hierarchi- cally to capture the underlying taxonomic context in latent space, and ii) constraining predictions across hierarchies to be consistent to build reliable affinity relationships ensure taxonomic context consistent. The theoretical analysis and extensive experiments on commonly used benchmarks with consistent performance gains both justify the superiority of our TIDA.
Despite obtaining high performance, TIDA and existing methods still suffer from several limitations: assuming that i) the domain distribution between labeled and unlabeled data is the same; ii) the glass-distribution is uniform; and iii) the number of novel classes known. Future work may focus exploring more practical OSSL situations that unlock the above constraints.
Limitations.
Acknovledgements. This work was supported in part by the National Key R&D Program of China (No. 18201), the Natural Science Foundation of China (No. 61972217, 32071459, 62176249, 62006133, 62271465), the MUR PNRR project FAIR (PE00000013) funded by the NextGenerationEUU, and the EU project Ai4Media (No. 951911).
References
[1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervise learning with distribution alignment and augmentation anchoring. In ICLR, 2019.
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurlPS, 2019.
[3] Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In NeurlPS, 2022.
[4] Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In NeurlPS, 2022.
[5] Kaidi Cao, Maria Brbie, and Jure Leskovec. Open-world semi-supervised learning. In ICLR, 2022.
[6] Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-tree trffitriinng for semi-supervised learning. In NeurIPS, 2022.
10

---

[7] Dong- Dong-Dong Chen, Wei Wang, Wei Gao, and Zhi-Hua Zhou. Tri-net for semi semi-supervised deep learning. In IJCAI, 2018.
[8] Jingzhou Chen, Peng Wang, Jian Liu, and Yuntao Qian. Label relation graphs enhanced hierarchical residual network for hierarchical grati-gramularity classification. In CVPR, 2022.
[9] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised leamers. In NewrIPS, 2020.
[10] Yanbei Chen, Massimiliano Mancini, Xiatian Zhu, and Zeynep Akata. Semi-supervised and unsupervised deep visual learning: A survey. TPAMI, 2022.
[11] Xuxin Cheng, Bowen Cao, Qichen Ye, Zhihong Zhu, Hongxiang Li, and Yuexian Zou, MI-Imel: Mutual learning and aarge-maginn contrastive learning for improving asr robustness in spoken language understanding. In ACL, 2023.
[12] Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, and Yuexian Zou. M 3 st: Mix at three levels for speech translation. ICASSP, 2023.
[13] Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, and Yuexian Zou. Ssvmr: Saliency-based self-training for video-music retrieval. In ICASSP, 2023.
[14] Xuxin Cheng, Zhihong Zhu, Ziyu Yao, Hongxiang Li, Yaowei Li, and Yuexian Zou. GhostT5: Generate More Features with Cheap Operations Improve Textless Spoken Question Answer- ing. In Proc. INTERSPEECH, 2023.
[15] Zesen Cheng, Pengchong Qiao, Kehan Li, Siheng Li, Pengxu Wei, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. segmentation. In CVPR, 2023.
rectification for weakly supervised semantic
[16] Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay K. Vasudevan, and Quoc V. Le. Autoaug- ment: Learning augmentation strategies from data. In CVPR, 2019.
[17] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurlPS, 2013.
[18] Arthur P. Dempster, Nan M. Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em algorithm. J R Stat Soc B, 1977.
[19] Jia Deng. Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In ECCV, 2014.
[20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[21] Yue Fan, Dengxin Dai, Anna Kukleva, and Bernt Schiele. Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning. In CVPR, 2022.
[22] Radhia Ferjaoui, Mohamed Ali Cherni, Fathia Abidi, and Asma Zidi. Deep residual learning based on resnet50 for COVID-19 recognition in lung CT images. In CoDIT, 2022.
[23] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. unified objective for novel class discovery. In ICCV, 2021.
[24] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Lue Van Gool. SCAN: learning classify images without labels. In ECCV, 2020.
[25] Lan-Zhe Guo, Yi-Ge Zhang, Zhi-Fan Wu, Jie-Jing Shao, and Yu-Feng Li. Robust semi- supervised learning when not all classes have labels. In NeurIPS, 2022.
[26] Lan-Zhe Guo, Zhen-Y Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zui-Hua Zhou. Safe deep semi- supervised learning for unscen-class unlabeled data. In ICML, 2020.
11

---

[27] Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. Hesc: hierarchical contrastive selective coding. In CVPR, 2022.
[28] Kai Han, Sylvestre-Alvise Rebuffi, Sébastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statisties. In ICLR, 2020.
[29] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. ICCV, 2019.
[30] Zhongyi Han, Xian-Jin Gui, Chaoran Cui, and Yilong Yin. Towards accurate and robust domain adaptation under noisy environments. In IJCAI, 2020.
[31] Rundong He, Zhongyi Han, Xiankai Lu, and Yilong Yin. Safe-student for safe deep semi- supervised learning with unseen-class unlabeled data. In CVPR, 2022.
[32] Yen- Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In ICLR, 2018.
[33] Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurlPS, 2021.
[34] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In CVPR, 2019.
[35] Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. inppecatioo-mmxxiizaaiie contrastive learning for compact jideo-and-language representations. In NeurlPS, 2022.
[36] Peng Jin, Jinfa Huang. Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In CVPR, 2023.
[37] Peng Jin, Hao Li, Zesen Cheng, Jinfa Huang, Zhennan Wang, Li Yuan, Chang Liu, and Jie Chen. Text-video retrieval with disentangled conceptualization and set-to-set alignment. In IJCAI, 2023.
[38] Ying Jin, Jiaqi Wang, and Dahua Lin. Semi-supervised semantic segmentation via gentle teaching assistant. In NeurIPS, 2022.
[39] Juho Kannala, Alex Lamb, Kenji Kawaguchi, Vikas Verma, Yoshua Bengio, David Lopez- Paz, Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. arXiv, 2019.
[40] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Tront, 2009.
[42] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Commun. of the ACM, 2017.
[43] Aoxue Li, Tiange Luo, Zhiwu Lu, Tao Xiang, and Liwei Wang. Large-scale few-shot learning: Knowledge transfer with class hierarchy. In CVPR, 2019.
[44] Gang Li, Xiang Li, Yujie Wang, Yichao Wu, Ding Liang, and Shanshan Zhang. DTG-SSOD: dense teacher guidance for semi-supervised object detection. In NeurIPS, 2022.
[45] Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, and Jie Chen. Toward 3d spatial reasoning for himan-like bext- visual question answering. TIP, 2022.
[46] Hao Li, Peng Jin, Zesen Cheng, Songyang Zhang, Kai Chen, Zhennan Wang, Chang Liu, and Jie Chen. Tg-vqa: Ternary game of video question answering. In ICCV, 2023.
12

---

[47] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. ICLR, 2021.
[48] Kehan Li, Zhennan Wang, Zesen Cheng, Runyi Yu, Yian Zhao, Guoli Song, Chang Liu, Li Yuan, and Jie Chen. Acseg: Adaptive conceptualization for unsupervised semantic segmentation. In CVPR, 2023.
[49] Yu-Feng Li and Zhi-Hua Zhou. Towards making unlabeled data never hurt. TPAMI, 2015.
[50] Jiaming Liu, Yangqiming Wang, Tongze Zhang, Yulu Fan, Qinli Yang, and Junming Shao. Open-world semi-supervised novel class discovery. In ICAI,, 2023.
[51] Mingxuan Liu, Subhankar Roy. Zhun Zhong, Nicu Sebe, and Elisa Ricci. Large-scale pre-trained models are surprisingly strong incremental novel class discovery. arXiv, 2023.
[52] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv, 2013.
[53] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. TPAMI, 2018.
[54] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS, 2018.
[55] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs, In CVPR, 2012.
[56] Yongri Piao, Chenyang Lu, Miao Zhang, and Huchuan Lu. Semi-supervised video salient object detection based on uncertainty-guidee pscudo labels. In NewrIPS, 2022.
[57] Nan Pu, Zhun Zhong, Xinyuan Ji, and Nicu Sebe. Federated generalized category discovery. arXiv, 2023.
[58] Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In CVPR, 2023.
[59] Pengchong Qiao, Zhidan Wei, Yu Wang, Zhennan Wang, Guoli Song, Fan Xu, Xiangyang Ji, Chang Liu, and Jie Chen. Fuzzy positive learning for semi-supervised semantic segmentation. In CVPR, 2023.
[60] Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan L. Yuille. Deep co-training for semi-supervised image recognition. In ECCV, 2018
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[62] Zhongzheng Ren, Raymond A. Yeh, and Alexander G. Schwing. Not all unlabeled data are equal: Learning to weight data in semi-supervised learning. In NeurlPS, 2020.
[63] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-awarr pseudo-label selection framework for semi-supervised learning. In ICLR, 2021.
[64] Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openldn: Learning to discover novel classes for open-world semi-supervised learning. In ECCV, 2022.
[65] Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. Towards realistic semi-supervised learning. In ECCV, 2022.
[66] Subhankar Roy, Mingxuan Liu, Zhun Zhong Nicu Sebe, and Elisa Ricci, Class-incremental novel class discovery. In ECCV, 2022.
13

---

[67] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic trans- formations and perturbations for deep sumi-supeeiiee learning. In NeurIPS, 2016.
[68] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NerulPS, 2020.
[69] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervissd deep learning results. In NeurlPS, 2017.
[70] Colin Troisemaine, Vincent Lemaire, Stéphane Gosselin, Alexandre Reiffers-Masson Joachim Plocon-Choled and Sandrine Vaton. Novel class discovery: an introduction and key concepts. arXiv, 2023.
[71] Laurens Van Der Maaten. Learning parametric embedding by preserving local structure. PMIR, 2009.
[72] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In ICLR, 2022.
[73] Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang. Linyi Yang. Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj. Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. USB: A unified semi-supervised learning benchmark for classification. In NeurIPS, 2022.
[74] Yu Wang, Pengehong Qiao, Chang Liu, Guoli Song, Xiawu Zheng, and Jie Chen. Out-of- distributed semantic pruning for robust semi- supersuuee learning. In CVPR, 2023.
[75] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In NerulPS, 2019.
[76] Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmen- tation with prototype-based consistency regularization. NewrlPS, 2022.
[77] Jun Yu, Liwen Zhang, Shenshen Du, Hao Chang, Keda Lu, Zhong Zhang, Ye Yu, Lei Wang, and Qiang Ling. Pseudo-label generation and various data augmentation for semi-supervised hyperspectral object detection. In CVPRW, 2022.
[78] Arman Zharmagambetto and Miguel A. Carreira- Pereiia-errrina.. Semi Semi- supervised learning with decision trees: Graph laplacian tree alternating optimization. In NeurIPS, 2022.
[79] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighbor- hood contrastive learning for novel class discovery. In CVPR, 2021.
[80] Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories an open world, In CVPR, 2021.
14


Reviews:
[{"review_id": "lR7L23swLt", "reviewer": "Reviewer_qAVC", "summary": "In this paper, a new Taxonomic context pIrors Discovering and Aligning (TIDA) which exploits the relationship of samples under various granularity is proposed. TIDA comprises two key components: i) A taxonomic context discovery module that constructs a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors; ii) A taxonomic context-based prediction alignment module that enforces consistency across hierarchical predictions to build the reliable relationship between classes among various granularity and provide additions supervision. ", "questions": "Is it possible to set a init value of $\\alpha$ and $\\beta$, and then determine the specific value by dynamically decreasing or increasing?", "limitations": "The authors addressed the limitations. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors identify the importance of multi-granularity priors in the context of OSSL and introduce the taxonomic context priors for solving the OSSL problem.\n2. A uniformed OSSL framework, which can discover taxonomic context priors without any extra supervision is proposed. With the proposed cross-hierarchical prediction alignment, the framework can effectively enhance the performance of the model.\n3. This paper is easy to follow.", "weaknesses": "The hyper-parameter $\\alpha$ and $\\beta$ depends on the specific datasets and for different datasets, experiments need to be conducted to determine the value. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments! Please find our detailed response below. \n>**Q1:** The hyperparameters $\\alpha$ and $\\beta$ depend on the specific datasets, and experiments need to be conducted to determine their values for different datasets.\n>\n>**A1:** To avoid over-tuning parameters, we only conduct experiments on one generic dataset and one fine-grained dataset to determine the values of $\\alpha$ and $\\beta$. On the other hand, we find that the trends of $\\alpha$ and $\\beta$ are similar in all generic datasets or fine-grained datasets.\n>\n> Following your suggestion, we have provided the evaluation of $\\alpha$ and $\\beta$ on the other datasets, as shown below. We can observe that the values of $\\alpha$ and $\\beta$ that yield the best performance are similar for both generic datasets ( $\\alpha$ = 0.2, $\\beta$ = 2.0) and fine-grained datasets ($\\alpha$ = 0.4, $\\beta$ = 2.5) respectively. We thus use one dataset to determine the values of $\\alpha$ and $\\beta$ for the generic (or fine-grained) setting and apply the same values of $\\alpha$ and $\\beta$ to all generic (or fine-grained) datasets. Please refer to L234-L235 in the original paper.\n>\n>**i. The ablation analysis for $\\alpha$**\n>| Dataset Type |  Dataset | 0.1 | 0.2 | 0.4 | 0.6 | 0.8\n>| :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n>Generic | CIFAR100 | 63.4| **65.3** | 64.7 | 61.3 | 59.3\n>Generic |  Tiny ImageNet | 35.8| **37.2** | 35.2 | 34.2 | 33.9\n>Fine-Grained  |  Standard Cars | 64.2 | 65.2 | **66.1** | 62.7 | 62.1\n>Fine-Grained  |  Oxford-IIIT Pets | 58.4 | 59.4 | **59.9** | 58.6 | 54.6\n>\n>**ii. The ablation analysis for $\\beta$**\n>| Dataset Type |  Dataset | 1.25 | 1.5 | 1.75| 2.0| 2.5| 3.0\n>| :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n>Generic | CIFAR100 | 61.9 | 62.1 | 63.6 | **65.3** | 61.7| 61.8\n>Generic |  Tiny ImageNet |33.9 | 34.0 | 35.6 | **37.2** | 36.7 | 34.2\n>Fine-Grained  |  Standard Cars |62.5 | 64.9 | 65.9 | 65.4| **66.1** | 65.3\n>Fine-Grained  |  Oxford-IIIT Pets | 54.1|55.3 | 58.3 | **60.2** | 59.9 | 59.4\n\n>**Q2:** Is it possible to dynamically decrease or increase the value of $\\alpha$ and $\\beta$ during training?\n>\n> **A2:** Good question. Indeed, dynamically changing the values of $\\alpha$ and $\\beta$ during training can pose a **challenging optimization problem** for the model. \n> * Specifically, our TIDA applies two additional classifiers to classify samples at the super and sub-class levels, respectively. The shapes/dimensions of these classifiers are determined by the values of $\\alpha$ and $\\beta$. \n> * If we dynamically alter $\\alpha$ and $\\beta$ during training, we need to reinitialize the parameters after each change. This prevents the model from effectively utilizing the knowledge acquired by the super- and sub-class level classifiers during previous training iterations. \n> * Furthermore, the constraint between the continuously learned target-level classifier and the repeatedly re-initialized super- and sub-class level classifiers will also hinder the convergence of the model.\n>\n> Below, we conduct an experiment to dynamically change the values of $\\alpha$ and $\\beta$ on CIFAR100. We set the initial values of $\\alpha$ and $\\beta$ to 1. Then, we decrease $\\alpha$ by 0.2 (or increase $\\beta$ by 0.25) every 40 epochs, for a total of 200 epochs. We can find that the results, particularly on the novel class, are worse. Nevertheless, the above optimization problem only appears when we use the static network, and we will explore this interesting idea in the future. \n>\n>|  | seen | novel | all\n>| :----: | :----: | :----: | :----: |\n>| TRSSL  | 68.5 | 52.1 | 60.3\n>|Our w/ fixed $\\alpha$/$\\beta$ | **73.3** | **56.6** | **65.7** |\n>|Our w/ Dynamical $\\alpha$/$\\beta$ | 40.1 | 8.9 | 5.4 |\n"}, {"review_id": "E1j3jaoZAA", "reviewer": "Reviewer_k9he", "summary": "This paper tackles open-world semi-supervised learning and proposes to use multi-granularity labels as taxonomic context priors to leverage hierarchical supervision to enhance representation learning and improve the quality of pseudo labels. A taxonomic context discovery module is used to construct hierarchical prototypes; a taxonomic context-based prediction alignment module is applied to enforce consistency across multi-granularity predictions. ", "questions": "1. The authors need to analyze the learned hierarchical tree of multi-granularity labels of each dataset. It could be the case that each super-class may contain different numbers of target-classes, and also each target-class may contain different numbers of sub-classes. Especially when the target-classes of one dataset are imbalanced in terms of similarity in semantic space. \n\n2. The learning of multi-granularity pseudo-labels could be heavily-biased when the class-distribution is imbalanced and the semantic distance of each target-class is imbalanced. ", "limitations": "The number of novel classes needs to be known. The class-distribution needs to be balanced. These assumptions make TIDA not robust for practical usage. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation and idea of this paper are clear and well-explained. Multi-granularity supervision is introduced to improve the representation of base and novel classes. The alignment among each granularity is enforced.", "weaknesses": "1. Additional baselines are needed for comparison: \n\n[1] Pu N, Zhong Z, Sebe N. Dynamic Conceptional Contrastive Learning for Generalized Category Discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 7579-7588.\n\n[2] Zhang S, Khan S, Shen Z, et al. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3479-3488.\n\n[3] Vaze S, Han K, Vedaldi A, et al. Generalized category discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7492-7501.\n\n[4] Wen X, Zhao B, Qi X. A Simple Parametric Classification Baseline for Generalized Category Discovery[J]. arXiv preprint arXiv:2211.11727, 2022.\n\n[5] Chiaroni F, Dolz J, Masud Z I, et al. Mutual information-based generalized category discovery[J]. arXiv preprint arXiv:2212.00334, 2022.\n\n2. The actual number of classes must be known. Otherwise, it is difficult to determine how many classes should be defined for sub-class, target-class, and super-class, respectively. \n\n3. The real composition of the sub-class and super-class may not go exactly as designed. It is difficult to guarantee each sub-class only contains samples of one target class, and each target-class only belongs to one super-class.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable comments! Please find our detailed response below.\n>**Q1:** Compare with additional baselines.\n>\n>**A1:** Below, we compare our method with [1-5]. Our method achieves the best results on all datasets, as shown below.  For a fair comparison, following [1-5], we also use ViT as the backbone and incorporate contrastive learning during training. Here, we report the accuracy of all classes. We have included these results in the new version.\n>\n>| Methods|CIFAR10|CIFAR100|Aircraft| SCar|\n>|:----:|:----:|:----:|:----:|:----:| \n>DCCL[1]|96.3|75.3|-|43.1|\n>PromptCAL[2]|97.9|81.2|52.2|50.2\n>GCD[3]|91.5|73.0|45.0|39.0\n>SimGCD[4]|97.1|80.1|54.2|53.8\n>PIM[5]|94.7|78.3|-|43.1\n>Ours|**98.2**|**82.3**|**54.6**|**54.7**\n\n>**Q2:** The number of classes must be known.\n>\n> **A2:**  The number of classes does not need to be know.  On the one hand, the reason for assuming the number of novel classes as known in the main paper is to ensure a fair and comprehensive comparison with previous works [3,4,19,52] that all follow this assumption. On the other hand, we demonstrate that our TIDA still works well and achieves SOTA performance even when the number of classes is unknown, as shown in Tab. VI of the supplementary. \n\n>**Q3:** It is difficult to guarantee that each sub-class only contains samples of one target class, and that each target class only belongs to one super-class in real world.\n> \n>**A3:**  We agree with you that this assumption may not always hold true in the real world, but it is generally satisfied in most cases and beneficial to our TIDA. Thus, we propose TCA module and employ a relaxed strategy to enforcing prediction consistency across hierarchies and finally achieve high-quality consistency.\n> \n> Here, we discuss why the proposed assumption is beneficial to our TIDA:\n>\n> i). **The performance in TIDA is positively correlated with the consistency quality.** We calculate $\\frac{1}{n_2}\\sum_{i=1}^{n_2}\\max_{j=1}^{n_1} A_{i,j}^{super}$ and $\\frac{1}{n_2}\\sum_{i=1}^{n_2} \\max_{j=1}^{n_3} A_{j,i}^{sub}$ to quantitatively assess the consistencies of the super-target and sub-target on CIFAR100. Here, $A^{super}$ denotes the affinity matrix of the super-/target class level and the item $A^{sup}_{i,j}$ of $A^{sup}$ denotes the number of samples from the j-th target class that are classified into the i-th super-class. $A^{sub} \\in R^{n_2 \\times n_3}$ is similar for the sub-target levels (Refers to Fig.6 in the main paper). With the higher quality consistency, the TIDA achieves higher accuracy.\n> ||Super-Target Consistency|Sub-Target Consistency|Acc. \n>  |:----:|:----:|:----:|:----:|\n> TIDA |**72.3**|**69.2**|**65.3**\n> TIDA w/o TCA|65.7|61.3 |57.7\n> \n> ii). **The consistency assumption is necessary for our TIDA.** As shown in Tab.3 of our main paper, the taxonomic priors fails to improve performance without TCA. This is beacuse the inconsistency among multi-granularity classification leads to hard-optimization problem.\n> \n> In future, we will investigate the multiple mapping situation that better meet the real-world application.\n\n>**Q4:** The authors need to analyze the learned hierarchical tree for each dataset. Especially when the target classes of one dataset are imbalanced in terms of similarity in the semantic space.\n\n> **A4:** **i).** For each dataset in main paper, TIDA always captures the meaningful and balanced hierarchical structure. This is because TIDA assumes that both the superclass and subclass follow a class-balanced distribution. Please refer to the **Sec.E.2-E.4** in the supplementary.\n>\n> **ii).** For imbalanced similarity, TIDA still works well and tend to learn balanced distribution. But due to the imbalanced similarity, the learned super-classes repersents different grained semantics. For examples, a tiny dataset cotains 6 target classes (bed, chair, couch, wardrobe, baby, dog) and the number of super-classes is 3. The learned super-classes include #1 (chair, couch), #2 (wardrobe, bed), and #3 (baby, dog). The granularity of semantic expressed by super class #1/2 and #3 is different.\n> \n>  We also verify the effectiveness of TIDA in similarity-imbalanced setting. We randomly sample 90 classes from CIFAR100 to construct an imbalanced similarity in the semantic space (IS-CIFAR90), where the number of subclasses varies for each superclass. We report the average accuracy on all classes, as shown below. Results show that our TIDA still outperforms the TRSSL[52].\n>||TRSSL|Ours |\n>|:----:|:----:|:----:|\n>S-CIFAR90|51.8|**53.4**\n\n> **Q5:** The learning of pseudo-labels could be heavily-biased when the class-distribution is imbalanced and the semantic distance of target-class is imbalanced.\n> \n> **A5:** Good Question! The learning of multi-granularity pseudo-labels may be affected, but our TIDA still enables construct meaningful taxonomic priors to enhance performance with these imbalanced setting.\n>\n>  **i).** Imbalanced class distribution: Due to the assumption of class balance, the learning of multi-granularity pseudo-labels slight biased. **But the distribution of the pseudo-label on target class learned by TIDA is closer to the real distribution than the TRSSL [52].** \n>  \n> Here, we conduct experiments following the TRSSL[52], which involves an imbalanced class distribution setting (more details refer to [52]). We report the average accuracy on all classes and the KL diversity of pseudo-label and Gound-Truth, as shown below. Results show that our TIDA still outperforms TRSSL.\n>  \n> |Imbalance factors=10|Class imbalanced priors|KL|Acc.|\n> |:----:|:----: |:----:|:----:|\n> TRSSL|$\\checkmark$ |0.0020|41.0\n> TRSSL|$\\times$|0.0026|38.9\n> TIDA|$\\times$|0.0017|**43.6**\n> \n> **ii).** Imbalanced semantic distance: The pseudo-labels learned by TIDA is balanced with this setting. Please refer to **A4** for details.\n> \n> **iii).** We will further investigate these two mentioned settings in future."}, {"review_id": "YyV6LzKwBp", "reviewer": "Reviewer_Qvan", "summary": "Previous research has primarily focused on using pre-defined single-granularity labels as priors for recognizing novel classes. However, classes naturally adhere to a taxonomy, enabling classification at multiple levels of granularity and offering richer supervision through underlying relationships. To address this, this paper proposes TIDA (Taxonomic context pIrors Discovering and Aligning), a unified framework that leverages sample relationships across various levels of granularity. TIDA discovers multi-granularity semantic concepts as taxonomic context priors (e.g., sub-class, target-class, and super-class) to enhance representation learning and improve pseudo label quality. Extensive experiments on seven datasets demonstrate TIDA's significant performance improvement, achieving a new state-of-the-art.", "questions": "1. Can pre-defined taxonomic priors be utilized? Some datasets already have defined taxonomic priors, while WordNet can be used [1] to extract such priors among categories.\n\n[1] I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively. ICLR 2020.\n\n2. Have other cluster algorithms been explored for obtaining taxonomic priors, besides the SK algorithm?\n\n3. Could you provide further clarification on why this work is suitable for open-world learning? I can understand its benefits for generating high-quality pseudo labels in semi-supervised learning.\n\n4. Can you compare with [2] in detail?\n\n[2] Open-world Semi-supervised Novel Class Discovery. IJCAI 2023.", "limitations": "In conclusion, this paper exhibits a clear motivation and comprehensive experimental verifications. However, the necessity of constructing taxonomic priors remains a main concern. It would be beneficial if the authors could provide specific scenarios where the construction of taxonomic priors is indeed essential. In addition, the comparison with [2] is essential.\n\nThis paper fully considers potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and convincing. It is reasonable that taxonomic context priors are helpful to extract discriminative features.\n2. The writing is clear. I like the figures in this paper.\n3. Experiments effectively verify the effectiveness of this work. Comprehensive experimental results are provided in the supplementary material.", "weaknesses": "1. The necessity of constructing taxonomic priors remains unclear. Various methods, such as using WordNet [1], can extract taxonomic priors among categories. I think that there is insufficient evidence to support the essentiality of constructing taxonomic priors in typical scenarios.\n\n[1] I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively. ICLR 2020.\n\n2. Some works, especially [2], on the taxonomic structure lack proper citation and comparison.\n\n[2] Open-world Semi-supervised Novel Class Discovery. IJCAI 2023.\n\n3. The Introduction section lacks high-level statements regarding the construction and correctness of taxonomic priors.  need to find these information in the Methodology section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1&Q4:** Can pre-defined taxonomic priors be utilized? \n>\n> **A1:** We **CANNOT** directly construct taxonomic priors using language models, such as the mentioned WordNet[1]. This because we have no knowledge about novel classes in open-world SSL (neither semantic names nor attributes). \n> Thus, to leverage taxonomic knowledge in open-world SSL, it is crucial to design a method that can automatically discover/construct the taxonomic prior without using any extra supervision. In this regard, we introduce TIDA, which is specifically designed for this purpose using a novel cross-hierarchical prediction alignment approach.\n\n>**Q2&Q7:** Compare with [2].\n>\n> **A2:** Thanks for bringing this work to my attention.\n> \n> **i).** Motivation: [2] focuses on designing multiple learnable prototypes and aims to progressively group similar prototypes to discover novel classes. In fact, the prototypes and prototype groups involved can only be considered as an instance hierarchy and target-classes hierarchy, but not as taxonomic priors (sub-classes, target-classes, super-classes). In contrast, our TIDA applies a consistently hierarchical cluster learning approach and aims to simultaneously discover multi-granularity concepts as auxiliary supervision to facilicate discoverying novel classes.\n>\n> **i):** Method: [2] proposes a bi-level contrastive learning method to preserve the pairwise similarity between the prototypes and the prototype group levels, aiming to enhance representation learning. Instead, we construct a set of hierarchical prototypes in the latent space to uncover the underlying taxonomic context priors. In addition, we enforce consistency across hierarchical predictions through the proposed taxonomic context-based prediction alignment.\n>\n> **iii).** As shown in the table below, **our TIDA outperforms [2] on all datasets**.\n>| |CIFAR10 | CIFAR100 | ImageNet-100\n>|:----: |:----: | :----:|:----:|\n> OpenNCD[2]|85.3 |41.2 |73.2\n> Ours |**93.8**|**65.3**|**77.6**\n>\n>We would like to reminder that [2] was released after the NeurIPS deadline (22 May 2023 vs 17 May 2023). We will compare it with [2] in the next version.\n\n> **Q3:** The Introduction section lacks high-level statements regarding the **construction** and **correctness** of taxonomic priors. \n>\n> **A3:** Thanks for this constructive suggestion. This is the high-level statements: \" In this paper, we develop two modules: i) Taxonomic Context Discovery (TCD) module, which discovers the underlying taxonomic context priors by **constructing a set of hierarchical prototypes to cluster samples;** ii) Taxonomic Context-based prediction Alignment (TCA) module, which **enforces consistency across hierarchical predictions** to build reliable relationships between classes at various levels of granularity and provide additional supervision.\" We have included these statements in the revision.\n\n> **Q5:** Have other cluster algorithms been explored for obtaining taxonomic priors, besides the SK algorithm?\n> \n> **A5:** Good questions. In this paper, we adopt Sinkhorn as the online clustering algorithm due to its superiority, wide application, simplicity, and distribution-based learning strategy. Following your suggestion, we apply our approach on ORCA[25] and RankStats[24] and find that the model is hard to converge. The main reason may be that these two methods mainly reply on pair-wise similarity constraint to optimize classifier while our TIDA is constrained on the overall class distribution.\n\n> **Q6:** Could you provide further clarification on why this work is suitable for open-world learning?\n> \n> **A6:** The core challenge of open-set semi-supervised learning (OSSL) is the lack of supervision/labels (e.g., semantic names and attributes) for novel classes. To solve it, **the key is to exploit some priors (e.g. class distribution, pairwise similarity) as auxiliary supervision to recognize novel/seen classes,** which is the main goal of existing OSSL works.\n> \n> Here, we explain why this work is suitable for OSSL from three aspects:\n> **i).** The consistent taxonomic priors learned by TIDA provides more richer supervision:\n> \n> * Different from previous works with a single label hierarchy, TIDA first explores and utilizes consistent taxonomic priors as multi-granularity supervision. Given these priors, our TIDA is able to explore the relationship of samples under different granularity. \n> * For examples, TIDA enforce to distinguish these hard-to-distinguish samples ( e.g. *Telephone* and *Television*) in CIFAR100 with fine-grained priors, however TRSSL fails to classify them with single supervision. Moreover, TIDA use the coarse-grained priors to constrain samples from *Chair* and *Couch* to be closer and be far from *Baby*, as additional supervison. Refer to Figure IV,V in the supplementary.\n>\n> **ii).** TIDA provide a feasible way to automatically learn taxonomic priors without extra supervisions.\n> * We would like to emphasize that enforcing taxonomic consistency under open-world SSL is challenging due to the lack of information about novel classes, such as semantic names and attributes. Thus, we propose a unified OSSL framework that can discover taxonomic context priors and enforce cross-hierarchical consistency in an unsupervised manner.\n> \n> **iii).** More encouragingly, it can also serve as a visualization tool to promote the understanding of novel class, which have a far-reaching impact on the community."}, {"review_id": "poeAzh659H", "reviewer": "Reviewer_uA1P", "summary": "This paper addresses the open-world semi-supervised learning (OSSL) problem and proposes taxonomic context priors discovering and aligning (TIDA), which considers the taxonomic hierarchy of classes.\n\nBasically, the proposed method is based on [19], which assigns pseudo-labels to unlabeled samples by means of the Sinkhorn-Knopp algorithm. The proposed method is characterized by Taxonomic Context Discovery (TCD), which uses the sum of the losses at different hierarchical levels and Taxonomic Context-based Prediction Alignment (TCA), which adjusts the similarity between the feature and the class prototype by incorporating the weights based on the similarities between the prototypes at different hierarchical levels.\n\nExperimental results on seven datasets show that the proposed method achieves better accuracy than existing methods.\n", "questions": "Among the weaknesses I listed, (b) performance evaluation when the number of classes is unknown, (d) reasons for performance degradation when using TCP alone, and (e) discussion of the number of superclasses and subclasses for different datasets are of particular importance. I look forward to getting responses for these points from the authors.", "limitations": "Limitations are discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* This paper proposes an OSSL method focusing on class hierarchy, which has not been considered in the past as far as I know. (However, as discussed below, similar ideas have already been considered in related problems.)\n\n* The theoretical justification of the proposed method is discussed based on the principle of the EM algorithm (lower bound maximization of log-likelihood).\n\n* Exhaustive experiments using seven datasets show that the proposed method is consistently more accurate than existing methods.\n", "weaknesses": "a. The idea of discovering unknown classes by focusing on the class hierarchy has been explored in the past [46], so the high-level novelty would be somewhat limited.\n\nb. In this paper, the number of unknown classes is known, which is not realistic. The supplementary material shows the experimental results for the case where the number of unknown classes is unknown (Tables V and VI). However, only the results for the case with the best clustering accuracy are presented. I do not think this is realistic, since clustering accuracy is not always evaluable in practice.\n\nc. The base networks used in the experiments are limited to ResNet-18/-50. Testing with more modern networks such as ViT and vision-language models would be nice to emphasize the effectiveness of the proposed method.\n\nd. Table 3 shows that accuracy improves when both TCP and TCA are used, but always degrades when TCA is not used. The reason for this is not clear.\n\ne. The results in Figure 4 suggest that the appropriate number of super-classes and sub-classes may differ depending on the dataset. Justification is needed.\n\nf. Some typos: \n\nL10: pIrors -> prIors\n\nL199: Sinkhorn-Koppn -> Sinkhorn-Knopp\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1:** Compare with DCCL[46].\n>\n> **A1:** **i).** Motivation: DCCL[46] aims to enhance representation learning by applying contrastive learning among dynamically estimated conceptions. In contrast, our TIDA aims to enhance pseudo label quality by hierarchically clustering samples and discovering multi-granularity concepts as auxiliary supervision.\n>\n> **ii).** Method: DCCL[46] alternately estimates underlying visual concepts and applies a dual-level contrastive learning. However, our TIDA proposes constructing a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors. It also enforces consistency across hierarchical predictions and provides additional reliable supervision through taxonomic context-based prediction alignment. In fact, DCCL[46] only has a basic single-level supervision at a specific moment. Instead, our TIDA always maintains multi-granularity supervision as a taxonomic prior and ensures its consistency.\n>\n> **iii).** **TIDA outperforms DCCL[46] on all datasets**: In A3, we show that our TIDA clearly outperforms DCCL [46]. That is, jointly using multi-granularity supervision has not been studied in DCCL[46] and our TIDA is very different from DCCL[46].\n\n> **Q2:** When the class number is unknown, only the results for the case with the best clustering accuracy are presented. I do not think this is realistic, since clustering accuracy is not always evaluable in practice.\n>\n> **A2:** Sorry for the confusion. The reason for only presenting the case with the best clustering accuracy is to ensure a fair and comprehensive comparison. The strategy is widely used in previous works [1-3].\n>\n> Moreover, following your suggestion, we carry out comparisons with TRSSL[1] on CIFAR100 with more estimated number classes. The accuracy on all classes is reported, as shown below. Our TIDA always outperforms [1].\n> | Estimated number | 80 | 90 | 100 | 110 | 120\n> |:----: |:----: | :----:| :----:| :----:|:----:|\n> TRSSL[4] | 54.2 | 58.5 | 60.3 |56.9 | 54.1\n> Ours |61.1 | 53.4 | 65.3| 62.9 | 60.5\n>\n> [1] Towards realistic semi-supervised learning. In ECCV, 2022.\n>\n> [2] Generalized category discovery. In In CVPR, 2022\n>\n> [3] A unified objective for novel class discovery. In ICCV, 2021.\n>\n> **Q3:** Additional results using ViT16 as the backbone network.\n>\n> **A3:** Below, we compare our method with [4-8]. For a fair comparison, following [4-8], we also use ViT as the backbone and incorporate contrastive learning during training. We report the accuracy of all classes. Our method achieves the best results on all datasets. We have included these results in the revision.\n>\n> | Methods |CIFAR10 | CIFAR100 | Aircraft | SCar|\n> |:----: |:----: | :----:| :----:| :----:|\n> DCCL[4] | 96.3 | 75.3 | - |43.1 |\n> PromptCAL[5] | 97.9| 81.2 | 52.2 | 50.2\n> GCD[6] | 91.5 | 73.0 | 45.0 | 39.0\n> SimGCD[7] | 97.1 | 80.1 | 54.2 | 53.8\n> PIM[8] |94.7 |78.3 |- |43.1\n> Ours |**98.2** |**82.3** |**54.6** |**54.7**\n>\n>[4] Dynamic Conceptional Contrastive Learning for Generalized Category Discovery,CVPR 2023.\n>\n>[5] Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery, CVPR 2023.\n>\n>[6] Generalized category discovery, CVPR 2022.\n>\n>[7] A Simple Parametric Classification Baseline for Generalized Category Discovery, ICCV 2023.\n>\n>[8] Mutual information-based generalized category discovery, arXiv, 2022.\n\n>**Q4:** The reasons for performance degradation when using TCP alone.\n>\n>**A4:** The inconsistency among multi-granularity classification leads to hard-optimization problem when using TCP alone. Actually, TCP applies multiple classifiers to cluster samples under different granularities. It can be regarded as multi-task learning with a shared backbone. However, due to a lack of supervision (e.g., semantic names and attributes), when using TCP alone, the training objective among classifiers is inconsistent and the optimization on the sub- and sup-classifiers is completely unsupervised. This may have a negative impact on model optimization and consequently result in a performance degradation. To address this problem, we propose TCA to align the training objectives among classifiers and enable the sub- and sup-classifiers to benefit from the knowledge of labeled data on the target classifier.\n\n>**Q5:** The results in Figure 4 suggest that the appropriate number of super-classes and sub-classes may differ depending on the dataset. Justification is needed.\n>\n> **A5:** Following your suggestion, we have provided the justification of α and β on the other datasets, as shown below.\n> We can observe that the values of α and β that yield the best performance are similar for generic datasets (α = 0.2, β = 2.0) and fine-grained datasets (α = 0.4, β = 2.5) respectively. We thus use one dataset to determine the values of α and β for the generic (or fine-grained) setting and apply the same values of α and β to all generic (or fine-grained) datasets. Please refer to L234-L235 in the original paper.\n>\n> **i.** The ablation analysis for α.\n> | Dataset Type | Dataset | 0.1 | 0.2 | 0.4 | 0.6 | 0.8\n>| :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n>Generic | CIFAR100 | 63.4| **65.3** | 64.7 | 61.3 | 59.3\n>Generic | Tiny ImageNet | 35.8| **37.2** | 35.2 | 34.2 | 33.9\n>Fine-Grained | Standard Cars | 64.2 | 65.2 | **66.1** | 62.7 | 62.1\n>Fine-Grained | Oxford-IIIT Pets | 58.4 | 59.4 | **59.9** | 58.6 | 54.6\n>\n> **ii.** The ablation analysis for β.\n>| Dataset Type | Dataset | 1.25 | 1.5 | 1.75| 2.0| 2.5| 3.0\n>| :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n>Generic | CIFAR100 | 61.94 | 62.1 | 63.6 | **65.3** | 61.7| 61.8\n>Generic | Tiny ImageNet |33.9 | 34 | 35.6 | **37.2** | 36.7 | 34.2\n>Fine-Grained | Standard Cars |62.5 | 64.9 | 65.9| 65.4| **66.1** | 65.3\n>Fine-Grained | Oxford-IIIT Pets | 54.1|55.3 | 58.3 | **60.2** | 59.9 | 59.4"}]

Author rebuttal:
We sincerely thank the ACs and reviewers for their great effort in handling our paper.

We have appropriately addressed all concerns raised by the reviewers. These include providing more comparisons with recent baselines/backbones (Reviewer #k9he, #Qvan, #uA1Ps), conducting more ablation studies on the hyper-parameters $\alpha$ and $\beta$ (Reviewer #qAVC, #Qvan), discussing the realistic setting of our method (Reviewer #k9he,#uA1Ps), more quantitative evaluations of the learned hierarchical tree (Reviewer #k9he), and providing a clearer explanation of our motivation (Reviewer #Qvan).

Paper strengths acknowledged by reviewers:
+ The motivation and idea are **clear and convincing**. It is **reasonable** to assume that taxonomic context priors are helpful in extracting discriminative features. (Reviewer #qAVC, #k9he, #Qvan, #uA1Ps)

+ Experiments **effectively validate** the effectiveness of this work. **Comprehensive experimental results** are provided in the supplementary material. (Reviewer #qAVC, #k9he, #Qvan, #uA1Ps)

+ The paper is **well-written** and easy to understand. (Reviewer #qAVC, #k9he, #Qvan)

+ This paper proposes an OSSL method that focuses on class hierarchy, which has **not been considered** in the past. (Reviewer #uA1Ps)

+ The **theoretical justification** of the proposed method is discussed based on the principles of the EM algorithm (Reviewer #uA1Ps)

We expect ACs and reviewers to fully consider the following factors when making the final decision: (1) a novel taxonomic consistency learning framework for open-world SSL, with significant effectiveness and theoretical justification confirmed by the reviewers, (2) comprehensive responses to all the reviewers' comments, and (3) the source code opened to the reviewers (in the supplementary material).

Please let us know if you have any additional questions or concerns. We are happy to provide clarification.

Authors of Submission #2448


Meta-review: In this paper, the authors introduced TIDA, a framework for Open-world Semi-Supervised Learning (OSSL), which leverages multi-granularity semantic concepts to enhance representation learning and improve pseudo label quality. The extensive experiments demonstrate TIDA's significant accuracy improvements, surpassing the current state of the art, particularly by 6.9% on TinyImageNet. The experimental results and motivation in the paper are explained in detail, and the proposed method demonstrates a certain level of innovation. Following the rebuttal phase, all the reviewers have provided positive evaluations. Therefore, the AC recommends accepting this paper.
Decision: Accept (poster)