Title: Discover and Align Taxonomic Context Priors  for Open-world Semi-Supervised Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: open-world semi-supervised learning; novel class discovery;
Abstract: Open-world Semi-Supervised Learning (OSSL) is a realistic and challenging task, aiming to classify unlabeled samples from both seen and novel classes using partially labeled samples from the seen classes. 
Previous works typically explore the relationship of samples as priors on the pre-defined single-granularity labels to help novel class recognition. In fact, classes follow a taxonomy and samples can be classified at multiple levels of granularity, which contains more underlying relationships for supervision. We thus argue that learning with single-granularity labels results in sub-optimal representation learning and inaccurate pseudo labels, especially with unknown classes. In this paper, we take the initiative to explore and propose a uniformed framework, called Taxonomic context prIors Discovering and Aligning (TIDA), which exploits the relationship of samples under various granularity. It allows us to discover multi-granularity semantic concepts as taxonomic context priors (i.e., sub-class, target-class, and super-class), and then collaboratively leverage them to enhance representation learning and improve the quality of pseudo labels.
Specifically, TIDA comprises two components: i) A taxonomic context discovery module that constructs a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors; ii) A taxonomic context-based prediction alignment module that enforces consistency across hierarchical predictions to build the reliable relationship between classes among various granularity and provide additions supervision. We demonstrate that these two components are mutually beneficial for an effective OSSL framework, which is theoretically explained from the perspective of the EM algorithm. Extensive experiments on seven commonly used datasets show that TIDA can significantly improve the performance and achieve a new state of the art. The source codes are publicly available at https://github.com/rain305f/TIDA.

Discover and Align Taxonomic Context Priors for Open-world Semi-Supervised Learning

Abstract

Open-world Semi-Supervised Learning (OSSL) is a realistic and challenging task, aiming to classify unlabeled samples from both seen and novel classes using partially labeled samples from the seen classes. Previous works typically explore the relationship of samples as priors on the pre-defined single-granularity labels to help novel class recognition. In fact, classes follow a taxonomy and samples can be classified at multiple levels of granularity, which contains more underlying relationships for supervision. We thus argue that learning with single-granularity labels results in sub-optimal representation learning and inaccurate pseudo labels, especially with unknown classes. In this paper, we take the initiative to explore and propose a uniformed framework, called T axonomic context pr I ors D iscovering and A ligning (TIDA), which exploits the relationship of samples under various granularity. It allows us to discover multi-granularity semantic concepts as taxonomic context priors ( i.e. , sub-class, target-class, and super-class), and then collaboratively leverage them to enhance representation learning and improve the quality of pseudo labels. Specifically, TIDA comprises two components: i) A taxonomic context discovery module that constructs a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors; ii) A taxonomic context-based prediction alignment module that enforces consistency across hierarchical predictions to build the reliable relationship between classes among various granularity and provide additions supervision. We demonstrate that these two components are mutually beneficial for an effective OSSL framework, which is theoretically explained from the perspective of the EM algorithm. Extensive experiments on seven commonly used datasets show that TIDA can significantly improve the performance and achieve a new state of the art. The source codes are publicly available at https://github.com/rain305f/TIDA.

1 Introduction

Deep neural networks have obtained impressive performance on a variety of visual and language tasks [42, 48, 61, 15, 11, 12, 13, 14, 35, 36, 37, 45, 46]. However, their success is largely dependent

图

：层次性

1

+

突出多层交互（

align

）

v.s.

单层聚类；

Chair

Super

Sub

...

Target

Align

Align

Target

(a) Previous Single Taxonomy Methods

Various Taxonomic

Context Priors

Seen/Novel

Inputs

Model

Model

Seen/Novel

Inputs

Furniture

Chair

Wooden

Chairs

Chair

Bed

Couch

Sub1

Super

Furniture

Limited Single

Granularity  Priors

Chair

Couch

Bed

Acc.

Epoch

Epoch

Acc.

Pseudo-Label Acc.

58.15%

Pseudo-Label Acc.

65.67%

Sub2

CIFAR100

CIFAR100

Ours

Baseline

Ours

Baseline

60

50

40

30

20

10

0

0

20

09

50

40

30

20

10

0

0

(b) Contextual Taxonomy Method (Ours)

on abundant labeled data, where the acquisition is expensive and time-consuming [75]. One popular solution to this problem is Semi-Supervised Learning (SSL) [69, 34, 63, 54, 62, 44, 56], which aims to reduce annotation costs by supplementing a small amount of labeled training data with a large number of unlabeled data.

sub2 Despite their significant success in various tasks [68, 77, 21, 75, 69, 53, 78, 38, 73, 59] , most existing SSL methods are designed under the closed-set assumption, i.e. , the training and test data share the same class set. Recently, several works [64, 26, 54, 31, 74] extend the standard SSL to an open-world setting, where the unlabeled set contains samples from novel (unknown) classes that are not present in the labeled set. In Open-world SSL (OSSL), the model is required to identify samples from both known and novel classes with partial labeled sample from seen classes. Due to the lack of labeled samples for novel classes, it is vital to exploit some priors as auxiliary supervision to learn discriminative representations and classifiers for all classes. Existing works either exploit pairwise similarity prior [64, 5, 25] or class distribution prior [65] to achieve this. However, as shown in Fig. 1(a), these methods only explore priors at a single granularity, which suffer from sub-optimization and inaccurate pseudo labels, due to the limited supervision.

In this paper, we argue that leveraging multiple levels of granularity as semantic priors ( e.g. , subclasses, classes, and super-classes, etc.) is a more preferable solution for OSSL [43], which helps us utilize more underlying relationship to supplement simplex single supervision. As illustrated in Fig. 1(b), in the super-class granularity, we can treat Couch (seen) and Chair (novel) as belonging to the same super-class Furniture . This builds a relationship between seen and novel classes and constraints samples from Chair and Couch be closer [43]. Meanwhile, in the sub-class granularity, Chairs are over-clustered into different sub-classes according to the materials, helping us to distinguish some hard negative samples [24]. These motivate us to inject such taxonomic context into OSSL for producing more discriminative representation learning and accurate pseudo-labeling. To this end, we propose a novel OSSL framework, named T axonomic context pr I ors D iscovering and A ligning (TIDA), to explore the relationship of samples (or classes) under various granularity. Without auxiliary supervision, TIDA can automatically discover multi-granularity taxonomic context priors, which are then collaboratively leveraged to improve representation and classifier learning. Meanwhile, the alignment across granularity ensures consistent taxonomic context and additional supervision.

Specifically, we develop two modules: i) Taxonomic Context Discovery (TCD) module, which discovers the underlying taxonomic context priors by constructing a set of hierarchical prototypes to cluster samples; ii) Taxonomic Context-based prediction Alignment (TCA) module, which enforces consistency across hierarchical predictions to build reliable relationships between classes at various levels of granularity and provide additional supervision. These two modules are mutually beneficial for learning discriminative representation and accurate pseudo-labeling for OSSL. As shown in

Fig. 1(b), our method can largely improve the quality of the pseudo label over the baseline. To sum up, the main contributions are as follows:

We identify the importance of multi-granularity priors in the context of OSSL and introduce a new type of prior knowledge, i.e. , taxonomic context priors, for solving the OSSL problem.

We introduce a uniformed OSSL framework, which can discover taxonomic context priors without any extra supervision. With the proposed cross-hierarchical prediction alignment, our framework can effectively enhance the performance of the model.

Experiments conducted on seven visual benchmarks show that our TIDA achieves new state-ofthe-art results. Additionally, we provide a theoretical analysis with the EM algorithm to better understand the underlying mechanism of our approach.

2 Related work

Open-world Semi-Supervised Learning (OSSL). Semi-Supervised Learning (SSL) aims to learn informative semantics from unlabeled data to reduce the dependence on human annotations [10, 2, 1, 39, 7, 60, 33, 3, 9, 30, 49, 67, 16, 6, 76, 4]. However, these methods assume that the unlabeled and labeled samples come from the same distribution, which is impractical in real-world applications. To address this limitation, ORCA [5] extends SSL to the more realistic and challenging open world, which assumes the unlabeled data are from both novel and labeled (seen) classes. NCLPS [25] proposes to exploit pairwise similarity priors with distribution alignment and applies an adaptive threshold to synchronize the learning pace between seen/novel classes. OpenLDN [64] optimizes a pairwise similarity loss by a bi-level way, which exploits the information available in the labeled set as priors to implicitly cluster samples from novel class. Later, TRSSL [65] proposes a classdistribution-aware pseudo-label method for OSSL, which utilizes sample uncertainty and class distribution as priors to generate pseudo-labels. However, they still only focus on priors learned on single label-hierarchy, which lacks accurate modeling of multi-level semantic relations between seen/novel category samples. For a new perspective, TIDA applies a more preferable solution, which subdivide semantic concepts from coarse to fine as taxonomic context ( i.e. , sub-class, target-class, and super-class). With taxonomic context to describe seen/novel samples, it enjoys a number of desirable properties, including flexible encoding of label relations, predictions consistent with label relations [19]. Our model can significantly improve performance by exploiting the label relations.

Novel Class Discovery (NCD). NCD is a challenging open-world problem, which most closely relates to OSSL and aims at clustering unlabeled samples of novel classes with the guidance of knowledge from seen categories [70, 32, 80, 79, 50, 66, 51, 57]. RankStats [28] utilizes rank statistics to transfer the knowledge of the labeled classes to the problem of clustering the unlabelled images. UNO [23] first formulates the NCD problem into a transportation problem by Shinkhorn-Knopp algorithm [17]. Different from OSSL, NCD has a strong assumption that all the unlabeled samples belong to novel classes, thus NCD methods are not applicable to OSSL problems. In this paper, we adopt some NCD methods as comparisons, but the results show that they fail to jointly distinguish seen/novel samples.

3 Methodology

Problem Setup. Similar to standard closed-world SSL, the training data for OSSL consist of labeled data D l = { x l i , y l i } N l i =1 ∈ X × Y l and unlabeled data D u = { x u i , y u i } N u i =1 ∈ X × Y u , where x l i is the i -th labeled image with the label y l i , x u i is the i -th unlabeled image belonging to the class y u i that is not available during training, and N l < N u . Different from standard SSL, OSSL assumes that Y l ⊂ Y u . We aims to train encoder and a |Y u | -ways classifiers to classify all classes. We denote Y l as seen classes and Y u \ Y l as novel classes Y n , i.e. , Y n = Y u \ Y l . Following [5], we assume the number of novel classes |Y n | is known.

Baseline. Different from standard SSL, the pseudo-labeling process is inherently ill-posed in OSSL since there are no labeled samples for novel classes. To solve this problem, TRSSL [65] and UNO [23] design a unified self-labeling classification objective based on Shinkhorn-Knopp algorithm [17]. They view pseudo-labeling as an optimal transport problem that can generate reliable pseudo-labels for unlabeled data (more details refers to [65, 23]. The objective can be formulated as:

Align (TCA in Sec.3.2)

Furniture

Chair

Plastic/Leather Chair

Chair

Discovery (TCD in Sec.3.1)

Image Encoder

Seen

Novel

unsupervised cluster

coarse-grained

fine-grained

target-grained

semi-supervised cluster

unsupervised cluster

merge

Sub priors

split

Super priors

TCA

Loss

TCA

Loss

TCD

Loss

TCD

Loss

TCD

Loss

Prototypes

Classficial

Boundary

Samples

Estimated

Boundary by

TC priors

Estimated

Prototypes by

TC priors

where y i ∈ R |Y u | + |Y l | denotes the ground-truth / pseudo-label for labeled / unlabeled sample x i . ˆ y j i is the output predicted by the model, representing the probability of x i belongs to j -th class. z i represents the features of sample x i and c j represents the prototype in j -th class. S ( · , · ) is cosine similarity function. Note that to mitigate over-fitting on noisy pseudo-labels, both methods first obtain pseudo labels for two views of each unlabeled sample by applying various perturbations. Then, the classification objective in Eq. 1 for unlabeled data is accomplished by optimizing each view using the pseudo-labels from the other view. For simplify, we do not explicitly display this pseudo-label exchange process in Eq. 1. In this paper, we use this self-labeling strategy as the baseline. Differently, we propose to explore underlying taxonomic context priors as a remedy for single-granularity supervision. As shown in Fig. 2, TIDA consists of two mutually benefited modules: Taxonomic Context Discovery (TCD) and Taxonomic Context-based prediction Alignment (TCA). Next, we describe TIDA in detail.

3.1 Taxonomic Context Discovery

The lack of supervision for novel categories in OSSL poses a challenge for models to learn accurate and discriminative features for classification. Inspired by [58, 27, 43], we argue that the underlying taxonomic context of data facilitate novel class discovery, which can provide rich relationships between samples under different category granularity for supervision. To achieve this goal, we build several learnable hierarchical prototypes C = {{ c l i } n l i =1 } L l =1 upon a shared feature encoder f θ : X → Z , which are normalized and can be regarded as semantic priors under various category granularity. L is the number of hierarchies and n l is the number of prototypes of the l -th layer and n 1 < n 2 < ... < n L . Note that the number of prototypes n l controls the granularity of classification, where larger/smaller n l means more fine-/coarse-grained semantic concepts at hierarchy l .

Specifically, TIDA first obtains normalized feature representation z i ∈ R K = f θ ( x i ) for the i -th sample x i by image encoder f θ . Then, we use z i and prototypes on each hierarchy to perform clustering based on the baseline. The combination of loss functions on each hierarchy is written as:

where y l i ∈ { 0 , 1 } |Y l + Y u | is the pseudo-label of x i on l -th hierarchy generated by Sinkhorn-Knopp algorithm [17], ˆ y l i denotes the prediction of model for x i and τ = 0 . 1 is the temperature. In this

×

softmax

(1, GLYPH(cmap:d835)

GLYPH(cmap:d835))

(GLYPH(cmap:d835)

GLYPH(cmap:d835)

×

GLYPH(cmap:d835)2)

feature

represetation

(1

GLYPH(cmap:d835))

(GLYPH(cmap:d835)

×

GLYPH(cmap:d835)

GLYPH(cmap:d835)

)

(GLYPH(cmap:d835)

×

GLYPH(cmap:d835)

2

)

The target-grained

pseudo-label/label

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

2

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),2

TCA Loss

softmax

Split/Merge

(1, GLYPH(cmap:d835)

2

)

Matrix multiplication

...

...

GLYPH(cmap:d835)

GLYPH(cmap:d835),2

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),2|GLYPH(cmap:d835)

Super1

Super2

Coach

Dog

Chair

Sub1

Furniture

Chair

Coach

Sub1

Sub2

Super1

Dog

Animal

Align

Super2

Sub2

(a)

(b)

taxonomic

context

target-grained

prototype

(1, GLYPH(cmap:d835)

2

)

paper, we apply TIDA with three prototype hierarchies, i.e. , C = {{ c 1 i } n 1 i =1 , { c 2 i } n 2 i =1 , { c 3 i } n 3 i =1 } , n 1 = α ∗ n 2 , n 2 = |Y l | + |Y u | , n 3 = β ∗ | C 2 | and 0 < α < 1 , β > 1 , which can achieve consistent well performance on all settings. More analyses about L are provided in the Appendix.

Discussion. In fact, the objective of Eq. 2 is a hierarchical classification task, where the model requires distinguishing samples at different granularity. As shown in Fig. 2, the first hierarchy can be regarded as the super-class or coarse-grained class level; the second hierarchy refers to the target-class / target-grained level (i.e., |Y l + Y u | -ways classification); and the third can be regarded as the sub-class or fine-grained class level. We call the learned hierarchical clusters as taxonomic context priors because they reflect the inter-class or intra-class relationships among various granularity. However, with unknown class and limited labeled data, training each hierarchy individually cannot guarantee the taxonomic context consistency among hierarchies (see left part of Fig. 3(a)). That is, the relationships between labels among hierarchies are unclear and inaccurate, leading to limited advantages to target-classifier learning. With the consistent taxonomic context priors, we can build a clear subsumption relationship among classes of adjacent hierarchies and thus infer the predictions of a sample on a certain hierarchy based on its predictions on other hierarchies. For example, as shown in the right part of Fig. 3(a), given the affinities between the classes of different hierarchies, if we know the coarse-grained label of a sample is Furniture , we can infer that its target-grained label is most likely Chairs or Coach and definitely not Dog . Similarly, if we know the fine-grained label of a sample is Wooden Chairs , we can infer that its target-grained label is most likely Chairs and definitely not Coach or Dog . This indicates that the key to taxonomic context consistency is having reliable affinity relationships between labels of various granularity.

3.2 Taxonomic Context-based Prediction Alignment

To achieve taxonomic context consistency, we propose the Taxonomic Context-based prediction Alignment (TCA), as shown in Fig. 3(b), which aims to establish reliable affinity relationships across hierarchies. TCA is inspired by the statement of [19, 8]: given a sample and its label on a certain hierarchy l , we can treat its unobserved labels on adjacent hierarchies as latent variables and use them to infer the probability belonging to each class on the hierarchy l . Then, the tree hierarchy constraint can be achieved by minimizing the cross-entropy loss between the label/pseudo-label and the inferred prediction of the sample. To this end, we regard coarse/fine-grained predictions as latent variables and use them to infer target-grained predictions. This allows us to explicitly build affinity relationships across hierarchies and supervise them with partially labeled samples from seen classes. In addition to ensuring taxonomic context consistency, TCA also offers two more benefits: i) coarse-/fine-grained classifications can aid help predict target-grained classes [19, 8, 43]; ii) the supervision of labeled samples of seen classes can improve discrimination on coarse/fine-grained.

Specifically, given coarse/fine-grained prototypes C r and target-grained prototypes C 2 , we first obtain the affinity matrix M r, 2 ∈ R n r × n 2 , where for each m r, 2 i,j ∈ M r, 2 , m r, 2 i,j = S ( c r i , c 2 j ) , c r i is the i -th prototype in C r , c 2 j is the j -th one in C 2 , and r = 1 , 3 . Second, we estimate the taxonomic context-based similarity between the feature of each sample and the target-grained prototypes C 2

with C r as latent variables, as follows:

Then, we can obtain the taxonomic context-based prediction at the target-grained level by:

where r indicates the hierarchy of C r . Finally, we can enforce the taxonomic context consistency/alignment learning based on the cross-entropy loss and the pseudo-labels (or ground truth) on the target level, formulated as:

Finally, we jointly perform taxonomic context discovery and taxonomic context-based prediction alignment during the training process, i.e. , optimizing the model with the objective of L tcd + L tca .

4 Theoretical Analysis

Assuming that the observed data D ∪ D are related to some latent variables sets which refer to the hierarchical prototypes C = {{ c l i } n l i =1 } L l =1 above, where n l is the number of latent variables in set C l , l = 1 , · · · , L and n 1 < n 2 < · · · < n L . In our TIDA, L = 3, and n 2 = |Y l | + |Y u | . . According to Jensen's inequality [47], we write l u

From the EM [18] perspective, TIDA aims to maximize the likelihood of the observed N l + N u samples based on these hierarchical prototypes C the surrogate objective function as follows:

To sum up, TIDA aims to estimate the posterior class probability Q ( c l j ) = p ( c l j ; x i , θ ) on each hierarchy at the E-steps. Then, TIDA draws each sample to the prototype of its assigned cluster on each hierarchy at the M-step by optimizing Eq. 6 with known Q ( c l j ) .

Theorem 1. By executing EM algorithm iteratively, samples from the same class in the feature space will be mapped into a d -variate von Mises-Fisher (vMF) distribution whose probabilistic density is given by g ( x | c l i , κ ; θ ) = c d ( κ ) e κ c l ⊤ i f θ ( x ) , where | c l i | = 1 and c l i represents the mean direction, κ = 1 /τ is the concentration parameter, and c d ( κ ) is the normalization factor.

E-step. In fact, the posterior class probability Q ( c l i ) = p ( c l i ; x i , θ ) = I ( x i ∈ c l j ) , which is equivalent to y l i in Eq. 2. In this paper, TIDA uses the Sinkhorn-Knopp algorithm on each hierarchy to assign the samples to the cluster center, as mentioned in Sec. 3.

M-step. Given Q ( c l j ) = p ( c l j ; x i , θ ) , TIDA aims to maximize the surrogate function in Eq. 6 in Mstep. Finally, as Theorem. 1 shown, samples will be mapped into different d -variate von Mises-Fisher (vMF) distributions under different hierarchies [72].

(a) Separate Objective Functions for Each Hierarchy. Due to Eq. 6 is hard to optimize, following [47], we assume that the distribution around each prototype c l i satisfy an isotropic Gaussian, i.e. ,

where c i,l s denotes z i 's assigned prototype ( p ( c i,l s ; x i , θ ) = 1 and c i,l s ∈ C l ) and τ = 1 ∝ σ l i 2 , which plays the role of temperature parameter. As we can see, Eq. 7 is equivalent to Eq. 2.

(b) Constrain the Consistency across Hierarchies in Feature Space. Since only optimizing Eq. 7 results in inconsistencies of these learned vMF distributions across hierarchies, we propose to obtain

the aggregated latent variables ˜ C 2 | r with variables set C r , then constrain the distribution around ˜ C 2 | r be consistent with the original one around C 2 , where r = 1 , 3 .

First, we establish the similarity relationship across adjacent hierarchy, i.e. , the mapping matrix M r, 2 = S ( C r ⊤ , C 2 ) , where for m r, 2 [ j, k ] ∈ M r, 2 , m 2 ,r [ j, k ] = S ( c r j , c 2 k ) . Then we obtained its aggregated variables set ˜ C 2 | r with latent variables C r as ˜ C 2 | r = { ˜ c 2 | r k } n 2 k =1 , ˜ c 2 | r k = ∑ n 2 j =1 c r j m r, 2 [ j, k ] . Note, the variables set ˜ C 2 | r dynamically aggregates semantics from r -th hierarchy and builds communication among clusters across hierarchies, which facilitates aligning hierarchical semantics with target tasks.

Given aggregated variables, we then apply consistent constraints on the sample's distribution around ˜ C 2 | r and the original ones in the target hierarchy, where r = 1 , 3 . Specifically, we use aggregated variables ˜ C 2 | r to calculate the objective function in Eq. 7, as follows:

where the index s in ˜ c i, 2 | r s corresponds to the assigned target-grained prototype c i, 2 s for feature z i ( p ( c 2 s ; x i , θ ) = 1 ). As we can see, Eq. 8 is equivalent to Eq. 5. More details are in Appendix.

5 Experiment

5.1 Experiment Setup

Datasets. We evaluate TIDA on four commonly used generic image classification datasets (i.e. CIFAR10 [41], CIFAR100 [41], TinyImageNet [20] and ImageNet-100 [20]) and three fine-grained datasets (i.e. Oxford-IIT Pet [52], Standford-Cars [40] and Aircraft [55]). Following [5, 65], we use the first half of classes as seen classes and the remaining as novel. More details are in the Appendix.

Implementation Details. Following [65, 5, 64], we use ResNet-50 [22] for ImageNet-100 and ResNet-18 [22] for the other datasets. For all datasets, we set the length of prototypes set L as 3, in which C = { C 1 , C 2 , C 3 } and | C 2 | = |Y l ∪ Y u | . For the number of prototypes C 1 and C 3 , we set | C 1 | = α ∗| C 2 | and | C 3 | = β ∗| C 2 | . We set α = 0 . 2 / 0 . 4 and β = 2 / 2 . 5 for generic / fine-grained datasets. We use a cosine annealing-based learning rate scheduler accompanied by a linear warmup, where we set the base learning rate to 0.5 / 1.5 for generic / fine-grained datasets. The warmup length is set to 10 epochs and the weight decay is set to 1e-4. More details are in the Appendix.

5.2 Comparison with State-of-The-Art

We first compare our TIDA with state-of-the-art methods [5, 23, 28, 29, 64, 65] on four generic datasets and three fine-grained datasets, where the results are reported in Tab. 1 and Tab. 2, respectively. It is clear that TIDA consistently outperforms all state-of-the-art methods across all datasets and metrics. Specifically, TIDA surpasses the current best competitor TRSSL [65] by 6.9% on TinyImageNet for All classes. Importantly, TIDA outperforms previous methods on most datasets for Novel classes by a large margin, e.g. , 7.9% on TinyImageNet and 6.4% on Stanford-Cars. These results experimentally demonstrate that exploring taxonomic context priors as auxiliary supervision is a beneficial way for discriminating seen and novel classes under OSSL.

5.3 Ablation Study

Effect of Coarse-grained Taxonomic Context Priors. To verify the effectiveness of Coarse-grained Taxonomic Context Priors ( C-TCP ), we conduct a comparison with three variants: a) the baseline that only clusters samples on target-grained; b) the model that clusters samples on both coarse-grained and target-grained without using Taxonomic Context-based prediction Alignment ( TCA ); c) the model which simultaneously clusters samples both on coarse-grained and target-grained with TCA. As we can see in Tab. 3, c) outperforms a) and b) but b) worsened a), showing that generic semantics will improve the performance only when the coarse-grained clustering is consistent with the target task.

Effect of Fine-Grained Taxonomic Context. Similarly, we also conduct a comparison with three variants to verify the effectiveness of Fine-grained Taxonomic Context Priors ( F-TCP ), i.e. , a) the baseline, d) the model additionally includes F-TCP learning, and e) the model additionally includes F-TCP learning and TCA. Results show that F-TCP learning facilitates the target task only when using ATC. Moreover, the fine-grained semantic works relatively better than the generic semantic (see the comparison between c) and e)), especially the performance on Novel classes. In addition, jointly considering C-TCP and F-TCP could further improve the performance, as shown in variant h).

Effect of Taxonomic Context-based Prediction Alignment. As discussed in the previous two ablation studies, the proposed TCA is indispensable to our TIDA. Specifically, by comparing b) v.s. c), d) v.s. e), and g) v.s. h), we can observe that only using coarse-grained taxonomic context learning or fine-grained taxonomic context learning, or both of them fail to achieve improvement over the baseline. However, after injecting the proposed taxonomic context-based prediction alignment module, the performance is significantly improved. Because the inconsistency among multi-granularity classification leads to hard-optimization problem when using TCD alone. Actually, TCD applies multiple classifiers to cluster samples under different granularity. It can be regarded as multi-task learning with a shared backbone. However, due to a lack of supervision (e.g., semantic names and attributes), when using TCD alone, the training objective among classifiers is inconsistent and the optimization on the sub- and sup-classifiers is completely unsupervised. This may have a negative impact on model optimization and consequently result in performance degradation. While, the TCA learn to align the training objectives among classifiers and enable the sub- and sup-classifiers to benefit from the knowledge of labeled data on the target classifier.

(a) The analysis of

(b)The analysis of

Tiny ImageNet

Tiny ImageNet

Standard Car

Standard Car

41

69

41

69

39

67

39

67

37

65

37

65

CC

35

63

Acc

35

63

A

33

61

33

61

31

31

29

59

29

59

27

57

27

57

25

55

25

55

0.10.2 0.4 0.6 0.81.0

0.10.2 0.4 0.6 0.8 1.0

1.0 1.25 1.5 1.75 2.0 2.5 3.0

1.0 1.25 1.5 1.75 2.0 2.5 3.0

(a) TRSSL

(b) Ours

(c) Fined-Grained TC

#sub1

Wardrobe

with closed door

with open door

Furniture

Coach

Bed

Chair

Wardrobe

Effect of Alignment across Multiple Granularity. It is interesting to see whether the performance improvement comes from the alignment across multiple granularity or alignment across a number of classifiers. To evaluate this, we introduce a variant f) with three target-grained clusters learning, i.e. , | C 1 | = | C 2 | = | C 3 | = |Y l | + |Y u | . Compared with f) and h), we obverse that the alignment across multiple target-granularity is beneficial for Seen classes but there is no improvement or even reduction for Novel classes. This indicates that the improvement mainly comes from interaction among different levels of granularity instead of different classifiers of the same granularity.

5.4 Hyper-Parameter Analysis

Impact of the Number of Super-Classes. As shown in Fig. 4(a), it is more suitable to assign a smaller value for α , which leads to a small number of coarse-grained classes. When α approaches to 1, TIDA degenerates to the model that ignores the coarse-grained granularity. The best result is achieved by α = 0 . 2 / 0 . 4 for TinyImageNet / Standard Car.

Impact of the Number of Sub-Classes. In contrast, as shown in Fig. 4(b), assigning a larger value for β leads to a higher performance, which enables us to obtain fine-grained prior. Again, when β nears 1, TIDA will largely overlook the fined-grained granularity and produce lower results. The best performance is obtained by β = 2 / 2 . 5 for Tiny ImageNet / Standard Car.

5.5 Qualitative Analysis

T-SNE Visualization. We use t-SNE [71] to visualize the features learned by TIDA and the baseline [65]. As shown in Fig. 5(a) and Fig. 5(b), TIDA produces more discriminative features than TRSSL [65], where the samples are generally better clustered. To better understand our method, we also visualize the fine-grained and coarse-grained features learned by TIDA, shown in Fig. 5(c) and Fig. 5(d). At the fine-grained level, samples from the same categories are divided into different clusters; while, samples belonging to the same super-class are clustered together at the coarse-grained level. This further verifies that TIDA can learn more semantic priors which are complementary.

Visualization of Affinity Matrix. In Fig. 6, we calculate the affinity matrix by counting the number of samples belonging to each target class that are classified into each super-class. Results show that samples of the same target class are commonly classified into the same super-class for TIDA w/ TCA. In addition, we find that similar target classes are generally mapped into the same super-class. For example, 'bowl, cup, lamp, plate, clock' are assigned with the 18th super-class. In contrast, for TIDA w/o TCA, samples of the same target class tend to be classified into different super-classes, resulting in unclear and inconsistent affinity relationships among classes across hierarchies. These results validate the effectiveness of our TCA in establishing a clear and consistent affinity relationship.

0

1

super-classes

target classes

(a) w/o

super-classes

target classes

(b) w/

#15

#12

#17

Clock

Bowl

Cup

Lamp

Plate

#17

Clock

Bowl

Cup

Lamp

Plate

6

11

16

21

26

31

36

51

56

61

66

71

76

81

86

91

96

6

11

162126

31

36

61

66

71

76

81

86

91

96

6 Contribution

In this paper, we propose a novel framework known as TIDA. It can effectively learn discriminative visual representations and improve pseudo-label accuracy by exploring the underlying taxonomic context priors. Specifically, TIDA consists of two core components: i) clustering samples hierarchically to capture the underlying taxonomic context in latent space, and ii) constraining predictions across hierarchies to be consistent to build reliable affinity relationships to ensure taxonomic context is consistent. The theoretical analysis and extensive experiments on commonly used benchmarks with consistent performance gains both justify the superiority of our TIDA.

Limitations . Despite obtaining high performance, TIDA and existing methods still suffer from several limitations: assuming that i) the domain distribution between labeled and unlabeled data is the same; ii) the class-distribution is uniform; and iii) the number of novel classes is known. Future work may focus on exploring more practical OSSL situations that unlock the above constraints.

Acknowledgements. This work was supported in part by the National Key R&D Program of China (No.2022ZD0118201), the Natural Science Foundation of China (No. 61972217, 32071459, 62176249, 62006133, 62271465), the MUR PNRR project FAIR (PE00000013) funded by the NextGenerationEU, and the EU project Ai4Media (No. 951911).

References

David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In ICLR , 2019.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS , 2019.

Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In NeurIPS , 2022.

Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In NeurIPS , 2022.

Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In ICLR , 2022.

Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning. In NeurIPS , 2022.

Dong-Dong Chen, Wei Wang, Wei Gao, and Zhi-Hua Zhou. Tri-net for semi-supervised deep learning. In IJCAI , 2018.

Jingzhou Chen, Peng Wang, Jian Liu, and Yuntao Qian. Label relation graphs enhanced hierarchical residual network for hierarchical multi-granularity classification. In CVPR , 2022.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS , 2020.

Yanbei Chen, Massimiliano Mancini, Xiatian Zhu, and Zeynep Akata. Semi-supervised and unsupervised deep visual learning: A survey. TPAMI , 2022.

Xuxin Cheng, Bowen Cao, Qichen Ye, Zhihong Zhu, Hongxiang Li, and Yuexian Zou. Ml-lmcl: Mutual learning and large-margin contrastive learning for improving asr robustness in spoken language understanding. In ACL , 2023.

Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, and Yuexian Zou. M 3 st: Mix at three levels for speech translation. In ICASSP , 2023.

Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, and Yuexian Zou. Ssvmr: Saliency-based self-training for video-music retrieval. In ICASSP , 2023.

Xuxin Cheng, Zhihong Zhu, Ziyu Yao, Hongxiang Li, Yaowei Li, and Yuexian Zou. GhostT5: Generate More Features with Cheap Operations to Improve Textless Spoken Question Answering. In Proc. INTERSPEECH , 2023.

Zesen Cheng, Pengchong Qiao, Kehan Li, Siheng Li, Pengxu Wei, Xiangyang Ji, Li Yuan, Chang Liu, and Jie Chen. Out-of-candidate rectification for weakly supervised semantic segmentation. In CVPR , 2023.

Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay K. Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR , 2019.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS , 2013.

Arthur P. Dempster, Nan M. Laird, and Donald B. Rubin. Maximum likelihood from incomplete data via the em algorithm. J R Stat Soc B , 1977.

Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In ECCV , 2014.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Yue Fan, Dengxin Dai, Anna Kukleva, and Bernt Schiele. Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning. In CVPR , 2022.

Radhia Ferjaoui, Mohamed Ali Cherni, Fathia Abidi, and Asma Zidi. Deep residual learning based on resnet50 for COVID-19 recognition in lung CT images. In CoDIT , 2022.

Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In ICCV , 2021.

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. SCAN: learning to classify images without labels. In ECCV , 2020.

Lan-Zhe Guo, Yi-Ge Zhang, Zhi-Fan Wu, Jie-Jing Shao, and Yu-Feng Li. Robust semisupervised learning when not all classes have labels. In NeurIPS , 2022.

Lan-Zhe Guo, Zhen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. Safe deep semisupervised learning for unseen-class unlabeled data. In ICML , 2020.

Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. Hcsc: hierarchical contrastive selective coding. In CVPR , 2022.

Kai Han, Sylvestre-Alvise Rebuffi, Sébastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In ICLR , 2020.

Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In ICCV , 2019.

Zhongyi Han, Xian-Jin Gui, Chaoran Cui, and Yilong Yin. Towards accurate and robust domain adaptation under noisy environments. In IJCAI , 2020.

Rundong He, Zhongyi Han, Xiankai Lu, and Yilong Yin. Safe-student for safe deep semisupervised learning with unseen-class unlabeled data. In CVPR , 2022.

Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In ICLR , 2018.

Zhuo Huang, Chao Xue, Bo Han, Jian Yang, and Chen Gong. Universal semi-supervised learning. In NeurIPS , 2021.

Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In CVPR , 2019.

Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. Expectation-maximization contrastive learning for compact video-and-language representations. In NeurIPS , 2022.

Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In CVPR , 2023.

Peng Jin, Hao Li, Zesen Cheng, Jinfa Huang, Zhennan Wang, Li Yuan, Chang Liu, and Jie Chen. Text-video retrieval with disentangled conceptualization and set-to-set alignment. In IJCAI , 2023.

Ying Jin, Jiaqi Wang, and Dahua Lin. Semi-supervised semantic segmentation via gentle teaching assistant. In NeurIPS , 2022.

Juho Kannala, Alex Lamb, Kenji Kawaguchi, Vikas Verma, Yoshua Bengio, David LopezPaz, Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. arXiv , 2019.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV , 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Tront , 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Commun. of the ACM , 2017.

Aoxue Li, Tiange Luo, Zhiwu Lu, Tao Xiang, and Liwei Wang. Large-scale few-shot learning: Knowledge transfer with class hierarchy. In CVPR , 2019.

Gang Li, Xiang Li, Yujie Wang, Yichao Wu, Ding Liang, and Shanshan Zhang. DTG-SSOD: dense teacher guidance for semi-supervised object detection. In NeurIPS , 2022.

Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, and Jie Chen. Toward 3d spatial reasoning for human-like text-based visual question answering. TIP , 2022.

Hao Li, Peng Jin, Zesen Cheng, Songyang Zhang, Kai Chen, Zhennan Wang, Chang Liu, and Jie Chen. Tg-vqa: Ternary game of video question answering. In ICCV , 2023.

Junnan Li, Pan Zhou, Caiming Xiong, and Steven C.H. Hoi. Prototypical contrastive learning of unsupervised representations. In ICLR , 2021.

Kehan Li, Zhennan Wang, Zesen Cheng, Runyi Yu, Yian Zhao, Guoli Song, Chang Liu, Li Yuan, and Jie Chen. Acseg: Adaptive conceptualization for unsupervised semantic segmentation. In CVPR , 2023.

Yu-Feng Li and Zhi-Hua Zhou. Towards making unlabeled data never hurt. TPAMI , 2015.

Jiaming Liu, Yangqiming Wang, Tongze Zhang, Yulu Fan, Qinli Yang, and Junming Shao. Open-world semi-supervised novel class discovery. In IJCAI , 2023.

Mingxuan Liu, Subhankar Roy, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Large-scale pre-trained models are surprisingly strong in incremental novel class discovery. arXiv , 2023.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv , 2013.

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. TPAMI , 2018.

Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS , 2018.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR , 2012.

Yongri Piao, Chenyang Lu, Miao Zhang, and Huchuan Lu. Semi-supervised video salient object detection based on uncertainty-guided pseudo labels. In NeurIPS , 2022.

Nan Pu, Zhun Zhong, Xinyuan Ji, and Nicu Sebe. Federated generalized category discovery. arXiv , 2023.

Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In CVPR , 2023.

Pengchong Qiao, Zhidan Wei, Yu Wang, Zhennan Wang, Guoli Song, Fan Xu, Xiangyang Ji, Chang Liu, and Jie Chen. Fuzzy positive learning for semi-supervised semantic segmentation. In CVPR , 2023.

Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, and Alan L. Yuille. Deep co-training for semi-supervised image recognition. In ECCV , 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Zhongzheng Ren, Raymond A. Yeh, and Alexander G. Schwing. Not all unlabeled data are equal: Learning to weight data in semi-supervised learning. In NeurIPS , 2020.

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In ICLR , 2021.

Mamshad Nayeem Rizve, Navid Kardan, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Openldn: Learning to discover novel classes for open-world semi-supervised learning. In ECCV , 2022.

Mamshad Nayeem Rizve, Navid Kardan, and Mubarak Shah. Towards realistic semi-supervised learning. In ECCV , 2022.

Subhankar Roy, Mingxuan Liu, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Class-incremental novel class discovery. In ECCV , 2022.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NeurIPS , 2016.

Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeruIPS , 2020.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS , 2017.

Colin Troisemaine, Vincent Lemaire, Stéphane Gosselin, Alexandre Reiffers-Masson, Joachim Flocon-Cholet, and Sandrine Vaton. Novel class discovery: an introduction and key concepts. arXiv , 2023.

Laurens Van Der Maaten. Learning a parametric embedding by preserving local structure. PMLR , 2009.

Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In ICLR , 2022.

Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. USB: A unified semi-supervised learning benchmark for classification. In NeurIPS , 2022.

Yu Wang, Pengchong Qiao, Chang Liu, Guoli Song, Xiawu Zheng, and Jie Chen. Out-ofdistributed semantic pruning for robust semi-supervised learning. In CVPR , 2023.

Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. In NeruIPS , 2019.

Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmentation with prototype-based consistency regularization. In NeurIPS , 2022.

Jun Yu, Liwen Zhang, Shenshen Du, Hao Chang, Keda Lu, Zhong Zhang, Ye Yu, Lei Wang, and Qiang Ling. Pseudo-label generation and various data augmentation for semi-supervised hyperspectral object detection. In CVPRW , 2022.

Arman Zharmagambetov and Miguel A. Carreira-Perpinan. Semi-supervised learning with decision trees: Graph laplacian tree alternating optimization. In NeurIPS , 2022.

Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In CVPR , 2021.

Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In CVPR , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the open-world semi-supervised learning (OSSL) problem and proposes taxonomic context priors discovering and aligning (TIDA), which considers the taxonomic hierarchy of classes.\n\nBasically, the proposed method is based on [19], which assigns pseudo-labels to unlabeled samples by means of the Sinkhorn-Knopp algorithm. The proposed method is characterized by Taxonomic Context Discovery (TCD), which uses the sum of the losses at different hierarchical levels and Taxonomic Context-based Prediction Alignment (TCA), which adjusts the similarity between the feature and the class prototype by incorporating the weights based on the similarities between the prototypes at different hierarchical levels.\n\nExperimental results on seven datasets show that the proposed method achieves better accuracy than existing methods.\n\nSTRENGTHS:\n* This paper proposes an OSSL method focusing on class hierarchy, which has not been considered in the past as far as I know. (However, as discussed below, similar ideas have already been considered in related problems.)\n\n* The theoretical justification of the proposed method is discussed based on the principle of the EM algorithm (lower bound maximization of log-likelihood).\n\n* Exhaustive experiments using seven datasets show that the proposed method is consistently more accurate than existing methods.\n\nWEAKNESSES:\na. The idea of discovering unknown classes by focusing on the class hierarchy has been explored in the past [46], so the high-level novelty would be somewhat limited.\n\nb. In this paper, the number of unknown classes is known, which is not realistic. The supplementary material shows the experimental results for the case where the number of unknown classes is unknown (Tables V and VI). However, only the results for the case with the best clustering accuracy are presented. I do not think this is realistic, since clustering accuracy is not always evaluable in practice.\n\nc. The base networks used in the experiments are limited to ResNet-18/-50. Testing with more modern networks such as ViT and vision-language models would be nice to emphasize the effectiveness of the proposed method.\n\nd. Table 3 shows that accuracy improves when both TCP and TCA are used, but always degrades when TCA is not used. The reason for this is not clear.\n\ne. The results in Figure 4 suggest that the appropriate number of super-classes and sub-classes may differ depending on the dataset. Justification is needed.\n\nf. Some typos: \n\nL10: pIrors -> prIors\n\nL199: Sinkhorn-Koppn -> Sinkhorn-Knopp\n\nQUESTIONS:\nAmong the weaknesses I listed, (b) performance evaluation when the number of classes is unknown, (d) reasons for performance degradation when using TCP alone, and (e) discussion of the number of superclasses and subclasses for different datasets are of particular importance. I look forward to getting responses for these points from the authors.\n\nLIMITATIONS:\nLimitations are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious research has primarily focused on using pre-defined single-granularity labels as priors for recognizing novel classes. However, classes naturally adhere to a taxonomy, enabling classification at multiple levels of granularity and offering richer supervision through underlying relationships. To address this, this paper proposes TIDA (Taxonomic context pIrors Discovering and Aligning), a unified framework that leverages sample relationships across various levels of granularity. TIDA discovers multi-granularity semantic concepts as taxonomic context priors (e.g., sub-class, target-class, and super-class) to enhance representation learning and improve pseudo label quality. Extensive experiments on seven datasets demonstrate TIDA's significant performance improvement, achieving a new state-of-the-art.\n\nSTRENGTHS:\n1. The motivation is clear and convincing. It is reasonable that taxonomic context priors are helpful to extract discriminative features.\n2. The writing is clear. I like the figures in this paper.\n3. Experiments effectively verify the effectiveness of this work. Comprehensive experimental results are provided in the supplementary material.\n\nWEAKNESSES:\n1. The necessity of constructing taxonomic priors remains unclear. Various methods, such as using WordNet [1], can extract taxonomic priors among categories. I think that there is insufficient evidence to support the essentiality of constructing taxonomic priors in typical scenarios.\n\n[1] I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively. ICLR 2020.\n\n2. Some works, especially [2], on the taxonomic structure lack proper citation and comparison.\n\n[2] Open-world Semi-supervised Novel Class Discovery. IJCAI 2023.\n\n3. The Introduction section lacks high-level statements regarding the construction and correctness of taxonomic priors.  need to find these information in the Methodology section.\n\nQUESTIONS:\n1. Can pre-defined taxonomic priors be utilized? Some datasets already have defined taxonomic priors, while WordNet can be used [1] to extract such priors among categories.\n\n[1] I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively. ICLR 2020.\n\n2. Have other cluster algorithms been explored for obtaining taxonomic priors, besides the SK algorithm?\n\n3. Could you provide further clarification on why this work is suitable for open-world learning? I can understand its benefits for generating high-quality pseudo labels in semi-supervised learning.\n\n4. Can you compare with [2] in detail?\n\n[2] Open-world Semi-supervised Novel Class Discovery. IJCAI 2023.\n\nLIMITATIONS:\nIn conclusion, this paper exhibits a clear motivation and comprehensive experimental verifications. However, the necessity of constructing taxonomic priors remains a main concern. It would be beneficial if the authors could provide specific scenarios where the construction of taxonomic priors is indeed essential. In addition, the comparison with [2] is essential.\n\nThis paper fully considers potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles open-world semi-supervised learning and proposes to use multi-granularity labels as taxonomic context priors to leverage hierarchical supervision to enhance representation learning and improve the quality of pseudo labels. A taxonomic context discovery module is used to construct hierarchical prototypes; a taxonomic context-based prediction alignment module is applied to enforce consistency across multi-granularity predictions.\n\nSTRENGTHS:\n1. The motivation and idea of this paper are clear and well-explained. Multi-granularity supervision is introduced to improve the representation of base and novel classes. The alignment among each granularity is enforced.\n\nWEAKNESSES:\n1. Additional baselines are needed for comparison: \n\n[1] Pu N, Zhong Z, Sebe N. Dynamic Conceptional Contrastive Learning for Generalized Category Discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 7579-7588.\n\n[2] Zhang S, Khan S, Shen Z, et al. Promptcal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3479-3488.\n\n[3] Vaze S, Han K, Vedaldi A, et al. Generalized category discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 7492-7501.\n\n[4] Wen X, Zhao B, Qi X. A Simple Parametric Classification Baseline for Generalized Category Discovery[J]. arXiv preprint arXiv:2211.11727, 2022.\n\n[5] Chiaroni F, Dolz J, Masud Z I, et al. Mutual information-based generalized category discovery[J]. arXiv preprint arXiv:2212.00334, 2022.\n\n2. The actual number of classes must be known. Otherwise, it is difficult to determine how many classes should be defined for sub-class, target-class, and super-class, respectively. \n\n3. The real composition of the sub-class and super-class may not go exactly as designed. It is difficult to guarantee each sub-class only contains samples of one target class, and each target-class only belongs to one super-class.\n\nQUESTIONS:\n1. The authors need to analyze the learned hierarchical tree of multi-granularity labels of each dataset. It could be the case that each super-class may contain different numbers of target-classes, and also each target-class may contain different numbers of sub-classes. Especially when the target-classes of one dataset are imbalanced in terms of similarity in semantic space. \n\n2. The learning of multi-granularity pseudo-labels could be heavily-biased when the class-distribution is imbalanced and the semantic distance of each target-class is imbalanced.\n\nLIMITATIONS:\nThe number of novel classes needs to be known. The class-distribution needs to be balanced. These assumptions make TIDA not robust for practical usage.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
In this paper, a new Taxonomic context pIrors Discovering and Aligning (TIDA) which exploits the relationship of samples under various granularity is proposed. TIDA comprises two key components: i) A taxonomic context discovery module that constructs a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors; ii) A taxonomic context-based prediction alignment module that enforces consistency across hierarchical predictions to build the reliable relationship between classes among various granularity and provide additions supervision.

STRENGTHS:
1. The authors identify the importance of multi-granularity priors in the context of OSSL and introduce the taxonomic context priors for solving the OSSL problem.
2. A uniformed OSSL framework, which can discover taxonomic context priors without any extra supervision is proposed. With the proposed cross-hierarchical prediction alignment, the framework can effectively enhance the performance of the model.
3. This paper is easy to follow.

WEAKNESSES:
The hyper-parameter $\alpha$ and $\beta$ depends on the specific datasets and for different datasets, experiments need to be conducted to determine the value.

QUESTIONS:
Is it possible to set a init value of $\alpha$ and $\beta$, and then determine the specific value by dynamically decreasing or increasing?

LIMITATIONS:
The authors addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07