SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu Wang $^{1,6}$  Zhun Zhong $^{2}$  Pengchong Qiao $^{1,6}$  Xuxin Cheng $^{1}$  Xiawu Zheng $^{3,4}$
Chang Liu<sup>5</sup> Nicu Sebe<sup>7</sup> Rongrong Ji<sup>3,4</sup> Jie Chen<sup>1,3,6*</sup>
$^{1}$ School of Electronic and Computer Engineering, Peking University, Shenzhen, China  
 $^{2}$ School of Computer Science, University of Nottingham, United Kingdom
$^{3}$ Peng Cheng Laboratory, Shenzhen, China  $^{4}$  Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University  $^{5}$  Department of Automation, Tsinghua University, Beijing, China
$^{6}$  AI for Science (AI4S)-Preferred Program, Peking University Shenzhen Graduate School, China
$^{7}$ Department of Information Engineering and Computer Science, University of Trento, Italy
{rain_wang, pcqiao, chengxx}  $@$  stu.pku.edu.cn {zhengxw01, chenj}  $@$  pcl.ac.cn zhunzhong007@gmail.com rrji@xmu.edu.cn liuchang2022@tsinghua.edu.cn niculae.sebe@unitn.it
Open-world Semi-Supervised Learning (OSSL) is a realistic and challenging task, aiming to classify unlabeled samples from both seen and novel classes using partially labeled samples from the seen classes. Previous works typically explore the relationship of samples as priors on the pre-defined single-granularity labels to help novel class recognition. In fact, classes follow a taxonomy and samples can be classified at multiple levels of granularity, which contains more underlying relationships for supervision. We thus argue that learning with single-granularity labels results in sub-optimal representation learning and inaccurate pseudo labels, especially with unknown classes. In this paper, we take the initiative to explore and propose a uniformed framework, called Taxonomic context prIors Discovering and Aligning (TIDA), which exploits the relationship of samples under various granularity. It allows us to discover multi-granularity semantic concepts as taxonomic context priors (i.e., sub-class, target-class, and super-class), and then collaboratively leverage them to enhance representation learning and improve the quality of pseudo labels. Specifically, TIDA comprises two components: i) A taxonomic context discovery module that constructs a set of hierarchical prototypes in the latent space to discover the underlying taxonomic context priors; ii) A taxonomic context-based prediction alignment module that enforces consistency across hierarchical predictions to build the reliable relationship between classes among various granularity and provide additions supervision. We demonstrate that these two components are mutually beneficial for an effective OSSL framework, which is theoretically explained from the perspective of the EM algorithm. Extensive experiments on seven commonly used datasets show that TIDA can significantly improve the performance and achieve a new state of the art. The source codes are publicly available at https://github.com/rain305f/TIDA.
Deep neural networks have obtained impressive performance on a variety of visual and language tasks [42, 48, 61, 15, 11, 12, 13, 14, 35, 36, 37, 45, 46]. However, their success is largely dependent
Figure 1: (a) Previous works (e.g. TRSSL [65]) typically focus on constraining models on the pre-defined single granularity, resulting in inferior pseudo labels. (b) Our method explicitly explores the multi-granularity priors for model training, producing more accurate pseudo labels.
on abundant labeled data, where the acquisition is expensive and time-consuming [75]. One popular solution to this problem is Semi-Supervised Learning (SSL) [69, 34, 63, 54, 62, 44, 56], which aims to reduce annotation costs by supplementing a small amount of labeled training data with a large number of unlabeled data.
Despite their significant success in various tasks [68, 77, 21, 75, 69, 53, 78, 38, 73, 59], most existing SSL methods are designed under the closed-set assumption, i.e., the training and test data share the same class set. Recently, several works [64, 26, 54, 31, 74] extend the standard SSL to an open-world setting, where the unlabeled set contains samples from novel (unknown) classes that are not present in the labeled set. In Open-world SSL (OSSL), the model is required to identify samples from both known and novel classes with partial labeled sample from seen classes. Due to the lack of labeled samples for novel classes, it is vital to exploit some priors as auxiliary supervision to learn discriminative representations and classifiers for all classes. Existing works either exploit pairwise similarity prior [64, 5, 25] or class distribution prior [65] to achieve this. However, as shown in Fig. 1(a), these methods only explore priors at a single granularity, which suffers from sub-optimization and inaccurate pseudo labels, due to the limited supervision.
In this paper, we argue that leveraging multiple levels of granularity as semantic priors (e.g., subclasses, classes, and super-classes, etc.) is a more preferable solution for OSSL [43], which helps us utilize more underlying relationship to supplement simplex single supervision. As illustrated in Fig. 1(b), in the super-class granularity, we can treat Couch (seen) and Chair (novel) as belonging to the same super-class Furniture. This builds a relationship between seen and novel classes and constraints samples from Chair and Couch be closer [43]. Meanwhile, in the sub-class granularity, Chairs are over-clustered into different sub-classes according to the materials, helping us to distinguish some hard negative samples [24]. These motivate us to inject such taxonomic context into OSSL for producing more discriminative representation learning and accurate pseudo-labeling. To this end, we propose a novel OSSL framework, named Taxonomic context prIors Discovering and Aligning (TIDA), to explore the relationship of samples (or classes) under various granularity. Without auxiliary supervision, TIDA can automatically discover multi-granularity taxonomic context priors, which are then collaboratively leveraged to improve representation and classifier learning. Meanwhile, the alignment across granularity ensures consistent taxonomic context and additional supervision.
Specifically, we develop two modules: i) Taxonomic Context Discovery (TCD) module, which discovers the underlying taxonomic context priors by constructing a set of hierarchical prototypes to cluster samples; ii) Taxonomic Context-based prediction Alignment (TCA) module, which enforces consistency across hierarchical predictions to build reliable relationships between classes at various levels of granularity and provide additional supervision. These two modules are mutually beneficial for learning discriminative representation and accurate pseudo-labeling for OSSL. As shown in
Fig. 1(b), our method can largely improve the quality of the pseudo label over the baseline. To sum up, the main contributions are as follows:
- We identify the importance of multi-granularity priors in the context of OSSL and introduce a new type of prior knowledge, i.e., taxonomic context priors, for solving the OSSL problem.
- We introduce a uniformed OSSL framework, which can discover taxonomic context priors without any extra supervision. With the proposed cross-hierarchical prediction alignment, our framework can effectively enhance the performance of the model.
- Experiments conducted on seven visual benchmarks show that our TIDA achieves new state-of-the-art results. Additionally, we provide a theoretical analysis with the EM algorithm to better understand the underlying mechanism of our approach.
Open-world Semi-Supervised Learning (OSSL). Semi-Supervised Learning (SSL) aims to learn informative semantics from unlabeled data to reduce the dependence on human annotations [10, 2, 1, 39, 7, 60, 33, 3, 9, 30, 49, 67, 16, 6, 76, 4]. However, these methods assume that the unlabeled and labeled samples come from the same distribution, which is impractical in real-world applications. To address this limitation, ORCA [5] extends SSL to the more realistic and challenging open world, which assumes the unlabeled data are from both novel and labeled (seen) classes. NCLPS [25] proposes to exploit pairwise similarity priors with distribution alignment and applies an adaptive threshold to synchronize the learning pace between seen/novel classes. OpenLDN [64] optimizes a pairwise similarity loss by a bi-level way, which exploits the information available in the labeled set as priors to implicitly cluster samples from novel class. Later, TRSSL [65] proposes a class-distribution-aware pseudo-label method for OSSL, which utilizes sample uncertainty and class distribution as priors to generate pseudo-labels. However, they still only focus on priors learned on single label-hierarchy, which lacks accurate modeling of multi-level semantic relations between seen/novel category samples. For a new perspective, TIDA applies a more preferable solution, which subdivide semantic concepts from coarse to fine as taxonomic context (i.e., sub-class, target-class, and super-class). With taxonomic context to describe seen/novel samples, it enjoys a number of desirable properties, including flexible encoding of label relations, predictions consistent with label relations [19]. Our model can significantly improve performance by exploiting the label relations.
Novel Class Discovery (NCD). NCD is a challenging open-world problem, which most closely relates to OSSL and aims at clustering unlabeled samples of novel classes with the guidance of knowledge from seen categories [70, 32, 80, 79, 50, 66, 51, 57]. RankStats [28] utilizes rank statistics to transfer the knowledge of the labeled classes to the problem of clustering the unlabelled images. UNO [23] first formulates the NCD problem into a transportation problem by Shinkhorn-Knopp algorithm [17]. Different from OSSL, NCD has a strong assumption that all the unlabeled samples belong to novel classes, thus NCD methods are not applicable to OSSL problems. In this paper, we adopt some NCD methods as comparisons, but the results show that they fail to jointly distinguish seen/novel samples.
Problem Setup. Similar to standard closed-world SSL, the training data for OSSL consist of labeled data  $\mathcal{D}^l = \{x_i^l,y_i^l\}_{i = 1}^{N_l}\in \mathcal{X}\times \mathcal{Y}^l$  and unlabeled data  $\mathcal{D}^u = \{x_i^u,y_i^u\}_{i = 1}^{N_u}\in \mathcal{X}\times \mathcal{Y}^u$ , where  $x_{i}^{l}$  is the  $i$ -th labeled image with the label  $y_{i}^{l}$ ,  $x_{i}^{u}$  is the  $i$ -th unlabeled image belonging to the class  $y_{i}^{u}$  that is not available during training, and  $N_{l} < N_{u}$ . Different from standard SSL, OSSL assumes that  $\mathcal{V}^l\subset \mathcal{V}^u$ . We aims to train encoder and a  $|\mathcal{V}^u|$ -ways classifiers to classify all classes. We denote  $\mathcal{V}^l$  as seen classes and  $\mathcal{V}^u\setminus \mathcal{V}^l$  as novel classes  $\mathcal{V}^n$ , i.e.,  $\mathcal{V}^n = \mathcal{V}^u\setminus \mathcal{V}^l$ . Following [5], we assume the number of novel classes  $|\mathcal{V}^n|$  is known.
Baseline. Different from standard SSL, the pseudo-labeling process is inherently ill-posed in OSSL since there are no labeled samples for novel classes. To solve this problem, TRSSL [65] and UNO [23] design a unified self-labeling classification objective based on Shinkhorn-Knopp algorithm [17]. They view pseudo-labeling as an optimal transport problem that can generate reliable pseudo-labels for unlabeled data (more details refers to [65, 23]. The objective can be formulated as:
$$
\mathcal {L} _ {c e} = - \sum_ {j = 1} ^ {| \mathcal {Y} ^ {l} | + | \mathcal {Y} ^ {u} |} y _ {i} ^ {j} \log \hat {y} _ {i} ^ {j}, \quad \hat {y} _ {i} ^ {j} = p (j; x _ {i}) = \frac {\exp (\mathcal {S} \left(z _ {i} , c _ {j}\right))}{\sum_ {m = 1} ^ {| \mathcal {Y} ^ {l} | + | \mathcal {Y} ^ {u} |} \exp (\mathcal {S} \left(z _ {i} , c _ {m}\right))}, \tag {1}
$$
Figure 2: The overall architecture of TIDA. First, we extract features using an image encoder. Then, we use the proposed Taxonomic Context Discovery (TCD) to build hierarchical prototypes to cluster features under various granularity by optimizing TCD losses  $\mathcal{L}_{tcd}$ . Given the discovered taxonomic context (i.e. hierarchical prototypes), the proposed Taxonomic Context-based prediction Alignment (TCA) estimates taxonomic context-based predictions on target hierarchy by splitting/merging cluster results on coarse/fine-grained into the target. Last, TCA constrains the estimated predictions to be consistent with the original one on the target hierarchy with TCA losses  $\mathcal{L}_{tca}$ . During inference, the encoder and target-grained prototypes are applied to classify seen/novel class samples. Colors indicate different classes for samples or different granularity for prototypes.
where  $y_{i} \in \mathbb{R}^{|\mathcal{Y}^{u}| + |\mathcal{Y}^{l}|}$  denotes the ground-truth / pseudo-label for labeled / unlabeled sample  $x_{i}$ .  $\hat{y}_{i}^{j}$  is the output predicted by the model, representing the probability of  $x_{i}$  belongs to  $j$ -th class.  $z_{i}$  represents the features of sample  $x_{i}$  and  $c_{j}$  represents the prototype in  $j$ -th class.  $S(\cdot ,\cdot)$  is cosine similarity function. Note that to mitigate over-fitting on noisy pseudo-labels, both methods first obtain pseudo labels for two views of each unlabeled sample by applying various perturbations. Then, the classification objective in Eq. 1 for unlabeled data is accomplished by optimizing each view using the pseudo-labels from the other view. For simplify, we do not explicitly display this pseudo-label exchange process in Eq. 1. In this paper, we use this self-labeling strategy as the baseline. Differently, we propose to explore underlying taxonomic context priors as a remedy for single-granularity supervision. As shown in Fig. 2, TIDA consists of two mutually beneficial modules: Taxonomic Context Discovery (TCD) and Taxonomic Context-based prediction Alignment (TCA). Next, we describe TIDA in detail.
3.1 Taxonomic Context Discovery
The lack of supervision for novel categories in OSSL poses a challenge for models to learn accurate and discriminative features for classification. Inspired by [58, 27, 43], we argue that the underlying taxonomic context of data facilitate novel class discovery, which can provide rich relationships between samples under different category granularity for supervision. To achieve this goal, we build several learnable hierarchical prototypes  $C = \{\{c_i^l\}_{i=1}^{n_l}\}_{l=1}^L$  upon a shared feature encoder  $f_\theta : \mathcal{X} \to \mathcal{Z}$ , which are normalized and can be regarded as semantic priors under various category granularity.  $L$  is the number of hierarchies and  $n_l$  is the number of prototypes of the  $l$ -th layer and  $n_1 < n_2 < \ldots < n_L$ . Note that the number of prototypes  $n_l$  controls the granularity of classification, where larger/smaller  $n_l$  means more fine-/coarse-grained semantic concepts at hierarchy  $l$ .
Specifically, TIDA first obtains normalized feature representation  $z_{i} \in \mathbb{R}^{K} = f_{\theta}(x_{i})$  for the  $i$ -th sample  $x_{i}$  by image encoder  $f_{\theta}$ . Then, we use  $z_{i}$  and prototypes on each hierarchy to perform clustering based on the baseline. The combination of loss functions on each hierarchy is written as:
$$
\mathcal {L} _ {t c d} = - \sum_ {l = 1} ^ {L} \sum_ {j = 1} ^ {n _ {l}} y _ {i} ^ {j, l} \log \hat {y} _ {i} ^ {j, l} = - \sum_ {l = 1} ^ {L} \sum_ {j = 1} ^ {n _ {l}} y _ {i} ^ {j, l} \log \frac {\exp \left(\mathcal {S} \left(z _ {i} , c _ {j} ^ {l}\right) / \tau\right)}{\sum_ {m = 1} ^ {n _ {l}} \exp \left(\mathcal {S} \left(z _ {i} , c _ {m} ^ {l}\right) / \tau\right)}, \tag {2}
$$
where  $y_{i}^{l}\in \{0,1\}^{|\mathcal{V}^{l} + \mathcal{V}^{u}|}$  is the pseudo-label of  $x_{i}$  on  $l$ -th hierarchy generated by Sinkhorn-Knopp algorithm [17],  $\hat{y}_i^l$  denotes the prediction of model for  $x_{i}$  and  $\tau = 0.1$  is the temperature. In this
Figure 3: (a) The left part denotes inconsistent taxonomic context, where the relationships between labels across different hierarchies are unclear and inaccurate. TCA can address this problem by explicitly building affinity relationships and aligning hierarchical predictions, resulting in a consistent taxonomic context. (b) The illustration of TCA. Given the taxonomic context and the target-grained prototype, we first calculate their affinity matrix. Then, we split/merged the prediction on other-grained into target-grained as Eq. 4. Finally, we apply the loss  $\mathcal{L}_{tca}$  constrains them to be consistent.
paper, we apply TIDA with three prototype hierarchies, i.e.,  $C = \{\{c_i^1\}_{i=1}^{n_1}, \{c_i^2\}_{i=1}^{n_2}, \{c_i^3\}_{i=1}^{n_3}\}$ ,  $n_1 = \alpha * n_2$ ,  $n_2 = |\mathcal{Y}^l| + |\mathcal{Y}^u|$ ,  $n_3 = \beta * |C^2|$  and  $0 < \alpha < 1$ ,  $\beta > 1$ , which can achieve consistent well performance on all settings. More analyses about  $L$  are provided in the Appendix.
Discussion. In fact, the objective of Eq. 2 is a hierarchical classification task, where the model requires distinguishing samples at different granularity. As shown in Fig. 2, the first hierarchy can be regarded as the super-class or coarse-grained class level; the second hierarchy refers to the target-class / target-grained level (i.e.,  $|\mathcal{Y}^l + \mathcal{Y}^u|$ -ways classification); and the third can be regarded as the sub-class or fine-grained class level. We call the learned hierarchical clusters as taxonomic context priors because they reflect the inter-class or intra-class relationships among various granularity. However, with unknown class and limited labeled data, training each hierarchy individually cannot guarantee the taxonomic context consistency among hierarchies (see left part of Fig. 3(a)). That is, the relationships between labels among hierarchies are unclear and inaccurate, leading to limited advantages to target-classifier learning. With the consistent taxonomic context priors, we can build a clear subsumption relationship among classes of adjacent hierarchies and thus infer the predictions of a sample on a certain hierarchy based on its predictions on other hierarchies. For example, as shown in the right part of Fig. 3(a), given the affinities between the classes of different hierarchies, if we know the coarse-grained label of a sample is Furniture, we can infer that its target-grained label is most likely Chairs or Coach and definitely not Dog. Similarly, if we know the fine-grained label of a sample is Wooden Chairs, we can infer that its target-grained label is most likely Chairs and definitely not Coach or Dog. This indicates that the key to taxonomic context consistency is having reliable affinity relationships between labels of various granularity.
3.2 Taxonomic Context-based Prediction Alignment
To achieve taxonomic context consistency, we propose the Taxonomic Context-based prediction Alignment (TCA), as shown in Fig. 3(b), which aims to establish reliable affinity relationships across hierarchies. TCA is inspired by the statement of [19, 8]: given a sample and its label on a certain hierarchy  $l$ , we can treat its unobserved labels on adjacent hierarchies as latent variables and use them to infer the probability belonging to each class on the hierarchy  $l$ . Then, the tree hierarchy constraint can be achieved by minimizing the cross-entropy loss between the label/pseudo-label and the inferred prediction of the sample. To this end, we regard coarse/fine-grained predictions as latent variables and use them to infer target-grained predictions. This allows us to explicitly build affinity relationships across hierarchies and supervise them with partially labeled samples from seen classes. In addition to ensuring taxonomic context consistency, TCA also offers two more benefits: i) coarse-/fine-grained classifications can aid help predict target-grained classes [19, 8, 43]; ii) the supervision of labeled samples of seen classes can improve discrimination on coarse/fine-grained.
Specifically, given coarse/fine-grained prototypes  $C^r$  and target-grained prototypes  $C^2$ , we first obtain the affinity matrix  $M^{r,2} \in \mathbb{R}^{n_r \times n_2}$ , where for each  $m_{i,j}^{r,2} \in M^{r,2}$ ,  $m_{i,j}^{r,2} = S(c_i^r, c_j^2)$ ,  $c_i^r$  is the  $i$ -th prototype in  $C^r$ ,  $c_j^2$  is the  $j$ -th one in  $C^2$ , and  $r = 1, 3$ . Second, we estimate the taxonomic context-based similarity between the feature of each sample and the target-grained prototypes  $C^2$
with  $C^r$  as latent variables, as follows:
$$
\mathcal {S} \left(z _ {i}, c _ {j} ^ {2}\right) = \sum_ {k = 1} ^ {n _ {r}} \mathcal {S} \left(z _ {i}, c _ {k} ^ {r}\right) \cdot m _ {k, j} ^ {r, 2} = \sum_ {k = 1} ^ {n _ {r}} \mathcal {S} \left(z _ {i}, c _ {k} ^ {r}\right) \cdot \mathcal {S} \left(c _ {k} ^ {r}, c _ {j} ^ {2}\right). \tag {3}
$$
Then, we can obtain the taxonomic context-based prediction at the target-grained level by:
$$
\hat {y} _ {i} ^ {j, 2 \mid r} = \frac {\exp \left(\sum_ {k = 1} ^ {n _ {r}} \mathcal {S} \left(z _ {i} , c _ {k} ^ {r}\right) / \tau \cdot \mathcal {S} \left(c _ {k} ^ {r} , c _ {j} ^ {2}\right)\right)}{\sum_ {m = 1} ^ {C l + C u} \exp \left(\sum_ {k = 1} ^ {n _ {r}} \mathcal {S} \left(z _ {i} , c _ {k} ^ {r}\right) / \tau \cdot \mathcal {S} \left(c _ {k} ^ {r} , c _ {m} ^ {2}\right)\right)}, \tag {4}
$$
where  $r$  indicates the hierarchy of  $C^r$ . Finally, we can enforce the taxonomic context consistency/alignment learning based on the cross-entropy loss and the pseudo-labels (or ground truth) on the target level, formulated as:
$$
\mathcal {L} _ {t c a} = \sum_ {r = 1, 3} ^ {| \mathcal {Y} ^ {l} | + | \mathcal {Y} ^ {u} |} \sum_ {j = 1} ^ {} y _ {i} ^ {j, 2} \log \hat {y} _ {i} ^ {j, 2 | r}. \tag {5}
$$
Finally, we jointly perform taxonomic context discovery and taxonomic context-based prediction alignment during the training process, i.e., optimizing the model with the objective of  $\mathcal{L}_{tcd} + \mathcal{L}_{tca}$ .
Assuming that the observed data  $\mathcal{D}^l\cup \mathcal{D}^u$  are related to some latent variables sets which refer to the hierarchical prototypes  $C = \{\{c_i^l\}_{i = 1}^{n_l}\}_{l = 1}^L$  above, where  $n_{l}$  is the number of latent variables in set  $C^l,l = 1,\dots ,L$  and  $n_1 < n_2 < \dots < n_L$ . In our TIDA,  $L = 3$ , and  $n_2 = |\mathcal{Y}^l | + |\mathcal{Y}^u |$ .
From the EM [18] perspective, TIDA aims to maximize the likelihood of the observed  $N_{l} + N_{u}$  samples based on these hierarchical prototypes  $C$ . According to Jensen's inequality [47], we write the surrogate objective function as follows:
$$
\theta^ {*}, C ^ {*} = \arg \max  _ {\theta , C} \sum_ {i} ^ {N _ {l} + N _ {u}} \sum_ {C ^ {l} \in C} \sum_ {c _ {j} ^ {l} \in C ^ {l}} Q \left(c _ {j} ^ {l}\right) \log p \left(x _ {i}, c _ {j} ^ {l}; \theta\right) \quad \text {w i t h} \quad Q \left(c _ {i} ^ {l}\right) = p \left(c _ {j} ^ {l}; x _ {i}, \theta\right). \tag {6}
$$
To sum up, TIDA aims to estimate the posterior class probability  $Q(c_{j}^{l}) = p(c_{j}^{l}; x_{i}, \theta)$  on each hierarchy at the E-steps. Then, TIDA draws each sample to the prototype of its assigned cluster on each hierarchy at the M-step by optimizing Eq. 6 with known  $Q(c_{j}^{l})$ .
Theorem 1. By executing EM algorithm iteratively, samples from the same class in the feature space will be mapped into a  $d$ -variate von Mises-Fisher (vMF) distribution whose probabilistic density is given by  $g(\pmb{x}|\pmb{c}_i^l,\kappa;\theta) = c_d(\kappa)e^{\kappa\pmb{c}_i^l \top f_\theta(\pmb{x})}$ , where  $|\pmb{c}_i^l| = 1$  and  $\pmb{c}_i^l$  represents the mean direction,  $\kappa = 1/\tau$  is the concentration parameter, and  $c_d(\kappa)$  is the normalization factor.
E-step. In fact, the posterior class probability  $Q(c_{i}^{l}) = p(c_{i}^{l};x_{i},\theta) = \mathbb{I}(x_{i}\in c_{j}^{l})$ , which is equivalent to  $y_{i}^{l}$  in Eq. 2. In this paper, TIDA uses the Sinkhorn-Knopp algorithm on each hierarchy to assign the samples to the cluster center, as mentioned in Sec. 3.