SlowGuess/NeurIPS_2023_OCR huggingface dataset:
M-step. Given  $Q(c_{j}^{l}) = p(c_{j}^{l};x_{i},\theta)$ , TIDA aims to maximize the surrogate function in Eq. 6 in M-step. Finally, as Theorem. 1 shown, samples will be mapped into different  $d$ -variate von Mises-Fisher (vMF) distributions under different hierarchies [72].
(a) Separate Objective Functions for Each Hierarchy. Due to Eq. 6 is hard to optimize, following [47], we assume that the distribution around each prototype  $c_i^l$  satisfy an isotropic Gaussian, i.e.,  $p(x_i; c_i^l, \theta) = \exp \left(-\left(z_i - c_i^l\right)^2 / 2\sigma_i^{l^2}\right) / \sum_{j=1}^{n_l} \exp \left(-\left(z_i - c_j^l\right)^2 / 2\sigma_j^{l^2}\right)$ . Then Eq. 6 is written as:
$$
\theta^ {*}, C ^ {*} = \arg \max  _ {\theta , C ^ {l}} \sum_ {i} ^ {N _ {l} + N _ {u}} \sum_ {C ^ {l} \in C} \sum_ {c _ {i} ^ {l} \in C ^ {l}} - \log \frac {\exp \left(z _ {i} \cdot c _ {s} ^ {i , l} / \tau\right)}{\sum_ {j = 1} ^ {n _ {l}} \exp \left(z _ {i} \cdot c _ {j} ^ {l} / \tau\right)}, \tag {7}
$$
where  $c_{s}^{i,l}$  denotes  $z_{i}$ 's assigned prototype ( $p(c_{s}^{i,l}; x_{i}, \theta) = 1$  and  $c_{s}^{i,l} \in C^{l}$ ) and  $\tau = 1 \propto \sigma_{i}^{l^{2}}$ , which plays the role of temperature parameter. As we can see, Eq. 7 is equivalent to Eq. 2.
(b) Constrain the Consistency across Hierarchies in Feature Space. Since only optimizing Eq. 7 results in inconsistencies of these learned vMF distributions across hierarchies, we propose to obtain
Table 1: Comparison with state-of-the-art methods on generic datasets.
Methods	CIFAR10	CIFAR100	ImageNet-100	Tiny ImageNet
Seen	Novel	All	Seen	Novel	All	Seen	Novel	All	Seen	Novel	All
DTC [29]	42.7	31.8	32.4	22.1	10.5	13.7	24.5	17.8	19.3	13.5	12.7	11.5
RankStats [28]	71.4	63.9	66.7	20.4	16.7	17.8	41.2	26.8	37.4	9.6	8.9	6.4
UNO [23]	86.5	71.2	78.9	53.7	33.6	42.7	66.0	42.2	53.3	28.4	14.4	20.4
ORCA [5]	82.8	85.5	84.1	52.5	31.8	38.6	83.9	60.5	69.7	-	-	-
OpenNCD [50]	83.5	86.7	85.3	53.6	33.0	41.2	84.0	65.8	73.2	-	-	-
TRSSL [65]	94.9	89.6	92.2	68.5	52.1	60.3	82.6	67.8	75.4	39.5	20.5	30.3
OpenLDN [64]	92.4	93.2	92.8	55.0	40.0	47.7	-	-	-	-	-	-
TIDA (Ours)	94.2	93.4	93.8	73.3	56.6	65.3	83.4	71.2	77.6	45.7	28.4	37.2
Table 2: Comparison with state-of-the-art methods on fine-grained datasets.
Methods	Oxford-IIIT Pet	FGVC Aircraft	Stanford-Cars
Seen	Novel	All	Seen	Novel	All	Seen	Novel	All
DTC [29]	20.7	16.0	13.5	16.3	16.5	11.8	12.3	10.0	7.7
RankStats [28]	12.6	11.9	11.1	13.4	13.6	11.1	10.4	9.1	6.6
UNO [23]	49.8	22.7	34.9	44.4	24.7	31.8	49.0	15.7	30.7
TRSSL [65]	70.9	36.1	53.9	69.5	41.2	55.4	83.5	37.1	60.4
OpenLDN [64]	66.8	33.1	50.4	-	-	45.7	-	-	38.7
TIDA (Ours)	75.7	39.2	59.9	71.1	43.7	57.4	85.9	43.5	66.0
the aggregated latent variables  $\tilde{C}^{2|r}$  with variables set  $C^r$ , then constrain the distribution around  $\tilde{C}^{2|r}$  be consistent with the original one around  $C^2$ , where  $r = 1,3$ .
First, we establish the similarity relationship across adjacent hierarchy, i.e., the mapping matrix  $M^{r,2} = \mathcal{S}(C^r^\top, C^2)$ , where for  $m^{r,2}[j,k] \in M^{r,2}$ ,  $m^{2,r}[j,k] = \mathcal{S}(c_j^r, c_k^2)$ . Then we obtained its aggregated variables set  $\tilde{C}^{2|r}$  with latent variables  $C^r$  as  $\tilde{C}^{2|r} = \{\tilde{c}_k^{2|r}\}_{k=1}^{n_2}$ ,  $\tilde{c}_k^{2|r} = \sum_{j=1}^{n_2} c_j^r m^{r,2}[j,k]$ . Note, the variables set  $\tilde{C}^{2|r}$  dynamically aggregates semantics from  $r$ -th hierarchy and builds communication among clusters across hierarchies, which facilitates aligning hierarchical semantics with target tasks.
Given aggregated variables, we then apply consistent constraints on the sample's distribution around  $\tilde{C}^{2|r}$  and the original ones in the target hierarchy, where  $r = 1,3$ . Specifically, we use aggregated variables  $\tilde{C}^{2|r}$  to calculate the objective function in Eq. 7, as follows:
$$
\theta^ {*}, C ^ {*} = \arg \max  _ {\theta , C} \sum_ {i} ^ {N _ {l} + N _ {u}} \sum_ {r = 1, 3} - \log \frac {\exp \left(z _ {i} \cdot \tilde {c} _ {s} ^ {i , 2 | r} / \tau\right)}{\sum_ {k = 1} ^ {n _ {2}} \exp \left(z _ {i} \cdot \tilde {c} _ {k} ^ {2 | r} / \tau\right)}, \tag {8}
$$
where the index  $s$  in  $\tilde{c}_s^{i,2|r}$  corresponds to the assigned target-grained prototype  $c_s^{i,2}$  for feature  $z_i$  ( $p(c_s^2; x_i, \theta) = 1$ ). As we can see, Eq. 8 is equivalent to Eq. 5. More details are in Appendix.
Datasets. We evaluate TIDA on four commonly used generic image classification datasets (i.e. CIFAR10 [41], CIFAR100 [41], TinyImageNet [20] and ImageNet-100 [20]) and three fine-grained datasets (i.e. Oxford-IIT Pet [52], Standford-Cars [40] and Aircraft [55]). Following [5, 65], we use the first half of classes as seen classes and the remaining as novel. More details are in the Appendix.
Implementation Details. Following [65, 5, 64], we use ResNet-50 [22] for ImageNet-100 and ResNet-18 [22] for the other datasets. For all datasets, we set the length of prototypes set  $L$  as 3, in which  $C = \{C^1, C^2, C^3\}$  and  $|C^2| = |\mathcal{Y}^l \cup \mathcal{Y}^u|$ . For the number of prototypes  $C^1$  and  $C^3$ , we set  $|C^1| = \alpha * |C^2|$  and  $|C^3| = \beta * |C^2|$ . We set  $\alpha = 0.2 / 0.4$  and  $\beta = 2 / 2.5$  for generic / fine-grained datasets. We use a cosine annealing-based learning rate scheduler accompanied by a linear warmup, where we set the base learning rate to  $0.5 / 1.5$  for generic / fine-grained datasets. The warmup length is set to 10 epochs and the weight decay is set to 1e-4. More details are in the Appendix.
Table 3: The ablation study. C-TCP: Coarse-grained Taxonomic Context Priors; F-TCP: Fine-grained Taxonomic Context Priors; TCA: Taxonomic Context-based prediction Alignment. When using TCA only, the model is equipped with three target-grained classifiers that are aligned by TCA.
#	C-TCP	F-TCP	TCA	CIFAR100	Tiny ImageNet	Stanford-Cars
Seen	Novel	All	Seen	Novel	All	Seen	Novel	All
a)	67.0	48.9	57.9	39.7	21.8	31.1	84.8	38.3	61.3
b)	✓	63.7	47.0	55.5	36.4	19.2	28.0	81.2	41.2	61.1
c)	✓	✓	71.3	51.6	61.6	43.7	27.1	35.8	84.9	40.8	62.9
d)	✓	65.3	45.6	55.3	36.3	20.4	28.8	78.1	31.2	54.8
e)	✓	✓	71.5	54.6	63.1	44.6	27.3	36.8	85.3	42.3	64.1
f)	✓	71.8	49.1	60.5	43.1	21.1	32.5	82.8	39.5	61.9
g)	✓	✓	69.9	45.4	57.7	36.3	19.8	29.0	76.4	30.2	53.8
h)	✓	✓	✓	73.3	56.6	65.3	45.7	28.4	37.2	85.9	43.5	66.0
5.2 Comparison with State-of-The-Art
We first compare our TIDA with state-of-the-art methods [5, 23, 28, 29, 64, 65] on four generic datasets and three fine-grained datasets, where the results are reported in Tab. 1 and Tab. 2, respectively. It is clear that TIDA consistently outperforms all state-of-the-art methods across all datasets and metrics. Specifically, TIDA surpasses the current best competitor TRSSL [65] by  $6.9\%$  on TinyImageNet for All classes. Importantly, TIDA outperforms previous methods on most datasets for Novel classes by a large margin, e.g.,  $7.9\%$  on TinyImageNet and  $6.4\%$  on Stanford-Cars. These results experimentally demonstrate that exploring taxonomic context priors as auxiliary supervision is a beneficial way for discriminating seen and novel classes under OSSL.
Effect of Coarse-grained Taxonomic Context Priors. To verify the effectiveness of Coarse-grained Taxonomic Context Priors (C-TCP), we conduct a comparison with three variants: a) the baseline that only clusters samples on target-grained; b) the model that clusters samples on both coarse-grained and target-grained without using Taxonomic Context-based prediction Alignment (TCA); c) the model which simultaneously clusters samples both on coarse-grained and target-grained with TCA. As we can see in Tab. 3, c) outperforms a) and b) but b) worsened a), showing that generic semantics will improve the performance only when the coarse-grained clustering is consistent with the target task.
Effect of Fine-Grained Taxonomic Context. Similarly, we also conduct a comparison with three variants to verify the effectiveness of Fine-grained Taxonomic Context Priors (F-TCP), i.e., a) the baseline, d) the model additionally includes F-TCP learning, and e) the model additionally includes F-TCP learning and TCA. Results show that F-TCP learning facilitates the target task only when using ATC. Moreover, the fine-grained semantic works relatively better than the generic semantic (see the comparison between c) and e)), especially the performance on Novel classes. In addition, jointly considering C-TCP and F-TCP could further improve the performance, as shown in variant h).
Effect of Taxonomic Context-based Prediction Alignment. As discussed in the previous two ablation studies, the proposed TCA is indispensable to our TIDA. Specifically, by comparing b) v.s. c), d) v.s. e), and g) v.s. h), we can observe that only using coarse-grained taxonomic context learning or fine-grained taxonomic context learning, or both of them fail to achieve improvement over the baseline. However, after injecting the proposed taxonomic context-based prediction alignment module, the performance is significantly improved. Because the inconsistency among multi-granularity classification leads to hard-optimization problem when using TCD alone. Actually, TCD applies multiple classifiers to cluster samples under different granularity. It can be regarded as multi-task learning with a shared backbone. However, due to a lack of supervision (e.g., semantic names and attributes), when using TCD alone, the training objective among classifiers is inconsistent and the optimization on the sub- and sup-classifiers is completely unsupervised. This may have a negative impact on model optimization and consequently result in performance degradation. While, the TCA learn to align the training objectives among classifiers and enable the sub- and sup-classifiers to benefit from the knowledge of labeled data on the target classifier.
Figure 4: The hyper-parameters analysis. (a)  $\alpha$ : The weight that decides the number of super-classes. (b)  $\beta$ : The weight that decides the number of sub-classes.
(a) TRSSL
Figure 5: The t-SNE visualization of features from TRSSL and TIDA on CIFAR100. For clarity, we randomly select 20 seen and 20 novel classes as examples from CIFAR100.
(c) Fined-Grained TC
(d) Coarse-Grained TC
Effect of Alignment across Multiple Granularity. It is interesting to see whether the performance improvement comes from the alignment across multiple granularity or alignment across a number of classifiers. To evaluate this, we introduce a variant f) with three target-grained clusters learning, i.e.,  $|C^1| = |C^2| = |C^3| = |\mathcal{Y}^l| + |\mathcal{Y}^u|$ . Compared with f) and h), we obverse that the alignment across multiple target-granularity is beneficial for Seen classes but there is no improvement or even reduction for Novel classes. This indicates that the improvement mainly comes from interaction among different levels of granularity instead of different classifiers of the same granularity.
5.4 Hyper-Parameter Analysis
Impact of the Number of Super-Classes. As shown in Fig. 4(a), it is more suitable to assign a smaller value for  $\alpha$ , which leads to a small number of coarse-grained classes. When  $\alpha$  approaches to 1, TIDA degenerates to the model that ignores the coarse-grained granularity. The best result is achieved by  $\alpha = 0.2 / 0.4$  for TinyImageNet / Standard Car.
Impact of the Number of Sub-Classes. In contrast, as shown in Fig. 4(b), assigning a larger value for  $\beta$  leads to a higher performance, which enables us to obtain fine-grained prior. Again, when  $\beta$  nears 1, TIDA will largely overlook the fined-grained granularity and produce lower results. The best performance is obtained by  $\beta = 2 / 2.5$  for Tiny ImageNet / Standard Car.
T-SNE Visualization. We use t-SNE [71] to visualize the features learned by TIDA and the baseline [65]. As shown in Fig. 5(a) and Fig. 5(b), TIDA produces more discriminative features than TRSSL [65], where the samples are generally better clustered. To better understand our method, we also visualize the fine-grained and coarse-grained features learned by TIDA, shown in Fig. 5(c) and Fig. 5(d). At the fine-grained level, samples from the same categories are divided into different clusters; while, samples belonging to the same super-class are clustered together at the coarse-grained level. This further verifies that TIDA can learn more semantic priors which are complementary.
Visualization of Affinity Matrix. In Fig. 6, we calculate the affinity matrix by counting the number of samples belonging to each target class that are classified into each super-class. Results show that samples of the same target class are commonly classified into the same super-class for TIDA w/ TCA. In addition, we find that similar target classes are generally mapped into the same super-class. For example, "bowl, cup, lamp, plate, clock" are assigned with the 18th super-class. In contrast, for TIDA w/o TCA, samples of the same target class tend to be classified into different super-classes, resulting in unclear and inconsistent affinity relationships among classes across hierarchies. These results validate the effectiveness of our TCA in establishing a clear and consistent affinity relationship.
Figure 6: Affinity matrix that calculates the number of samples of each target class classified into each super-class on CIFAR100. For clarity, we rearrange the target classes, which belong to the same super-class being grouped together (see Appendix). Here, (a) and (b) follow the same order.
6 Contribution
In this paper, we propose a novel framework known as TIDA. It can effectively learn discriminative visual representations and improve pseudo-label accuracy by exploring the underlying taxonomic context priors. Specifically, TIDA consists of two core components: i) clustering samples hierarchically to capture the underlying taxonomic context in latent space, and ii) constraining predictions across hierarchies to be consistent to build reliable affinity relationships to ensure taxonomic context is consistent. The theoretical analysis and extensive experiments on commonly used benchmarks with consistent performance gains both justify the superiority of our TIDA.
Limitations. Despite obtaining high performance, TIDA and existing methods still suffer from several limitations: assuming that i) the domain distribution between labeled and unlabeled data is the same; ii) the class-distribution is uniform; and iii) the number of novel classes is known. Future work may focus on exploring more practical OSSL situations that unlock the above constraints.
Acknowledgements. This work was supported in part by the National Key R&D Program of China (No.2022ZD0118201), the Natural Science Foundation of China (No. 61972217, 32071459, 62176249, 62006133, 62271465), the MUR PNRR project FAIR (PE00000013) funded by the NextGenerationEU, and the EU project Ai4Media (No. 951911).
[1] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. In ICLR, 2019.
[2] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, 2019.
[3] Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In NeurIPS, 2022.
[4] Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang, Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. In NeurIPS, 2022.
[5] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi-supervised learning. In ICLR, 2022.
[6] Baixu Chen, Junguang Jiang, Ximei Wang, Pengfei Wan, Jianmin Wang, and Mingsheng Long. Debiased self-training for semi-supervised learning. In NeurIPS, 2022.
[67] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NeurIPS, 2016.
[68] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeruIPS, 2020.
[69] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
[70] Colin Troisemaine, Vincent Lemaire, Stéphane Gosselin, Alexandre Reiffers-Masson, Joachim Flocon-Cholet, and Sandrine Vaton. Novel class discovery: an introduction and key concepts. arXiv, 2023.
[71] Laurens Van Der Maaten. Learning a parametric embedding by preserving local structure. PMLR, 2009.
[72] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. PiCO: Contrastive label disambiguation for partial label learning. In ICLR, 2022.
[73] Yidong Wang, Hao Chen, Yue Fan, Wang Sun, Ran Tao, Wenxin Hou, Renjie Wang, Linyi Yang, Zhi Zhou, Lan-Zhe Guo, Heli Qi, Zhen Wu, Yu-Feng Li, Satoshi Nakamura, Wei Ye, Marios Savvides, Bhiksha Raj, Takahiro Shinozaki, Bernt Schiele, Jindong Wang, Xing Xie, and Yue Zhang. USB: A unified semi-supervised learning benchmark for classification. In NeurIPS, 2022.
[74] Yu Wang, Pengchong Qiao, Chang Liu, Guoli Song, Xiawu Zheng, and Jie Chen. Out-of-distributed semantic pruning for robust semi-supervised learning. In CVPR, 2023.
[75] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. In NeruIPS, 2019.
[76] Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmentation with prototype-based consistency regularization. In NeurIPS, 2022.
[77] Jun Yu, Liwen Zhang, Shenshen Du, Hao Chang, Keda Lu, Zhong Zhang, Ye Yu, Lei Wang, and Qiang Ling. Pseudo-label generation and various data augmentation for semi-supervised hyperspectral object detection. In CVPRW, 2022.
[78] Arman Zharmagambetov and Miguel A. Carreira-Perpinan. Semi-supervised learning with decision trees: Graph laplacian tree alternating optimization. In NeurIPS, 2022.
[79] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In CVPR, 2021.
[80] Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In CVPR, 2021.