SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets. We evaluate the proposed TransHP on five datasets with hierarchical labels, i.e., ImageNet [10], iNaturalist-2018 [11], iNaturalist-2019 [11], CIFAR-100 [25], and DeepFashion-inshop [26]. The hierarchical labels of ImageNet are from WordNet [1], with details illustrated on Mike's website. Both the iNaturalist-2018/2019 have two-level hierarchical annotations: a super-category (14/6 classes) for the genus, and 8, 142/1, 010 categories for the species. CIFAR-100 also has two-level hierarchical annotations: the coarse level has 20 classes, and the fine level has 100 classes. DeepFashion-inshop is a retrieval dataset with three-level hierarchy. To modify it for the classification task, we random select  $\frac{1}{2}$  images from each class for training, and the remaining  $\frac{1}{2}$  images for validation. Both the training and validation set contain 2 coarse classes, 17 middle classes, and 7, 982 fine classes, respectively.
Training and inference details. Our TransHP adopts an end-to-end training process. We use a lightweight transformer as our major baseline, which has 6 heads (half of ViT-B) and 12 blocks. The dimension of the embedding and the prompt token is 384 (half of ViT-B). We train it for 300 epochs on 8 Nvidia A100 GPUs and PyTorch. The base learning rate is 0.001 with cosine learning rate. We set the batch size, the weight decay and the number of warming up epochs as
Table 3: TransHP brings consistent improvement on various backbones on ImageNet.
Accuracy (%)	ViT-B/16	ViT-L/16	DeiT-S	DeiT-B
Baseline	76.68*	76.37*	79.82	81.80
TransHP	79.51	78.80	80.55	82.35
* The performance of our reproduced ViT-B/16 and ViT-L/16 are slightly worse than 77.91 and 76.53 in its original paper [9], respectively.
Table 4: Comparison between TransHP and two most recent state-of-the-art methods. We replace their CNN backbones with the relatively strong transformer backbone for fair comparison.
Accuracy (%)	ImageNet	iNat-2018	iNat-2019	CIFAR-100	DeepFashion
Baseline	76.21	63.01	69.31	84.98	88.54
Guided	76.05	63.11	69.66	85.10	88.32
HiMulConE	77.52	63.46	70.87	85.43	88.87
TransHP	78.65	64.21	71.62	86.85	89.93
1,024, 0.05 and 5, respectively. Importantly, TransHP only adds small overhead to the baseline. Specifically, compared with the baseline (22.05 million parameters), our TransHP only adds 0.60 million parameters (about  $+2.7\%$ ) for ImageNet. When using ViT-B as the backbone, our TransHP only adds  $+1.4\%$  parameters. Due to the increase of parameters and the extra cost of the backward of several  $L_{coarse}$ , the training time increases by  $15\%$  on our baseline and  $12\%$  on ViT-B for ImageNet. For inference, the computation overhead is very light. The baseline and TransHP both use around 50 seconds to finish the ImageNet validation with 8 A100 GPUs.
4.2 TransHP Improves the Accuracy
Improvement on ImageNet and the ablation study. We validate the effectiveness of TransHP on ImageNet and conduct the ablation study by comparing TransHP against two variants, as well as the baseline. As illustrated in Fig. 4, the two variants are: 1) we do not inject any prompts, but use the coarse labels to supervise the class token in the intermediate layers: similar with the final fine-level classification, the class token is also used for coarse-level classification. 2) we inject learnable tokens, but do not use the coarse labels as their supervision signal. Therefore, these tokens do not contain any coarse class information. From Fig. 4, we draw three observations as below: 1) Comparing TransHP against the baseline, we observe a clear improvement of  $+2.44\%$  top-1 accuracy, confirming the effectiveness of TransHP on ImageNet classification. 2) Variant 1 ("No prompts") achieves some improvement  $(+1.37\%)$  over the baseline as well, but is still lower than TransHP by  $-1.07\%$ . It shows that using the hierarchical labels to supervise the intermediate state of the class token is also beneficial. However, since it does not absorb the prompting information, the improvement is relatively small. We thus infer that the hierarchical prompting is a superior approach for utilizing the hierarchical labels. 3) Variant 2 ("No coarse labels") barely achieves any improvement over the baseline, though it also increases the same amount of parameters as TransHP. It indicates that the benefit of TransHP is not due to the increase of some trainable tokens. Instead, the coarse class information injected through the prompt tokens matters.
TransHP gains consistent improvements on more datasets. Besides the most commonly used dataset ImageNet, we also conduct experiments on some other datasets, i.e., iNaturalist-2018, iNaturalist-2019, CIFAR-100 and DeepFashion. For these datasets, we use two settings, i.e., training from scratch (w/o Pre) and finetuning from the ImageNet-pretrained model (w Pre). The experimental results are shown in Table 2, from which we draw two observations. First, under both settings, TransHP brings consistent improvement over the baselines. Second, when there is no pre-training, the improvement is even larger, especially on small datasets. For example, we note that on the smallest CIFAR-100, the improvement under "w/o Pre" and "w Pre" are  $+5.32\%$  and  $+1.87\%$ , respectively. We infer it is because TransHP considerably alleviates the data-hungry problem of the transformer, which is further validated in Section 4.3.
TransHP improves various backbones. Besides the light transformer baseline used in all the other parts of this section, Table 3 evaluates the proposed TransHP on some more backbones, i.e., ViT-B/16
Table 5: Comparison between TransHP and prior state-of-the-art hierarchical classification methods under the insufficient data scenario. "N%" means using N% ImageNet training data.
Accuracy (%)	100%	50%	20%	10%
Baseline	76.21	67.87	44.60	25.24
Guided	76.05	67.74	45.02	25.67
HiMulConE	77.52	69.23	48.50	30.76
TransHP	78.65	70.74	53.71	37.93
[9], ViT-L/16 [9], DeiT-S [27], and DeiT-B [27]. We observe that for the ImageNet classification, TransHP gains  $2.83\%$ ,  $2.43\%$ ,  $0.73\%$ , and  $0.55\%$  improvement on these four backbones, respectively.
Comparison with state-of-the-art hierarchical classification methods. We compare the proposed TransHP with two most recent hierarchy-based methods, i.e. Guided [3], HiMulConE [4]. We do not include more competing methods because most prior works are based on the convolutional backbones and are thus not directly comparable with ours. Since the experiments on large-scale datasets is very time-consuming, we only select the most recent state-of-the-art methods and re-implement them on the same transformer backbone (based on their released code). The experimental results are shown in Table 4. It is clearly observed that the proposed TransHP achieves higher improvement and surpasses the two competing methods. For example, on the five datasets, TransHP surpasses the most recent state-of-the-art HiMulConE by  $+1.13\%$  (ImageNet),  $+0.75\%$  (iNat-2018),  $+0.75\%$  (iNat-2019),  $+1.42\%$  (CIFAR-100) and  $1.06\%$  (DeepFashion), respectively. We also notice that while Guided achieves considerable improvement on the CNN backbones, its improvement over our transformer backbone is trivial. This is reasonable because improvement over higher baseline (i.e., the transformer backbone) is relatively difficult. This observation is consistent with [4].
4.3 TransHP Improves Data Efficiency
We investigate TransHP under the data-scarce scenario. To this end, we randomly select  $1/10$ ,  $1/5$ , and  $1/2$  training data from each class in ImageNet (while keeping the validation set untouched). The results are summarized in Table 5, from which we draw three observations as below:
First, as the training data decreases, all the methods undergo a significant accuracy drop. This is reasonable because the deep learning method in its nature is data-hungry, and arguably this data-hungry problem is further underlined in transformer [9]. Second, compared with the baseline and two competing hierarchy-based methods, TransHP presents much higher resistance against the data decrease. For example, when the training data is reduced from  $100\% \rightarrow 10\%$ , the accuracy drop of the baseline and two competing methods are  $50.97\%$ ,  $50.38\%$  and  $46.76\%$ , respectively. In contrast, the accuracy drop of the proposed TransHP ( $40.72\%$ ) is significantly smaller. Third, since TransHP undergoes relatively smaller accuracy decrease, its superiority under the low-data regime is even larger. For example, its surpasses the most competing HiMulConE by  $1.13\%$ ,  $1.51\%$ ,  $5.21\%$  and  $7.17\%$  under the  $100\%$ ,  $50\%$ ,  $20\%$  and  $10\%$  training data, respectively. Combining all these observations, we conclude that TransHP improves the data efficiency. The efficiency can be explained intuitively by drawing upon two perspectives, one philosophical and the other technical. Philosophical Perspective: Imagine knowledge as the essence of everything humans have summarized over time. When you possess knowledge, you have the distilled essence of myriad experiences and learnings. The proposed method leverages this accumulated knowledge. In scenarios where data is limited, the power of such distilled knowledge becomes even more pronounced. Technical Perspective: Now, think of data not just as isolated pieces of information but in categories. Even when the dataset might seem limited, there could still be ample samples within broader categories. This means that for these 'coarser' categories, accuracy can be achieved rapidly. Once the accuracy at this coarse level is established, the model can then use this foundation to prompt further. It's like planting a tree - you start with a strong base and then branch out.
4.4 TransHP Improves Model Explainability
We analyze the receptive field of the class token to understand how TransHP reaches its prediction. Basically, the transformer integrates information across the entire image according to the attention map, yielding its receptive field. Therefore, we visualize the attention map of the class token in Fig. 5.
Baseline
tench
TransHP (coarse)
fish
TransHP (fine)
tench
Baseline
German shepsherd
TransHP (coarse)
dog
TransHP (fine)
German shepsherd
bernese mountain dog
dome
protective covering
Figure 5: Visualization of the attention map for analyzing the receptive field. For TransHP, we visualize a block before and after receiving the prompt (i.e., coarse and fine), respectively. The "coarse" block favors an overview for coarse recognition, and the "fine" block further filters out the non-relevant regions after receiving the prompt.
house finch
doformat
For the proposed TransHP, we visualize the attention map at the prompting block (which handles the coarse-class information) and the last block (which handles the fine-class information). For the ViT baseline, we only visualize the attention score map of the the last block. We draw two observations from Fig. 5:
First, TransHP has a different attention pattern compared with the baseline. The baseline attention generally covers the entire foreground, which is consistent with the observation in [9]. In contrast, in TransHP, although the coarse block attends to the overall foreground as well, the fine block concentrates its attention on relatively small and critical regions, in pace with the "prompting  $\rightarrow$  predicting" procedure. For example, given the "hen" image on the second row (left), TransHP attends to the overall foreground before receiving the coarse-class prompt (i.e., the bird) and focuses to the eyes and bill for recognizing the "hen" out from the "bird". Second, TransHP shows better capacity for ignoring the redundant and non-relevant regions. For example, given the "doormat" image on the fourth row (right), TransHP ignores the decoration of "GO AWAY" after receiving the coarse-class prompt of "covering". Similar observation is with the third row (right), where TransHP ignores the walls when recognizing the "dome" out from "protective covering".
This paper proposes a novel Transformer with Hierarchical Prompt (TransHP) for image classification. Before giving its final prediction, TransHP predicts the coarse class with an intermediate layer and correspondingly injects the coarse-class prompt to condition the subsequent inference. An intuitive effect of our hierarchical prompting is: TransHP favors an overview of the object for coarse prediction and then concentrates its attention to some critical local regions after receiving the prompt, which is similar to the human visual recognition. We validate the effectiveness of TransHP through extensive experiments and hope the hierarchical prompting reveals a new insight for understanding the transformers.
Limitation. We presently focus on the image classification task in this paper, while there are some other tasks that are the potential to benefit from hierarchical annotations, e.g., semantic segmentation. Therefore, we would like to extend TransHP for more visual recognition tasks in the future.
[18] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), 2022.
[19] Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels. arXiv preprint arXiv:2111.11595, 2021.
[20] Kanishk Jain, Shyamgopal Karthik, and Vineet Gandhi. Test-time amendment with a coarse classifier for fine-grained classification. arXiv preprint arXiv:2302.00368, 2023.
[21] Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE international conference on computer vision, pages 2740-2748, 2015.
[22] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[23] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL/IJCNLP (1), 2021.
[24] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[27] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
Figure 6: The illustration of TransHP with multiple layers of hierarchy.  $k$  and  $l$  are two insider layers, and  $L$  is the final layer.
Figure 7: The top-1 accuracy on ImageNet w.r.t the transformer layer from which to add prompt tokens. The highest two transformer layers (which do not 1 have too coarse-level labels) play an important role.
A Multiple layers of hierarchy
We illustrate the TransHP in Fig. 6 when a dataset has multiple layers of hierarchy.
B Coarse-level classes of CIFAR-100
[0]: aquatic mammals, [1]: fish, [2]: flowers, [3]: food containers, [4]: fruit and vegetables, [5]: household electrical devices, [6]: household furniture, [7]: insects, [8]: large carnivores, [9]: large man-made outdoor things, [10]: large natural outdoor scenes, [11]: large omnivores and herbivores, [12]: medium mammals, [13]: non-insect invertebrates, [14]: people, [15]: reptiles, [16]: small mammals, [17]: trees, [18]: vehicles-1, and [19]: vehicles-2.
C Importance analysis of classification at different hierarchical levels
From Table 1 (Line 1), each transformer layer is responsible for one level classification. We remove the prompt tokens from the coarsest level to the finest level. In Fig. 7,  $n$  denotes that
Table 6: The analysis of the number of coarse-level classes on the CIFAR-100 dataset. "N-class" denotes that there are  $N$  classes for the coarse-level classification.
Accuracy (%)	baseline	2-class	5-class	10-class	20-class
w/o Pre	61.77	63.34	63.12	64.47	67.09
w Pre	84.98	86.40	86.35	86.50	86.85
Table 7: Comparison between TransHP with the original baseline and the "No prompts" baseline.
Accuracy (%)	iNat-2018	iNat-2019	CIFAR-100	DeepFashion
Baseline (w/o Pre)	51.07	57.33	61.77	83.42
No prompts (w/o Pre)	51.88	58.45	63.78	84.23
TransHP (w/o Pre)	53.22	59.24	67.09	85.72
Baseline (w Pre)	63.01	69.31	84.98	88.54
No prompts (w Pre)	63.41	70.73	85.50	89.59
TransHP (w Pre)	64.21	71.62	86.85	89.93
the prompt tokens are added from the  $n$ th transformer layer. We conclude that only the last two coarse level classifications (arranged at the 9th and 10th transformer layer) contribute most to the final classification accuracy. That means: (1) it is not necessary that the number of hierarchy and transformer layers are equal. (2) it is no need to adjust any parameters from too coarse level hierarchy. (Note that: though the current balance parameter for the 8th transformer layer is 0.15, when it is enlarged to 1, no further improvement is achieved.)
D Analysis of the number of coarse-level classes
As shown in Appendix B, the CIFAR-100 dataset has 20 coarse-level classes. When we combine them into 10 coarse-level classes, we have ([0-1]), ([2-17]), ([3-4]), ([5-6]), ([12-16]), ([8-11]), ([14-15]), ([9-10]), ([7-13]), and ([18-19]). When we combine them into 5 coarse-level classes, we have ([0-1-12-16]), ([2-17-3-4]), ([5-6-9-10]), ([8-11-18-19]), and ([7-13-14-15]). When we combine them into 2 coarse-level classes, we have ([0-1-7-8-11-12-13-14-15-16]) and ([2-3-4-5-6-9-10-17-18-19]). The experimental results are listed in Table 6.
We observe that: 1) Generally, using more coarse-level classes is better. 2) Using only 2 coarse-level classes still brings over  $1\%$  accuracy improvement.
E The comparison with the "No prompts" baseline
In this section, we provide more experiments with the "No prompts" baseline. The detail of the "No prompts" baseline is shown in Fig. 4 (2). The experimental results are shown in Table 7. We find that though "No prompts" baseline surpasses the original baseline, our TransHP still shows significant superiority over this baseline.
F Additional  $L_{coarse}$  with DeiT.
We introduce the experimental results by only adopting  $L_{coarse}$  in DeiT. Note that the  $L_{coarse}$  is imposed on the class token as shown in Fig. 4 (2). We find that the TransHP still shows performance improvement compared with only using  $L_{coarse}$  on DeiT-S and DeiT-B: compared with DeiT-S (79.82%) and DeiT-B (81.80%), "only with  $L_{coarse}$ " achieves 79.98% and 81.76% while the TransHP achieves 80.55% and 82.35%, respectively.