SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenhao Wang *
ReLER, University of Technology Sydney
Yifan Sun
Wei Li
Yi Yang
This paper explores a hierarchical prompting mechanism for the hierarchical image classification (HIC) task. Different from prior HIC methods, our hierarchical prompting is the first to explicitly inject ancestor-class information as a tokenized hint that benefits the descendant-class discrimination. We think it well imitates human visual recognition, i.e., humans may use the ancestor class as a prompt to draw focus on the subtle differences among descendant classes. We model this prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). TransHP consists of three steps: 1) learning a set of prompt tokens to represent the coarse (ancestor) classes, 2) on-the-fly predicting the coarse class of the input image at an intermediate block, and 3) injecting the prompt token of the predicted coarse class into the intermediate feature. Though the parameters of TransHP maintain the same for all input images, the injected coarse-class prompt conditions (modifies) the subsequent feature extraction and encourages a dynamic focus on relatively subtle differences among the descendant classes. Extensive experiments show that TransHP improves image classification on accuracy (e.g., improving ViT-B/16 by  $+2.83\%$  ImageNet classification accuracy), training data efficiency (e.g.,  $+12.69\%$  improvement under  $10\%$  ImageNet training data), and model explainability. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information. The code is available at: https://github.com/WangWenhao0716/TransHP.
Hierarchical image classification (HIC) aims to exploit the semantic hierarchy to improve prediction accuracy. More concretely, HIC provides additional coarse labels (e.g., Rose) which indicate the ancestors of the relatively fine labels (e.g., China Rose and Rose Peace). The coarse labels usually do not need manual annotation and can be automatically generated based on the fine labels, e.g., through WordNet [1] or word embeddings [2]. Since it barely increases any annotation cost while bringing substantial benefit, HIC is of realistic value and has drawn great research interest [3, 4].
This paper explores a novel hierarchical prompting mechanism that well imitates the human visual recognition for HIC. Specifically, a person may confuse two close visual concepts (e.g., China Rose and Rose Peace) when the scope of interest is large (e.g., the whole Plantae). However, given a prompt narrowing down the category range (e.g., the rose family), the person can shift his/her focus to the subtle variations within the coarse class. We duplicate this procedure for deep visual recognition
*Work done during Wenhao Wang's internship at Baidu Inc.
(a)
Figure 1: The comparison between Vision Transformer (ViT) and the proposed Transformer with Hierarchical Prompting (TransHP). In (a), ViT attends to the overall foreground region and recognizes the goldfish from the 1000 classes in ImageNet. In (b), TransHP uses an intermediate block to recognize the input image as belonging to the fish family and then injects the corresponding prompt. Afterward, the last block attends to the face and crown, which are particularly informative for distinguishing the goldfish against other fish species. Please refer to Fig. 5 for more visualizations. Note that TransHP may have multiple prompting blocks corresponding to multi-level hierarchy.
based on the transformer prompting technique. The transformer prompting typically uses prompts (implemented as tokens or vectors) to adapt a pre-trained transformer for different downstream tasks [5, 6, 7], domains [8], etc. In this paper, we inject coarse-class prompt into the intermediate stage of a transformer. The injected coarse-class prompt will then modify the following feature extraction for this specific coarse class, yielding the so-called hierarchical prompting. To the best of our knowledge, explicitly injecting the coarse class information as a prompt has never been explored in the HIC community.
We model our hierarchical prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). Fig. 1 compares our TransHP against a popular transformer backbone ViT [9]. TransHP consists of three steps: 1) TransHP learns a set of prompt tokens to represent all the coarse classes and selects an intermediate block as the "prompting block" for injecting the prompts. 2) The prompting block on-the-fly predicts the coarse class of the input image. 3) The prompting block injects the prompt token of the predicted class (i.e., the target prompt token) into the intermediate feature. Specifically, TransHP concatenates the prompt tokens with feature tokens (i.e., the "class" token and the patch tokens) from the preceding block, and then feeds them into the prompting block, where the feature tokens absorb information from the target prompt through cross-attention<sup>3</sup>.
Although TransHP is based on the prompting mechanism of the transformer, it has fundamental differences against prior transformer prompting techniques. A detailed comparison is in Section 2. We hypothesize this hierarchical prompting will encourage TransHP to dynamically focus on the subtle differences among the descendant classes. Fig. 1 validates our hypothesis by visualizing the attention map of the final-block class token. In Fig. 1 (a), given a goldfish image, the baseline model (ViT) attends to the whole body for recognizing it from the entire 1000 classes in ImageNet. In contrast, in Fig. 1 (b), since the intermediate block has already received the prompt of "fish", TransHP mainly attends to the face and crown which are particularly informative for distinguishing the goldfish against other fish species. Please refer to Section 4.4 for more visualization examples.
We conduct extensive experiments on multiple image classification datasets (e.g., ImageNet [10] and iNaturalist [11]) and show that the hierarchical prompting improves the accuracy, data efficiency
In practice, the absorption is in a "soft" manner which assigns all the prompt tokens with soft weights.
and explainability of the transformer: (1) Accuracy. TransHP brings consistent improvement on multiple popular transformer backbones and five image classification datasets. For example, on ImageNet, TransHP improves ViT-B/16 [9] by  $+2.83\%$  top-1 accuracy. (2) Data efficiency. While reducing the training data inevitably compromises the accuracy, TransHP maintains better resistance against the insufficient data problem. For example, when we reduce the training data of ImageNet to  $10\%$ , TransHP enlarges its improvement over the baseline to  $+12.69\%$ . (3) Explainability. Through visualization, we observe that the proposed TransHP shares some similar patterns with human visual recognition [12, 13], e.g., taking an overview for coarse recognition and then focusing on some critical local regions for the subsequent recognition after prompting. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information.
Prior HIC methods have never explored the prompting mechanism. We note that the prompting technique has not been introduced to the computer vision community until the very recent 2022 [14, 15, 16, 17, 18]. Two most recent and state-of-the-art HIC methods in 2022 are not related to the prompting technique either. Specifically, Guided [3] integrates a cost-matrix-defined metric into the supervision of a prototypical network. HiMulConE [4] builds an embedding space in which the distance between two classes is roughly consistent with the hierarchy (e.g., two sibling classes sharing the same ancestor are relatively close, and the classes with different ancestors are far away). Some earlier works [19, 20, 21] are valuable; however, they are also not directly related to the topic of prompting.
A deeper difference resulting from the prompting mechanism is how the mapping function of the deep model is learned. Specifically, a deep visual recognition model can be viewed as a mapping function from the raw image space into the label space. All these prior methods learn a shared mapping for all the images to be recognized. In contrast, the proposed TransHP uses the coarse-class prompt to condition itself (from an intermediate block). It can be viewed as specifying an individual mapping for different coarse classes, yielding a set of mapping functions. Importantly, TransHP makes all these mapping functions share the same transformer and conditions the single transformer into different mapping functions through the prompting mechanism.
Prompting was first proposed in NLP tasks [22, 23, 24], and then has drawn research interest from the computer vision community, e.g. continual learning [14, 15], image segmentation [16], and neural architecture search [17]. VPT [18] focuses on how to fine-tune pre-trained ViT models to downstream tasks efficiently. Prompting can efficiently adapt transformers to different tasks or domains while keeping the transform's parameters untouched.
Based on the prompting mechanism, our hierarchical prompting makes some novel explorations, w.r.t. the prompting objective, prompting structure, prompt selection manner, and training process. 1) Objective: previous methods usually prompt for different tasks or different domains. In contrast, TransHP prompts for coarse classes in the hierarchy, in analogy to the hierarchical prompting in human visual recognition. 2) Structure: previous methods usually inject prompt tokens to condition the whole model. In contrast, in TransHP, the bottom blocks is completely shared, and the prompt tokens are injected into the intermediate blocks to condition the subsequent inference. Therefore, the prompting follows a hierarchical structure in accordance to the semantic hierarchy under consideration. 3) Prompt selection: TransHP pre-pends all the prompt tokens for different coarse classes and autonomously selects the prompt of interest, which is also new (as to be detailed in Section 3.4). 4) Training process: The prompting technique usually consists of two stages, i.e., pre-training a base model and then learning the prompts for novel downstream tasks. When learning the prompt, the pre-trained model is usually frozen. This pipeline is different from our end-to-end pipeline, i.e. no more fine-tuning after this training.
3 Transformer with Hierarchical Prompting
We first revisit a basic transformer for visual recognition (ViT [9]) and the general prompting technique in Section 3.1. Afterward, we illustrate how to reshape an intermediate block of the
Figure 2: (i) A prompting block in TransHP. Instead of manually selecting the prompt of the coarse class, the prompting block pre-pends the whole prompt pool consisting of  $M$  prompts ( $M$  is the number of coarse classes) and performs autonomous selection. Specifically, it learns to predict the coarse class (Section 3.2) and spontaneously selects the corresponding prompt for absorption through soft weighting (Section 3.4), i.e., the predicted class has the largest absorption weight. (ii) Autonomous prompt selection. We visualize the absorption weights of all the 20 coarse-class prompts for some CIFAR-100 images. It shows how TransHP selects the prompts when the coarse class prediction is correct ( $a$  and  $b$ ), ambiguous ( $c$  and  $d$ ), and incorrect ( $e$  and  $f$ ), respectively. The red and green columns correspond to the ground-truth (GT) class and the false classes, respectively. The detailed investigation is in Section 3.4.
backbone into a hierarchical prompting block for TransHP in Section 3.2. Finally, we investigate how the prompting layer absorbs the prompt tokens into the feature tokens in Section 3.4.
Vision Transformer (ViT) first splits an image into  $N$  patches ( $\{x_{i} \in \mathbb{R}^{3 \times P \times P} \mid i = 1,2,\dots,N\}$ , where  $P \times P$  is the patch size) and then embeds each patch into a  $C$ -dimensional embedding by  $\mathbf{x}_i = \operatorname{Embed}(x_i)$ . Afterward, ViT concatenates a class token  $x_{cls}^0 \in \mathbb{R}^C$  to the patch tokens and feed them into the stacked transformer blocks, which is formulated as:
$$
\left[ \mathbf {x} _ {c l s} ^ {l}, \mathbf {X} ^ {l} \right] = B _ {l} \left(\left[ \mathbf {x} _ {c l s} ^ {l - 1}, \mathbf {X} ^ {l - 1} \right]\right), \quad l = 1, 2, \dots , L \tag {1}
$$
where  $\mathbf{x}_{cls}^{l}$  and  $\mathbf{X}^l$  are the class token and the patch tokens after the  $l$ -th transformer block  $B_{l}$ , respectively. After the total  $L$  blocks, the final state of the class token  $(\mathbf{x}_{cls}^{L})$  is viewed as the deep representation of the input image and is used for class prediction. In this paper, we call the concatenation of class token and patch tokens (i.e.,  $[\mathbf{x}_{cls}^{l-1}, \mathbf{X}^{l-1}]$ ) as the feature tokens.
Prompting was first introduced in Natural Language Processing to switch the same transformer model for different tasks by inserting a few hint words into the input sentences. More generally, it conditions the transformer to different tasks, different domains, etc, without changing the transformer parameters but only changing the prompts. To condition the model for the  $k$ -th task (or domain), a popular practice is to select a prompt  $\mathbf{p}_k$  from a prompt pool  $\mathbf{P} = \{\mathbf{p}_0, \mathbf{p}_1, \dots\}$  and pre-pend it to the first block. Correspondingly, Eqn. 1 turns into:
$$
\left[ \mathbf {x} _ {c l s} ^ {l}, \mathbf {X} ^ {l}, \mathbf {p} _ {k} ^ {l} \right] = B _ {l} \left(\left[ \mathbf {x} _ {c l s} ^ {l - 1}, \mathbf {X} ^ {l - 1}, \mathbf {p} _ {k} ^ {l - 1} \right]\right), \tag {2}
$$
where  $\mathbf{p}_k\in \mathbf{P}$  (the superscript is omitted) conditions the transformer for the  $k$  -th task.
3.2 The Prompting Block of TransHP
The proposed TransHP selects an intermediate transformer block  $B_{l}$  and reshapes it into a prompting block for injecting the coarse-class information. Let us assume that there are  $M$  coarse classes.
Table 1: The balance parameters used for  $\mathcal{L}_{\text{coarse}}$  of different levels (The last 1 is the balance parameter for the final classification.). “-” denotes that this transformer layer does not have prompt tokens.
λ	0	1	2	3	4	5	6	7	8	9	10	11
ImageNet	0.1	0.1	0.1	0.1	0.1	0.15	0.15	0.15	0.15	1	1	1
iNaturalist-2018	-	-	-	-	-	-	1	-	-	-	-	1
iNaturalist-2019	-	-	-	-	-	-	1	-	-	-	-	1
CIFAR-100	-	-	-	-	-	-	-	-	1	-	-	1
DeepFashion	-	-	-	-	-	-	0.5	-	1	-	-	1
Correspondingly, TransHP uses  $M$  learnable prompt tokens  $\mathbf{P}_M = [\mathbf{p}_0,\mathbf{p}_1,\dots,\mathbf{p}_{M - 1}]$  to represent these coarse classes. Our intention is to inject  $\mathbf{p}_k$  into the prompting layer, if the input image belongs to the  $k$ -th coarse class.
Instead of manually selecting the  $k$ -th prompt  $\mathbf{p}_k$  (as in Eqn. 2), TransHP pre-pends the whole prompting pool  $\mathbf{P}_M = [\mathbf{p}_0, \mathbf{p}_1, \dots, \mathbf{p}_{M-1}]$  to the prompting layer and makes the prompting layer automatically select  $\mathbf{p}_k$  for absorption. Specifically, through our design, TransHP learns to automatically 1) predict the coarse class, 2) select the corresponding prompt for absorption through "soft weighting", i.e., high absorption on the target prompt and low absorption on the non-target prompts. The learning procedure is illustrated in Fig. 2 (i). The output of the prompting layer is derived by:
$$
\left[ \mathbf {x} _ {c l s} ^ {l}, \mathbf {X} ^ {l}, \hat {\mathbf {P}} _ {M} \right] = B _ {l} \left(\left[ \mathbf {x} _ {c l s} ^ {l - 1}, \mathbf {X} ^ {l - 1}, \mathbf {P} _ {M} \right]\right), \tag {3}
$$
where  $\hat{\mathbf{P}}_M$  is the output state of the prompt pool  $\mathbf{P}_M$  through the  $l$ -th transformer block  $B_{l}$ .  $\hat{\mathbf{P}}_M$  will not be further forwarded into the following block. Instead, we use  $\hat{\mathbf{P}}_M$  to predict the coarse classes of the input image. To this end, we compare  $\hat{\mathbf{P}}_M$  against a set of coarse-class prototypes and derive the corresponding similarity scores by:
$$
S = \left\{\mathbf {p} _ {i} ^ {\mathrm {T}} \mathbf {w} _ {i} \right\}, i = 1, 2, \dots , M, \tag {4}
$$
where  $\mathbf{w}_i$  is the learnable prototype of the  $i$ -th coarse class. We further use a softmax plus cross-entropy loss to supervise the similarity scores, which is formulated as:
$$
\mathcal {L} _ {\text {c o a r s e}} = - \log \frac {\mathbf {p} _ {y} ^ {\mathrm {T}} \mathbf {w} _ {y}}{\sum_ {i = 1} ^ {M} \exp \left(\mathbf {p} _ {i} ^ {\mathrm {T}} \mathbf {w} _ {i}\right)}, \tag {5}
$$
where  $y$  is the coarse label of the input image. We note there is a difference between the above coarse classification and the popular classification: the popular classification usually compares a single representation against a set of prototypes. In contrast, our coarse classification conducts a set-to-set comparison (i.e.,  $M$  tokens against  $M$  prototypes).
3.3 Overall Structure
Multiple transformer blocks for multi-level hierarchy. Some HIC datasets (e.g., ImageNet-1000) have a multi-level hierarchy. According to the coarse-to-fine multi-level hierarchy, TransHP may stack multiple prompting blocks. Each prompting block is responsible for a single level in the hierarchy, and the prompting block for the coarser level is placed closer to the bottom of the transformer. The detailed structure is shown in Appendix A. Correspondingly, the overall loss function is formulated as:
$$
\mathcal {L} = \mathcal {L} _ {\text {f i n e}} + \sum_ {l} \lambda_ {l} \cdot \mathcal {L} _ {\text {c o a r s e}} ^ {l}, \tag {6}
$$
where  $\mathcal{L}_{\mathrm{fine}}$  is the final classification loss,  $\mathcal{L}_{\mathrm{coarse}}^l$  is the coarse-level loss from the  $l$ -th prompting block, and  $\lambda_l$  is the balance parameter.
Through the above training, the prompting layer explicitly learns the coarse-class prompts, as well as predicts the coarse class of the input image.
The position of the prompting block. Currently, we do not have an exact position scheme for inserting the prompting block, given that the hierarchy of different datasets varies a lot. However, we recommend a qualitative principle to set the position for inserting the prompts: if the number of
Figure 3: TransHP gradually focuses on the predicted coarse class when absorbing the prompts, yielding an autonomous selection. (a) The absorption weight of the target prompt. (b) The ratio of the target prompt weight against the largest non-target prompt weight. The dataset is CIFAR-100. We visualize these statistics on both the training and validation sets.
coarse classes is small (large), the position of the corresponding prompting blocks should be close to the bottom (top). Based on this principle, we can obtain the roughly-optimal position scheme through cross-verification. We empirically find that TransHP is robust to the position scheme to some extent (Fig. 7 in the Appendix). Table 1 summarizes the setting of the balance parameters and the position of prompting layers.
3.4 TransHP Spontaneously Selects the Target Prompt
We recall that we do not manually select the coarse-class prompt for TransHP. Instead, we concatenate the entire prompt set, i.e.,  $\mathbf{P}_M = \{\mathbf{p}_1,\mathbf{p}_2,\dots ,\mathbf{p}_M\}$ , with the feature tokens. In this section, we will show that after TransHP is trained to convergence, the prompting block will spontaneously select the target prompt  $\mathbf{p}_k$  ( $k$  is the predicted coarse class) for absorption and largely neglect the non-target prompts  $\mathbf{p}_{i\neq k}$ .
Specifically, the self-attention in the transformer make each token absorb information from all the tokens (i.e., the feature tokens and the prompt tokens). In Eqn. 3, given a feature token  $\mathbf{x} \in [\mathbf{x}_{cls}, \mathbf{X}]$  (the superscript is omitted for simplicity), we derive its absorption weights on the  $i$ -th prompt token from the self-attention, which is formulated as:
$$
w (\mathbf {x} \leftarrow \mathbf {p} _ {i}) = \frac {\exp \left(Q (\mathbf {x}) ^ {\mathrm {T}} K (\mathbf {p} _ {i}) / \sqrt {d}\right)}{\sum \exp \left(Q (\mathbf {x}) ^ {\top} K ([ \mathbf {x} _ {c l s} , \mathbf {X} , \mathbf {P} _ {M} ]) / \sqrt {d}\right)}, \tag {7}
$$
where  $Q()$  and  $K()$  project the input tokens into query and keys, respectively.  $d$  is the scale factor.
Based on the absorption weights, we consider two statistics:
- The absorption weight of the target prompt, i.e.,  $w(\mathbf{x} \gets \mathbf{p}_k)$ . It indicates the importance of the target prompt among all the tokens.
- The absorption ratio between the target / largest non-target prompt, i.e.,  $R(\mathrm{T}:\mathbb{N}) = w(\mathbf{x}\leftarrow \mathbf{p}_k) / \max \{w(\mathbf{x}\leftarrow \mathbf{p}_{i\neq k})\}$ . It measures the importance of the target prompt compared with the most prominent non-target prompt.
Fig. 3 visualizes these statistics at each training epoch on CIFAR-100 [25], from which we make two observations:
Remark 1: The importance of the target prompt gradually increases to a high level. From Fig. 3 (a), it is observed that the absorption weight on the prompt token undergoes a rapid increase and finally reaches about 0.09. We note that 0.09 is significantly larger than the averaged weight 1/217 (1 class token + 196 patch tokens + 20 prompt tokens).
Remark 2: The target prompt gradually dominates among all the prompts. From Fig. 3 (b), it is observed that the absorption weight on the target prompt gradually becomes much larger than the non-target prompt weight (about  $4\times$ ).
(1) Baseline  $76.21\%$
(2) No prompts  $77.58\%$
(3) No coarse labels 76.42%
(4) Ours  $78.65\%$
Figure 4: Comparison between TransHP and its variants on ImageNet. 1) A variant uses the coarse labels to supervise the class token in the intermediate layers (No prompts). 2) A variant injects additional tokens without supervision from the coarse-class labels (No coarse labels). 3) TransHP injects coarse-class information through prompt tokens and achieves the largest improvement (Ours).
Table 2: The top-1 accuracy of TransHP on some other datasets (besides ImageNet). "w Pre" or "w/o Pre" denotes the models are trained from ImageNet pre-training or from scratch, respectively.
Accuracy (%)	iNaturalist-2018	iNaturalist-2019	CIFAR-100	DeepFashion
Baseline (w/o Pre)	51.07	57.33	61.77	83.42
TransHP (w/o Pre)	53.22	59.24	67.09	85.72
Baseline (w Pre)	63.01	69.31	84.98	88.54
TransHP (w Pre)	64.21	71.62	86.85	89.93
Combining the above two observations, we infer that during training, the prompting block of TransHP learns to focus on the target prompt  $\mathbf{p}_k$  (within the entire prompt pool  $\mathbf{P}_M$ ) for prompt absorption (Remark 2), yielding a soft-weighted selection on the target prompt. This dynamic absorption on the target prompt largely impacts the self-attention in the prompting layer (Remark 1) and conditions the subsequent feature extraction.
Fig.2 (ii) further visualizes some instances from CIFAR-100 (20 coarse classes) for intuitively understanding the prompt absorption. We note that the coarse prediction may sometimes be incorrect. Therefore, we use the red (green) column to mark the prompts of the true (false) coarse class, respectively. In (a) and (b), TransHP correctly recognizes the coarse class of the input images and makes accurate prompt selection. The prompt of the true class has the largest absorption weight and thus dominates the prompt absorption. In (c) and (d), TransHP encounters some confusion for distinguishing two similar coarse classes (due to their inherent similarity or image blur), and thus makes ambiguous selection. In (e) and (f), TransHP makes incorrect coarse-class prediction and correspondingly selects the prompt of a false class as the target prompt.