Hierarchical Prompting Improves Visual Recognition On Accuracy, Data Efficiency and Explainability
Authors: Wenhao Wang
Yifan Sun
Wei Li
Yi Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: hierarchical prompting
visual recognition
vision transformer

Abstract: When humans try to distinguish some inherently similar visual concepts, e.g., Rosa Peace and China Rose, they may use the underlying hierarchical taxonomy to prompt the recognition. For example, given a prompt that the image belongs to the rose family, a person can narrow down the category range and thus focuses on the comparison between different roses. In this paper, we explore the hierarchical prompting for deep visual recognition (image classification, in particular) based on the prompting mechanism of the transformer. We show that the transformer can take the similar benefit by injecting the coarse-class prompts into the intermediate blocks. The resulting Transformer with Hierarchical Prompting (TransHP) is very simple and consists of three steps: 1) TransHP learns a set of prompt tokens to represent the coarse classes, 2) learns to predict the coarse class of the input image using an intermediate block, and 3) absorbs the prompt token of the predicted coarse class into the feature tokens. Consequently, the injected coarse-class prompt conditions (influences) the subsequent feature extraction and encourages better focus on the relatively subtle differences among the descendant classes. Through extensive experiments on popular image classification datasets, we show that this simple hierarchical prompting improves visual recognition on classification accuracy (e.g., improving ViT-B/16 by $+2.83\%$ ImageNet classification accuracy), training data efficiency (e.g., $+12.69\%$ improvement over the baseline under $10\%$ ImageNet training data), and model explainability.

Under review as a conference paper at ICLR 2023
HIERARCHICAL PROMPTING IMPROVES VISUAL RECOGNITION ON ACCURACY, DATA EFFICIENCY AND EXPLAINABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
When humans try to distinguish some inherently similar visual concepts, e.g.. Rosa Peace and China Rose, they may use the underlying hierarchical taxonomy to prompt the recognition. For example, given a prompt that the image belongs the rose family, person can narrow down the category range and thus fo- cuses on the comparison between different roses. In this paper, we explore the hierarchical prompting for deep visual recognition (image classification, in par- ticular) based on the prompting mechanism of the transformer. We show that the transformer can take the similar benefit by injecting the coarse-class prompts into the intermediate blocks. The resulting Transformer with Hierarchical Prompting (TransHP) very simple and consists of three steps: 1) TransHP learns a set prompt tokens represent the coarse classes, 2) learns to predict the coarse class of the input image using an intermediate block, and 3) absorbs the prompt token of the predicted coarse class into the feature tokens. Consequently, the injected coarse-class prompt conditions (influences) the subsequent feature extraction and encourages better focus on the relatively subtle differences among the descendant classes. Through extensive experiments on popular image classification datasets, we show that this simple hierarchical prompting improves visual recognition on classification accuracy (e.g., improving ViT-B/16 by 2.83% ImageNet classifi- cation accuracy), training data efficiency (e.g., +1..69% improvement over the baseline under 10% ImageNet training data), and model explainability.
INTRODUCTION
For human visual recognition, awareness of the underlying semantic hierarchy is sometimes benefi- cial, especially when the object is difficult to recognize. More specifically, when trying to distinguish some inherently similar visual concepts, person may the hierarchical taxonomy to prompt the recognition. For example, the China Rose is easily confused with the Rosa Peace when the scope- of-interest is the whole Plantae (or even larger). However, given the prompt that the image belongs to the rose family (i.e., the ancestor class), person can narrow down the category range and shift his/her focus the subtle variation between different roses. Therefore, the prompt of the coarse (ancestor) class in the hierarchy conditions (influences) the subsequent inference and benefits the fine (descendant) class recognition.
In this paper, we explore the above hierarchical prompting for deep visual recognition. We base our exploration on the prompting mechanism of the transformer, which typically uses prompt to condition the model for different tasks (Li & Liang, 2021; Gu et al., 2021; He et al., 2021), different domains (Ge et al., 2022), etc. For the first time, we show that in the image classification task, the transformer can benefit from being prompted with coarse class information. To this end, we inject the coarse-class prompts into the intermediate block dynamically condition the subsequent feature extraction. Such a hierarchical prompting is similar in the human visual recognition.
Specifically, exploiting the underlying semantic hierarchy to improve visual recognition has attracted great research interest and yielded several popular tasks, e.g., hierarchical image classification and hierarchical semantic segmentation. Considering that classification is fundamental for many com- puter vision tasks, this paper focuses hierarchical image classification. Many popular image classification datasets (c.g., ImageNet and iNaturalist) can well accommodate this task because they already provide hierarchical annotations ("coarse fine" labels). Compared with prior literature

---

Under review as a conference paper at ICLR 2023
Astention map
Transformer Black B,
Transformer B,
Absorbing prompt tokens
2] Predicting course cless
Transformer Block
Transformer Block
(the prompting block)
Learning prompt tokens
Materal
Transformer Block B;
Transformer
Class absorption
token
Patch token
absorption
Patch token
Prompt token
[A]
(b)
Figure 1: The comparison between Vision Transformer (ViT) and the proposed Transformer with Hierarchical Prompting (TransHP). (a), ViT attends to the overall foreground region and recog- nizes the goldfish from the 1000 classes in ImageNet. In (b), TransHP uses an intermediate block to recognize the input image as belonging to the fish family and then injects the corresponding prompt. Afterwards, the last block attends to the face and crown which are particularly informative for distin- guishing the goldfish against other fish species. Please refer to Fig. for more visualizations. Note that TransHP may have several prompting blocks, and only add one in (b) for demonstration.
this topic, our method has significant differences due the employed prompting mechanism. Please refer to Section 2 (Related Works) for detailed comparison.
We model our intuition into Transformer with Hicrarchical Prompting (TransHP) based on Vision Transformer (ViT) (Fig.) 1 (a)). TransHP is very simple, as illustrated in Fig. 1 (b). Without loss of generality, Fig. assumes the hierarchy has only two levels for simplicity, Lee, coarse level and fine level. In other word, each image has coarse label (e.g., the fish) and fine label (e.g., the goldfish), simultaneously. TransHP selects an intermediate block as the "prompting block" to inject the coarse class information. Specifically, given the feature tokens (i.e., the "class" token and the patch tokens) output from the preceding block, the prompting block concatenates them with a set of prompt tokens. Each prompt token represents coarse class and is learnable (Section 3.2). The prompting block learns to predict the coarse class of the input image and to select the corresponding prompt token through weighted absorption (i.e., high absorption the target prompt and low ab- sorption on the non-target prompt). Therefore, during inference, the prompt injection concentrates on the predicted coarse class on the fly and dynamically conditions the subsequent recognition.
Since our prompting mechanism follows the coarse-to-fine (or ancestortooddescendan)) semantic structure, we term as the hierarchical prompting. We hypothesize this hierarchical prompting (and conditioning) will encourage TransHP to focus on the subtle differences among the descendant classes for better diserimination. Fig. partially validates our hypothesis by visualizing the attention map of the class token in the last transformer block. In Fig. 1 (a), given a goldfish as the input image, the baseline model (ViT) attends to the whole body for recognizing it from the entire 1000 classes in ImageNet. In contrast, in TransHP in Fig. (b), since the intermediate block has already received the prompt of "fish", the final block mainly attends to the face and crown which are particularly informative for distinguishing the goldfish against other fish species. Please refer to Section 4.4 for more visualization examples.
We conduct extensive experiments multiple image classification datasets (e.g., ImageNet (Deng et al., 2009) and iNaturalist (Van Hom et al., 2018)) and show that the hierarchical prompting im- proves the accuracy, data efficiency and explainability of the transformer: (1) Accuracy. TransHP
2

---

Under review as a conference paper at ICLR 2023
brings consistent improvement on multiple popular transformer backbones and five image classifi- cation datasets. For example, on ImageNet, TransHP improves ViT-B/16 (Dosovitskiy et al., 2021) by +2.83% top-1 accuracy. (2) Data efficiency. While reducing the training data inevitably com- promises the accuracy, TransHP maintains hetter resistance against the insufficient data problem. For example, when we reduce the training data of ImageNet to 10%, TransHP enlarges its improve- ment over the baseline to + +12.69% (3) Explainability. Through visualization, we observe that the proposed TransHP shares some similar patterns with human visual recognition (Johannesson al., 2016; Mirza et al., 2018), e.g., taking an overview for coarse recognition and then focusing on some critical local regions for the subsequent recognition after prompting.
2 RELATED WORKS
Hierarchical visual recognition is based on the hierarchy underlying the visual concepts. Many works have explored hierarchical labels for improving the final fine-level classification. For example, Guided (Landrieu & Garnot, 2021) integrates a cost-matrix-effine metric into the supervision of a prototypical network. HiMulConE (Zhang et al., 2022a) builds an embedding space which the distance between two classes are roughly consistent with the hierarchy (e.g., two sibling classes sharing same ancestor are relatively close and the classes with different ancestors are far away).
This paper has fundamental differences from these prior works from the viewpoint of the learned mapping function. Specifically, a deep visual recognition model can be viewed as a mapping func- tion from the raw image space into the label space. All these prior methods learns shared mapping for all the images to recognized. In contrast, the proposed TransHP uses the coarse-class prompt to condition itself (from an intermediate block). It can be viewed specifying individual map- ping for different coarse classes, yielding a set of mapping functions. Importantly, TransHP makes all these mapping functions share same transformer, and conditions the single transformer into different mapping functions through the prompting mechanism.
Prompting first proposed in NLP tasks (Brown et al., 2020; Gao et al., 2021; Jiang al., 2020), and then has drawn research interest from the computer vision community, c.g. continual learning (Wang al., 2022b;a), image segmentation (Lüddecke & Ecker, 2022), and neural architecture search (Zhang et al., 2022b). VPT (Jia et al., 2022) focuses on how to fine-tune pre-trained ViT models to downstream tasks efficiently. Prompting can efficiently adapt transformers to different tasks or domains while keeping the transforms' parameters untouched.
Based on the prompting mechanism, our hierarchical prompting makes some novel explorations, W.E. the prompting objective, prompting structure, prompt selection manner, and training process. 1) Objective: previous methods usually prompt for different tasks or different domains. In contrast, TransHP prompts for coarse classes in the hierarchy, in analogy to the hierarchical prompting in human visual recognition. 2) Structure: previous methods usually inject prompt tokens to condition the whole model. In contrast, in TransHP, the bottom blocks is completely shared, and the prompt tokens are injected into the intermediate blocks to condition the subsequent inference. Therefore, the prompting follows hierarchical structure in accordance to the semantic hierarchy under consid- eration. 3) Prompt selection. TransHP pre- rre-pends all the prompt tokens for different coarse classes and autonomously selects the prompt of interest, which is also new (as to be detailed in Section 3.3). 4) Training process. The prompting technique usually consists of two stages, iee, pre-training a base model and then learning the prompts for novel downstream tasks. When learning the prompt, the pre-trained model is usually frozen. This pipeline is different from our end-to-end pipeline, i.e. no more fine-tuning after this training.
3 TRANSFORMER WITH HIERARCHICAL PROMPTING
We first revisit a basic transformer for visual recognition (VIT (Dosovitskiy et al., 2021)) and the general prompting technique in Section 3.1. Afterwards, we illustrate how to reshape an intermediate block of the backbone into a hierarchical prompting block for TransHP in Section 3.2. Finally, we investigate how the prompting layer absorbs the prompt tokens into the feature tokens in Section 3.3.
3

---

Under review as a conference paper at ICLR 2023
tross-entropy
probability scores
(9
(N)
SoftMax
similarity scores
2034
coanse-dass prototypes
[0
64)
Transformer Block B. la prompting block)
2013
prompt
(i)
(i)
Figure 2: (i) illustrates the structure of a prompting block in TransHP. Instead of manually selecting the prompt of the coarse class, our prompting block pre-pends the whole prompt pool consisting of M prompts (M is the number of coarse classes) and performs autonomous selection. Specifically, learns to predict the coarse class (Section 3.2) and spontaneously selects the corresponding prompt for absorption through soft weighting (Section 3.3), i.e., the predicted class has the largest absorption weight. (ii) visualizes the absorption weights of all the 20 coarse-class prompts for some CIFAR- 100 images. It shows how TPansHP selects the prompts when the coarse class prediction is correct (a and b), ambiguous (c and d), and incorrect (e and f), respectively. The red and green columns correspond to the true and false classes, respectively. The detailed investigation is in Section 3.3.
3.1 PRELIMINARIES FOR VISION TRANSFORMER AND PROMPTING
Vision Transformer (ViT) first splits an image into patches (fI; E R3xxxx i= where P x P is the patch size) and then embeds each patch into C-dimensional embedding by X; = Embed (x+). Afterwards, ViT concatenates a class token x0 E RC to the patch tokens and feed them into the stacked transformer blocks, which is formulated as:
X2, x = Bi (1x xx-)),''') 1=1,2,..,,
(1)
where x/e and X are the class token and the patch tokens after the /-th transformer block B respectively. After the total blocks, the final state of the class token (x(1)) viewed as the deep representation of the input image and is used for class prediction. In this paper, we call the concatenation of class token and patch tokens (ie., [x x'-l]) as the feature tokens.
Prompting was first introduced in Natural Language Processing switch the same transformer model for different tasks by inserting few hint words into the input sentences. More generally, conditions the transformer to different tasks, different domains, etc, without changing the trans- former parameters but only changing the prompts. To condition the model for the k-th task (or domain), a popular practice is to select a prompt Рı from prompt pool P {Po. Pi, and pre-pend it the first block. Correspondingly, Eqn. turns into:
X X pp] = B₁ ([x x xl-1.p(--1) I]),
where ри E P (the superscript is omitted) conditions the transformer for the k-th task.
3.2 THE PROMPTING BLOCK TRANSHP
The proposed TransHP selects intermediate transformer block B; and reshapes it into prompting block for injecting the coarse-class information. Let us assume that there are M coarse classes. Correspondingly, TransHP uses M learnable prompt tokens PM [Po- P1. Pa--1l 1] represent these coarse classes. Our intention is to inject Pa into the prompting layer, if the input image belongs the k-th coarse class.

---

Under review as a conference paper at ICLR 2023
0.10
4.5
4.0
0.08
wxx-
3.5
0.06
3.0
2.5
0.04
2.0
0.02
vel
val
50
100
150 epoch
200
250
300
6 50
100
150 epech
200
250
360
(a)
(b)
TransHP gradually focuses on the predicted coarse class when absorbing the prompts, yielding an autonomous selection. (a) The absorption weight of the target prompt. (b) The ratio of the target prompt weight against the largest non-target prompt weight (Eqn. 8). The dataset is CIFAR- 100. We visualize these statistics on both the training and validation sets.
Figure 3:
Instead of manually selecting the k-th prompt Ps: (as in Eqn. 2), TransHP pre-pends the whole prompting pool PM [Pu. Pi, PM-11 the prompting layer and makes the prompting layer automatically select Pı. for absorption. Specifically, through our design, TransHP learns to automat- ically 1) predict the coarse class, 2) select the corresponding prompt for absorption through "soft weighting", i.e., high absorption on the target prompt and low absorption on the non-target prompts. The learning procedure is illustrated in Fig. 2 (i). The output of the prompting layer is derived by:
Xels X PM = Bi (P X
(3)
where Pa is the output state of the prompt pool PM through the 1-th transformer block Bj. P PM will not be further forwarded into the following block. Instead, we use PM to predict the coarse classes of the input image. To this end, compare P against a set of coarse-class prototypes and derive the corresponding similarity scores by:
S (p. -
(4)
where w; is the learnable prototype of the i-th coarse class. We further use softmax plus cross- entropy loss to supervise the similarity scores, which is formulated as:
P W,
Looaree -log
(5)
Di= exp (p.'w.)
where y is the coarse label of the input image. We note there is a difference between the above coarse classification and the popular classification: the popular classification usually compares single representation against set of prototypes. In contrast, our coarse classification conducts set-to-set comparison (i.e., M tokens against M prototypes). Note that TransHP may have several prompting blocks. We only introduce one prompting block in this section, and other blocks follow similar procedure. The whole picture is demonstrated in Appendix A.
Through the above training, the prompting layer explicitly learns the coarse-class prompts, as well as predicting the coarse class of the input image. Given all the prompts P {P1, P2. .PM) and the predicted coarse class k argmax (p. w:), TransHP will spontaneously select pи for prompt
absorption in soft weighting manner, as be explained in the following Section 3.3.
3.3 INVESTIGATING THE PROMPT ABSORPTION
The self-attention mechanism in the transformer allows each token to absorb information from all the tokens (including the feature tokens and the prompt tokens). In this section, we investigate how much the target prompt (i.e., the prompt of the predicted coarse class) is absorbed through self- attention. In Eqn. 3, given a feature token x E [x,s. X] (the superscript is omitted for simplicity), derive its absorption weights from the self-attention, which is formulated
5

---

Under review as a conference paper at ICLR 2023
Q(x)+ ((Xxtss X, PM
(x) Softmax
(6)
vdz
where Q0 and K() are the operation that maps the input tokens into query and keys, respectively. dg is the scale factor. More specifically, in W W(x), the absorption weight the i-th prompt token is calculated by:
exp
w(x - pi) = Σ exp (Q(x) T K ((xcte) P, PMID/vdx)
(7)
Based on the absorption weights, we consider two statistics:
The absorption weight of the target prompt, i.e., w(x +- ps).
The ratio of the target prompt score to the largest non-target prompt score:
R(T:N)= w(x P&)/ maxfw(x + Piąk)].
(8)
The target prompt weight w(x P&] indicates the importance of the target prompt among all the tokens. :N) measures the importance of the target prompt compared with the most promi- nent non-target prompt. Fig. visualizes these statisties at each training epoch on CIFAR-100 (Krizhevsky et al., 2009), from which we make two observations:
Remark 1: The importance of the target prompt gradually inereases to high level. From Fig. 3 (a). it is observed that the absorption weight on the prompt token undergoes rapid increase and finally reaches about 0.09. We note that 0.09 is significantly larger than the averaged weight 1/217 (there are class token + 196 patch tokens + 20 prompt tokens).
The target prompt gradually dominates among all the prompts. From Fig. 3 (b), it observed that although the absorption weight on the target prompt is close to the non-target prompt weight at the start, it gradually becomes much larger than the non-target prompt weight (about 4x).
Remark 2:
Combining the above two observations, we infer that during training, the prompting block of Tran- sHP learns to focus on the target prompt ps (within the entire prompt pool PM) for prompt absorp- tion (Remark 2), yielding a seig-weighted selection on the target prompt. This dynamic absorption on the target prompt largely impacts the self-attention in the prompting layer (Remark 1) and con- ditions the subsequent feature extraction.
We further visualize some instances for intuitively understanding the prompt absorption in 2 (ii). Specifically, visualize the absorption weights on all the 20 prompts for CIFAR-100. We note that the coarse prediction may sometimes be incorrect. Therefore, we use the red (green) column to mark the prompts of the true (false) coarse class, respectively. In (a) and (b), TransHP correctly recognizes the coarse class of the input images and makes accurate prompt selection. The prompt of the true class has the largest absorption weight and thus dominates the prompt absorption. In (c) and (d), TransHP encounters some confusion for distinguishing two similar coarse classes (due to their inherent similarity image blur), and thus makes ambiguous selection. In (e) and (f), TransHP makes incorrect coarse-class prediction and correspondingly selects the prompt of false class as the target prompt. The names of 20 coarse classes Appendix A.2.
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Please refer Appendix A.3.
4.2 TRANSHP IMPROVES THE ACCURACY
Improvement ImageNet and the ablation study. We validate the effectiveness of TransHP on ImageNet and conduct the ablation study by comparing TransHP against two variants, as well as the baseline. As illustrated in Fig. the two variants are: do not inject any prompts, but use

---

Under review as a conference paper at ICLR 2023
Sytne
Eoarrs
fymm
Lys
Class token
Prompt token
Ecouras
(1) Baseline
(2) No prompts 77.58%
(3) No coarse labels
(4) Ours 78.65%
76.21%
76.42%
Figure 4: Comparison between TransHP and its variants (and the baseline) on ImageNet. 1) A variant uses the coarse labels supervise the class token in the intermediate layers (No prompts). 2) A variant injects additional tokens without supervision from the coarse-class -lass labels (No coarse labels). 3) TransHP injects coarse-class information through prompt tokens and achieves the largest improvement (Ours).
Table 1: The top-1 accuracy of TPansHP on some other datasets (besides ImageNet). "w Pre" or "w/o Pre" denotes the models are trained from ImageNet pre-training or from scratch, respectively.
Accuracy (%)
iNaturalist-2018
iNaturalist-2019
CIFAR-100
DeepFashion
Baseline (w/o Pre)
51.07
57.33
61.77
83.42
TransHP (w/o Pre)
53.22
59.24
67.09
85.72
Baseline (w Pre)
63.01
69.31
84.98
88.54
TransHP Pre)
64.21
71.62
86.85
89.93
the coarse labels supervise the class token in the intermediate layers: similar with the final fine-
level classification, the class token is also used for coarse-levee classification. 2) we inject learnable tokens, but do not use the coarse labels as their supervision signal. Therefore, these tokens do not contain any coarse class information. From Fig. 4, we draw three observations as below:
1) Comparing TransHP against the baseline, we observe a clear improvement of +2.44% top-1 accuracy, confirming the effectiveness of TransHP on ImageNet classification. 2) Variant 1 ("No prompts") achieves some improvement (+1.37%% over the baseline as well, but still lower than TransHP by 1.00%. It shows that using the hierarchical labels supervise the intermediate state of the class token also beneficial. However, since it does not absorb the prompting information, the improvement is relatively small. We thus infer that the hierarchical prompting is superior approach for utilizing the hierarchical labels. 3) Variant 2 ("No coarse labels") barely achieves any improvement over the baseline, though it also increases the same amount of parameters TransHP. indicates that the benefit of TPansHP not due to the increase of some trainable tokens. Instead, the coarse class information injected through the prompt tokens matters.
TransHP gains consistent improvements on more datasets. Besides the most commonly used some other datasets, i.e., iNaturalist-201.
dataset ImtgeNet, we also conduct experiments iNaturalist- 200u CIFAR- 100 and DecpFashion. For these datasets, we use two settings, i.e., training from scratch (w/o Pre) and finetuning from the ImageNet-preerainee model (w Pre). The experimen- tal results are shown in Table 1, from which we draw two observations. First, under both settings, TransHP brings consistent improvement over the baselines. Second, when there is no pre- traininnn the improvement is even larger, especially small datasets. For example, we note that on the smallest CIFAR- -100 the improvement under "w/o Pre" and "w Pre" are +5.32% and +1.87%, re- spectively. We infer it is because TransHP considerably alleviates the data-hungry problem of the transformer, which is further validated in Section 4.3.
TransHP improves various backbones. Besides the light transformer baseline used in all the other parts of this section, Table 2 evaluates the proposed TransHP on some more backbones, i.e. ViT- B/16 (Dosovitskiy et al., 2021), ViT-L/16 (Dosovitskiy al., 2021), DeiT-S (Touvron et al., 2021), and DeÎT-B (Touvron et al., 2021). We observe that for the ImageNet classification, TransHP gains 2.83%, 2.43%, 0.73%, and 0.55% improvement these four backbones, respectively.
Comparison with -tate- hierarchical classification methods. We compare the proposed TransHP with two most recent bissed methods, i.e. Guided (Landrieu & Garnot, 2021),

---

Under review as a conference paper at ICLR 2023
Table 2: TPansHP brings consistent improvement on various backbones ImageNet.
Accuracy (%)
ViT-B/16
ViT-I/16
DeïT-S
Deil-B
Baseline
76.68
76,37"
79.82
81.80
TransHP
79.51
78.80
80.55
82.35
* The performance our reproduced ViT-B/16 and VIT-I/16 are slightly worse than 77.91 and 76.53 in its original paper (Dosovitskiy et al., 2021), respectively.
Table 3: Comparison between TransHP and two most recent state-of thethh-arttttt methods. We replace
their CNN backbones with the relatively strong transformer backbone for fair comparison.
Accuracy (%)
ImageNet
iNat-2018
iNat-2019
CIFAR- 100
DeepFashion
Baseline
76.21
63.01
69.31
84.98
88.54
Guided
76.05
63.11
69.66
85.10
88.32
HiMulConE
77.52
63.46
70.87
85.43
88.87
TransHP
78.65
64.21
71.62
86.85
89.93
HiMulConE (Zhang et al., 2022a). We do not include more competing methods because most prior works are based the convolutional backbones and are thus not directly comparable with ours. Since the experiments on large-scale datasets is very time-consuming, we only select the most re- cent state-of-the-an art methods and re-implement them on the same transformer backbone (based on their released code). The experimental results are shown in Table 3. It is clearly observed that the proposed TransHP achieves higher improvement and surpasses the two competing methods. For example, on the five datasets, TPansHP surpasses the most recent trat-offtthe-art HiMulConE by +1.13% (ImageNet), +0.75% (iNat-2018). +0.75% (iNatt22119, +1.4% (CIFAR-100) and 1.06% (DeepFashion), respectively. We also notice that while Guided achieves considerable improvement on the CNN backbones, its improvement over our transformer backbone is trivial. This is reason- able because improvement over higher baseline (i.e., the transformer backbone) is relatively difficult. This observation is consistent with (Zhang et al., 2022a).
4.3 TRANSHP IMPROVES DATA EFFICIENCY
We investigate TransHP under the data-scarce scenario. To this end, we randomly select 1/10, 1/5, and 1/2 training data from each class in ImageNet (while keeping the validation set untouched). The results are summarized in Table 4, from which we draw three observations as below:
First, as the training data decreases, all the methods undergo significant accuracy drop. This is reasonable because the deep learning method in its nature is data- data-hungry, and arguably this data- hungry problem is further underlined in transformer (Dosovitskiy et al., 2021). Second, compared with the baseline and two competing hierarchy-basee methods, TransHP presents much higher resis- tance against the data decrease. For example, when the training data is reduced from 100% 10%, the accuracy drop of the baseline and two competing methods are 50.97%. 50.38% and 46. 46.76% re- spectively. In contrast, the accuracy drop of the proposed TransHP (40. 40.72%) is significantly smaller. Third, since TransHP undergoes relatively smaller accuracy decrease, its superiority under the low- data regime is even larger. For example, its surpasses the most competing HiMulConE by 1.13%, 1.51 %, 5.21% and 7.17% under the 100%, 50%, 20% and 10% training data, respectively. Combin- ing all these observations, we conclude that TransHP improves the data efficiency.
4.4 TRANSHP IMPROVES MODEL EXPLAINABILITY
We analyze the receptive field of the class token to understand how TransHP reaches its prediction. Basically, the transformer integrates information across the entire image according to the attention map, yielding its receptive field. Therefore, we visualize the attention map of the class token in Fig. 5. For the proposed TransHP, we visualize the attention map at the prompting block (which handles the coarse-class information) and the last block (which handles the fine-class information). For the ViT baseline, we only visualize the attention score map of the the last block. We draw two observations from Fig. 5:
First, TransHP has a different attention pattern compared with the baseline. The baseline attention generally covers the entire foreground, which is consistent with the observation in Dosovitskiy et al.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison between TransHP and prior state-of-the-ard hierarchical classification methods
under the insufficient data scenario. "N% means using N% ImageNet training data.
Accuracy (%)
100%
50%
20%
10%
Baseline
76.21
67.87
44.60
Guided
76.05
67.74
45.02
25.67
HiMulConE
77.52
69.23
48.50
30.76
TransHP
78.65
70.74
53.71
37,93
Imput image
Beseline
TransHi (coamss)
TranshiP (Sme)
Input image
Baseline
TranshiP (course)
TransHP (fine)
shepsherd
shepsherd
hon
bird
hen
dogi
ostrich
bird
ostrich
dome
pro protective covering
domee
60 AMR
60 AMY
Fc MM
doormat
doonmat
Figure 5: Visualization of the attention map for analyzing the receptive field. For TransHP, visualize block before and after the receiving the prompt (i.e., coarse and fine), respectively. The "coarse" block favors overview for recognition and the "fine" block further filters the non-relevant regions after receiving the prompt.
(2021). In contrast, in TransHP, although the coarse block attends to the overall foreground as well, the fine block concentrates its attention relatively small and critical regions, in pace with the "prompting - predicting" procedure. For example, given the "hen" image on the second row (left), TransHP attends the overall foreground before receiving the coarse-class prompt (i.e., the bird) and focuses to the eyes and bill for recognizing the "hen" out from the "bird". Second, TransHP shows better capacity for ignoring the redundant and non-relevant regions. For example, given the "doormat" image on the fourth row (right), TransHP ignores the decoration of "GO AWAY" after receiving the coarse-class prompt "covering". Similar observation with the third row (right), where TransHP ignores the walls when recognizing the "dome" out from "protective covering".
5 CONCLUSION
This paper proposes a novel Transformer with Hierarchical Prompting (TransHP) for image classi- fication. Before giving its final prediction, TransHP predicts the coarse class with an intermediate layer and correspondingly injects the coarse-class prompt to condition the subsequent inference. An intuitive effect of our hierarchical prompting is: TransHP favors overview of the object for coarse prediction and then concentrates its attention to some critical local regions after receiving the prompt, which is similar to the human visual recognition. We validate the effectiveness of Tran- sHP through extensive experiments and hope the hierarchical prompting reveals a new insight for understanding the transformers.

---

Under review as a conference paper at ICLR 2023
ETHIC STATEMENT
Our work strictly follows the ICLR Code of Ethics: e.g. (1) The datasets used for evaluating methods are all publicly available and respect others' privacy. (2) Our work mainly focuses on the theoretical contribution, which hardly brings any harm to any people or the natural environment. (3) We try our best to finish the work uphold high standards of scientific excellence. Under this process, all experimental results are reported accurately and honestly. (4) We acknowledge that there may be bias from the training data, while our proposed method does not include new bias. (5) Our method may fail on out-of-distributie datasets. The proposed method is trained and tested on the same domain, and no guarantce made about whether the trained models are generalizable.
REPRODUCIBILITY STATEMENT
The proposed TPa is reproducible. We provide enough method details and training hyper- parameters in the main text as well as the appendix. The code will be released based on the accep- tance.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: / //openzeview. http:/fooommmddd//e cNNTTT
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL/UCNLP (1), 2021.
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang, Domain adaptation via prompt learning. arXiv preprint 22f2:2200.06688 2022.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning, arXiv preprint Sttt:2109.043332 2021.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of paramette--effcciet transfer learning, arXiv preprint arXiv:2110.04366 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), 2022.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
Omar Johannesson, Ian M Thornton, Irene J Smith, Andrey Chetverikov, and Arni Kristjánsson. Visual foraging with fingers and eye gaze. i-Perception, 712):04416691166377777 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
Loic Landrieu and Vivien Sainte Fare Gamot. Leveraging class hierarchies with metric-guided prototype learning. In British Machine Vision Conference (BMVC), 2021.
10

---

Under review as a conference paper at ICLR 2023
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint wrXiv:2101.001900 2021.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Timo Lüddeecke and Alexander Ecker. Image segmentation using text and image prompts. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7086- 7096, 2022.
George A Miller. WordNet: An electronic lexical database. MIT press, 1998.
M Berk Mirza, Rick A Adams, Christoph Mathys, and Karl J Friston. Human visual exploration reduces uncertainty about the sensed world. PloS one, 13(1)=e019429, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers. & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yuen-YYulo Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. European Conference on Computer Vision, 2022a.
Zifeng Wang, Zizhao Zhang, Yuen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi- label contrastive learning framework. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16660-16669, 2022a.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. 2022b.
A
APPENDIX
A.1 MULTIPLE LAYERS IERRRCIY..
We illustrate the TransHP in Fig, 6 when dataset has multiple layers of hierarchy.
A.2 COARSE-LEVEL CLASSES OF CIFAR-100
[0]: aquatic mammals, [1]: fish, [2]: flowers, [3]: food containers, [4]: fruit and vegetables, [5]: household electrical devices, [6]: household furniture, [7]: insects, [8]: large carnivores, [9]: large
man-made outdoor things, [10]: large natural outdoor scenes, [11]: large omnivores and herbivores, [12]: medium mammals, [13]: non-insect invertebrates, [14]: people, [15]: reptiles, [16]: small mammals, [17]: trees, [18]: vehicles-1, and [19]: vehicles-2.
11

---

Under review as a conference paper at ICLR 2023
Classify
Transformer Encoder Block B,
Transformer Encoder Block B
Transformer Encoder Block B& (The prompting block)
Transformer Encoder Block B
Transformer Encoder Block B, (The prompting block)
Figure 6: The illustration of TransHP with multiple layers of hierarchy. k and I are two insider layers, and is the final layer.
Table 5: The balance parameters used for Leoarse of different levels (The last 1 is the balance parameter for the final classification.). "_" denotes that this transformer layer does not have prompt tokens.
À
0
2
3
4
6
7
8
10
11
ImageNet
0.1
0.1
0.1
0.1
0.1
0.15
0.15
0.15
0.15
1
iNaturalist-22
INaturalist-2019
CIFAR-106 100
1
DeepFashion
0.5
A.3 IMPLEMENTATION DETAILS
Datasets. We evaluate the proposed TransHP on five datasets with hierarchical labels, i.e., Ima- geNet (Deng et al., 2009), INaturalist-2018 (Van Homn et al., 2018), iNaturalist-2019 (Van Hom al., 2018), CIFAR-100 (Krizhevsky et al., 2009), and DeepPashionninshhop (Liu al., 2016). The hierarchical labels of ImageNet are from WordNet (Miller, 1998), with details illustrated on Mike's website. Both the iNaturalist 201 8/2019 have two-level hierarchical annotations: super- cuper-cateoory (14/6 classes) for the genus, and 8, 142/1, 010 categories for the species. CIFAR-100 also has two- level hierarchical annotations: the coarse level has 20 classes, and the fine level has 100 classes. DecpFashion-inshhs is retrieval dataset with three-level hierarchy. To modify it for the classifica- tion task, we random select 1/2 images from each class for training, and the remaining 1/2 images for validation. Both the training and validation set contain 2 coarse classes, 17 middle classes, and 7, 982 fine classes, respectively.
Training details. Our TransHP adopts end-to-end training process. We lightweight trans- former as our major baseline, which has heads (half of ViT-B) and 12 blocks. The dimension of the embedding and the prompt token is 384 (half of ViT-B). We train it for 300 epochs 8 Nvidia
12

---

Under review as a conference paper at ICLR 2023
79
78.50
78.57
78.85
78.66
78.05
accuracy (%)
78.34
78
77.74
77
76.21
76
Adding prompt tokens from nth transformer layer
Figure 7: The top-1 aceuracy on ImageNet w.zt the transformer layer from which to add prompt tokens. The highest two transformer layers (which do notl have too coarse-level labels) play an important role.
Table 6: The analysis of the number of coarse-level classes on the CIFAR-100 dataset. "N-class" denotes that there are N classes for the coarse-level classification.
Accuracy (%)
baseline
2-class
5-class
10-class
20-class
w/o Pre
61.77
63.34
63.12 86.35
64.47 86.50
67.09
Pre
84.98
86.40
86.85
A A100 GPUs and PyTorch (Paszke et al., 2019). The base learning rate is 0.001 with cosine learning rate. We set the batch size, the weight decay and the number of warming up epochs as 1, 024, 0.05 and 5, respectively. We have qualitative principle to set the position for inserting the prompts: if the number of coarse classes is small (large), the position of the corresponding prompting blocks should be close to the bottom (top). Table 5 in Appendix A.4 summarizes the setting of the balance parameters and the position of prompting layers. We note that our setting of the balance parame- ters is not elaborately tuned, and shows some robustness for TransHP. Moreover, we find that when given many levels of hierarchy (e.g., the ImageNet-1000 has at most 12 levels), the benefit from the highest levels (close the root) is relatively trivial (See Appendix A.5). Importantly, TransHP only adds small overhead to the baseline. Specifically, compared with the baseline (22.05 million parameters), our TransHP only adds 0.60 million parameters (about +2.7%) for ImageNet. When using ViT-B as the backbone, our TransHP only adds +1.4% parameters.
A.4 THE BALANCE PARAMETERS OF DIFFERENT DATASETS.
Please refer to Table 5 for the positions to insert prompt and corresponding balance parameters.
A.5 IMPORTANCE ANALYSIS OF CLASSIFICATION AT DIFFERENT HIERARCHICAL LEVELS.
From Table 5 (Line 1), each transformer layer is responsible for one level classification. We remove the prompt tokens from the coarsest level to the finest level. In Fig. 7, denotes that the prompt tokens are added from the nth transformer layer. We conclude that only the last two coarse level classifications (arranged at the 9th and 10th transformer layer) contribute most to the final classifi cation accuracy. That means: (1) it is not necessary that the number of hierarchy and transformer layers are equal. (2) it is no need to adjust any parameters from too coarse level hierarchy. (Note that: though the current balance parameter for the 8th transformer layer is 0.15, when it is enlarged to 1, no further improvement achieved.)
A.6 ANALYSIS OF THE NUMBER OF COARSE-LEVER CLASSES
As shown in Appendix A.2, the CIFAR- 100 dataset has 20 coarse-level classes. When combine them into 10 coarse-level classes, we have ([0-1]). IJ). ([2-17). ([3-4), ([5-6)), ([12-16]), ([8-11], ([14- 15)), (09-10), ((7-13)), and ([18-19)). When combine them into 5 coarse-level classes, have
13

---

Under review as a conference paper at ICLR 2023
Table 7: Comparison between TPansHP with the original baseline and the "No prompts" baseline.
Accuracy (%)
iNat-20T8
1Nat-2019
CIFAR-T00
DeepPashion
Baseline (w/o Pre)
51.07
57.33
61.77
83,42
No prompts (w/o Pre)
51.88
58.45
63.78 78
84.23
TransHP (w/ Pre)
53.22
59.24
67.09
85.72
Baseline (w Pre)
63.01
69.31
84.98
88.54
No prompts (w Pre)
63.41
70.73
85.50
89.59
TransHP (w Pre)
64.21
71.62
86.85
89.93
Table 8: The top-1 accuracy of TransHP on some other datasets (besides ImageNet) with standard VIT-B/16 backbone. "w Pre" "w/o Pre" denotes the models trained from ImageNet pre- training or from scratch, respectively.
Accuracy (%)
iNaturalist20118
INaturalist-2019
CIFAR-I0
DeepFashion
VIT-B/16 (w/o Pre)
52.96
58.24
62.91
84.28
TransHP (w/o Pre)
54.33
60.14
69.32
86.82
V/T-B/16 (w Pre)
64.10
70.22
87.13
89.14
TransHP (w Prc)
66.43
73.14
88.76
90.31
((0-1-12-16)), ((2117(3-4)), ((5-6-9-10)) (18-11-18-19),, and ((7-13-14-15)). When combine them into coarse-level classes, we have ([0-1-7-8-1112--3:1141113 and (2233-4----9-10-117- 18-19]). The experimental results listed in Table 6.
We observe that: 1) Generally, using more coarse-level classes is better. 2) Using only coarse-level classes still brings over 1% accuracy improvement.
A.7 THE COMPARISON WITH THE "No PROMPTS" BASELINE.
In this section, we provide more experiments with the "No prompts" baseline. The detail of the "No prompts" baseline shown in Fig. 4 (2). The experimental results are shown in Table 7. We find that though "No prompts" baseline surpasses the original baseline, our TransHP still shows significant superiority over this baseline.
A.8 MORE EXPERIMENTS WITH THE VIT-B/16 BACKBONE.
In this section, we provide more experiments with the standard ViT-B/16 backbone. The experi- mental results shown in Table 8. We find that matter with pre-trained models without, the TransHP achieves consistent improvement on these datasets.
A.9 ADDITIONAL Lecoarse WITH DEIT.
We introduce the experimental results by only adopting Leourse in DeiT. Note that the Leoarse is imposed on the class token as shown in Fig. (2). We find that the TransHP still shows per- formance improvement compared with only using Lecorse on DeiT-S and DeiT-B: compared with DeiT-S (79.82%) and DeiT-B (81.80 (8%., "only with Lewarse" achieves 79.98% and 81.76% while the TPansH achieves 80.55% and 82.35%. respectively.
A.10 LIMITATION OF TRANSHP.
One limitation of our method that we presently focus on the image classification task in this paper, while there some other tasks that the potential benefit from hierarchical annotations, e.g..
14

---

Under review as a conference paper at ICLR 2023
semantic segmentation. That being said, note that experiments five datasets very comprehensive compared with prior hierarchical image classification literature.
A.11 EFFICIENCY COMPARISON.
Due to the increase of parameters (+2.7% and the extra cost of the backward of several Lecarses, the training time increases by 15% on our baseline and 12% on ViT-B for ImageNet. For inference, the computation overhead is very light. The baseline and TransHP both use around 50 seconds to finish the ImageNet validation with A100 GPUs.
baseline and +1.4% VIT-B for ImageNet)
15


Reviews:
[{"review_id": "rGPPXoHb5a", "reviewer": "Reviewer_N53d", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to leverage the injetced prompt token to capture the coarse class feature to benefit the visual recognition.\nWith the additive corase class labels the proposed approach can outperform the baseline with less than 2% additional parameters.\n", "summary_of_the_review": "The idea of using additive tokens to extract coarse class information to help the fine visual recognition is promising.\nHowever, I think finetuning all the parameters is not interesting and some experiments are missing and some statements are confusing.\n", "strength_and_weaknesses": "Strength:\n1. The idea of using additive tokens to extract coarse class information to help the fine visual recognition is promising.\n2. The visualization of the attention map in Fig.5 is interesting. Still, I wonder whether fine attention dominates the fine-grained classification, e.g., this attention is the key to distinguishing all classes in one coarse category.\n\nWeakness:\n1. For prompt tuning, the backbone is frozen while only the prompt token is learnable. However, I find no specific statement about which parameters need to be trained.\nI think the proposed TransHP needs to fintune all the models and cannot be termed as 'prompt'.\nIn Fig.1, the fire symbol used in prompt tuning always means the only tuned parameter while the ice symbol means the frozen parameter.\n\n2. This is not clear whether some additional paramters with coarse labels rather than the proposed hierarchical prompt tokens are the key to the performance gain.\nAs shown in Table 2, DeiT-S and DeiT-B is only a little inferior compared with the proposed approach. If we adpot the L_coarse in DeiT, whether the performance can be comparable with the proposed TransHP?\n\n3. In \"Remark 1\" on page 6, the authors use all the tokens to calculate the absorption weight of the target prompt.\nSince the absorption weight of the target prompt represents which coarse label the input image is from, then only use the prompt tokens are nature.\nI wonder why use all the tokens including the feature tokens to calculate the absorption weight.\nIs this a reweighting mechnasim for feature tokens?\n\n4. How the coarse labels are selected and ablation studies on the coarse labels may be necessary, e.g., the number of the coarse labels (flowers and trees can be combined as one coarse label).\n\n5. In this paper, all the prompt tokens are prepend after the feature tokens and trained with soft weighting.\nThis means the prompt token, which is sigificantly different from the target coarse label, is also used to guide the fine features.\nI wonder whether a hard weighting mechaism can serve as a sparse Mixture of experts to help the fine-grained visual recognition and reduce the computation cost. \n6. I wonder why a new ViT baseline is created in the main experiment when ViT-small and ViT-base are widely adopted.\nActually, ViT-small has a 384-dimension embedding and 6 heads 12 blocks, which is the same as your proposed new lightweight transformer.\n\n7. From my opinion, the proposed hierarchical prompt tokens can learn the coarse label and align the features.\nIt seems that this architecture is designed for fine-grained tasks.\nHowever, all the experinments are conducted on datasets with several coarse labels. \nFor me, the trivial increment on ImageNet (82.80->82.35) at the cost of expensive fintuing is not interesting.\nIf the proposed hierarchical architecture can achieve great performance gain on fine-grained tasks with tuning only the target prompt token, then this is more meaningful.\nSo, can the TransHP serves as a good prompt tuning approach for fine-grained tasks?\n\n8. Lcoarse in Eq.5 is confusing, how coarse labels participate in the calculation of loss when only learnable prototypes and prompt tokens are used?\nHow are the prototypes learned?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "SuTTuvqJCC", "reviewer": "Reviewer_RDBe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method to learn hierarchical information for image classification. The proposed method introduces prompt tokens into the intermediate layers in ViT, and these prompt tokens are used to make a prediction about the coarse class of the input image. This allows the prompts to learn coarse class information and transfer this information with other tokens in the self-attention layers. The method is evaluated on multiple datasets and backbones and shows better performance than baselines.", "summary_of_the_review": "I think the proposed idea of using prompts to learn hierarchical information is interesting. But I am not how generalized is the proposed method. I also think the experimental evaluation could be improved.", "strength_and_weaknesses": "Strength\n1. The idea of using prompts to learn hierarchical information is interesting. \n2. The method is evaluated on multiple dataset and backbones and shows better performance than baselines.\n3. The paper is easy to follow.\n\nWeakness\n1. The paper didn't explain how to determine the coarse classes. It seems that the datasets used in the paper all have defined hierachical labels, which could be directly used for the proposed method. What if there are no hierarchical labels? Do we have to manually label it? Also, what is the influence of the coarse labels on the final performance. For example, if the dataset can only be divided into very few coarse labels, how much improvement the method could get?\n2. The paper didn't explain which layer to add the prompt and what is the effect of different layers.\n3. The author mentioned multiple layers of hierarchy, it is better to explain this in the method and Fig. 1.\n4. In Fig. 4, the no prompt variant can also improve over the baseline. (1) The author should explain more details about the implementation of this variant. (2) I think this variant should be used as the baseline which the proposed method should compare with in all the experiments. The naive baseline don't use hierarchical labels thus it is not fair.\n5. In Table 1, some results seem not very strong. E.g., with ImageNet pre-training, the baseline ViT can only achieve 84.98% top-1 accuracy. The public available results of ViT-B with Imagenet pretraining on CIFAR-100 is around 94%. I understand the author used a smaller ViT here. Thus I suggest the author to use ViT-B as the baseline which makes the results stronger and more convincing.\n6. In Fig.5, it is better to provide the coarse and fine label of each image.\n7. The method seems to be limited to ViT backbone and image classification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bppdypz0P7C", "reviewer": "Reviewer_fQu3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes the following improvements based on Vision Transformer (ViT).\n1. The paper adds a Transformer Block to the existing ViT to predict coarse categories.\n2. After obtaining the coarse category results, a simple self-attention mechanism is used to obtain the coarse category weights.\n3. Finally, the coarse category is input into the final Transformer Block to get the final result.\n\nThe paper has been validated on datasets with hierarchical labels such as ImageNet, iNaturalist-2018, iNaturalist-2019, CIFAR-100, and DeepFashion-inshop.\n\nThe paper provides some target prompt weights. The paper argues that this is a manifestation of improving interpretability.", "summary_of_the_review": "I think there is some innovation in the paper. But there is a lot of ambiguity in organization and presentation. There is also a lack of comparisons of training details and efficiency. \n\nThe paper lacks an analysis of the limitations of the proposed approach. Also, there seems to be only some weight for interpretability. A more detailed analysis of interpretability is lacking. \n\nSo my opinion is marginally above the acceptance threshold. ", "strength_and_weaknesses": "Advantages:\n1. The idea of introducing coarse class labels is straightforward and efficient. It has been verified in previous CNN work.\n2. The idea of Prompt fully improved the results of the original baseline in the experiment.\n3. The paper's proposed method seems helpful for interpreting the ViT backbone.\n\nDisadvantages:\n1. Lack of training details. I am confused about how to train the model proposed in the paper. Is it necessary to train 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐜𝐤 first? Or 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐜𝐤 and the final classification network are trained together?\n2. Lack of analysis of general conditions. If there are no hierarchical labels, can the method proposed in the paper still be applicable? Which improvements are needed?\n3. The paper structure and writing are a little confusing. Some of the details are not well explained. For example, the formula symbols are confusing, and the subscripts and subscripts are incorrect.\n4. The authors did not discuss the limitations of the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0_Kx5dnaf_r", "reviewer": "Reviewer_S4MJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Hierarchical Prompting that follows coarse-to-fine semantic structure. The authors predefine $M$ learnable prompts (coarse classes) and inject them into the Transformer layers. The $M$ learnable prompts are used to predict the coarse-class prototype $S$ and are optimized by a softmax loss $\\mathcal{L}_{\\text {coarse }}$. The authors also provide the visualization results to show that TransHP gradually focuses on the predicted coarse class during the training phase. Experimental results on five datasets demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "See the weaknesses section. I hope the authors can address my concerns.", "strength_and_weaknesses": "* Strength\n\n1. The proposed method shows consistent improvements on five selected datasets, various backbones, and the data efficiency setting.\n\n2. The authors provide the visualization results of the attention map to demonstrate how the proposed method improves model explainability.\n\n* Weaknesses\n\nFirst, I recommend the authors discuss the difference (may provide additional experimental results) with a previous related ECCV paper VPT [1]. Both VPT [1] and TransHP inject extra learning prompts into the visual Transformer encoder, which may weaken the contribution of this paper. This paper looks like just add extra intermediate loss supervision to VPT.\n\nBesides, some implementation details of this paper are not clear to me. I have the following questions:\n\n1. How to select the coarse label $y$ in Eq.(5) ? Does it bring additional annotating requirements?\n\n2. What's the default value of the hyper-parameter $M$ (number of the prompts)? How to pick the value of $M$? Does the value of $M$ same for all the datasets?\n\n3. In Eq.(3), the symbol $l$ means the $l$-th transformer block. What's the value of $l$? Does the authors apply the learnable prompts to one block, or all the blocks?\n\n[1] Visual Prompt Tuning, Jia et al, ECCV 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject