TransHP: Image Classification with Hierarchical Prompting
Authors: Wenhao Wang
Yifan Sun
Wei Li
Yi Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: hierarchical image classification
hierarchical prompting
vision transformer
TL;DR: Hierarchical prompting improves visual recognition on accuracy, data efficiency and explainability.

Abstract: This paper explores a hierarchical prompting mechanism for the hierarchical image classification (HIC) task. Different from prior HIC methods, our hierarchical prompting is the first to explicitly inject ancestor-class information as a tokenized hint that benefits the descendant-class discrimination. We think it well imitates human visual recognition, i.e., humans may use the ancestor class as a prompt to draw focus on the subtle differences among descendant classes. We model this prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). TransHP consists of three steps: 1) learning a set of prompt tokens to represent the coarse (ancestor) classes, 2) on-the-fly predicting the coarse class of the input image at an intermediate block, and 3) injecting the prompt token of the predicted coarse class into the intermediate feature. Though the parameters of TransHP maintain the same for all input images, the injected coarse-class prompt conditions (modifies) the subsequent feature extraction and encourages a dynamic focus on relatively subtle differences among the descendant classes. Extensive experiments show that TransHP improves image classification on accuracy (e.g., improving ViT-B/16 by +2.83% ImageNet classification accuracy), training data efficiency (e.g., +12.69% improvement under 10% ImageNet training data), and model explainability. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information. The code is available at: https://github.com/WangWenhao0716/TransHP.

TransHP: Image Classification with Hierarchical Prompting
Wenhao Wang"
Yifan Sun Baidu Inc.
ReLER, University of Technology Sydney
Wei Li Zhejiang University
Yi Yang Zhejiang University
Abstract
This paper explores a hierarchical prompting mechanism for the hierarchical image classification (HIC) task. Different from prior HIC methods, our hierarchical prompting is the first to explicitly inject ancestor-class information as a tokenized hint that benefits the descendant-class discrimination. We think it well imitates human visual recognition, i.e., humans may use the ancestor class as prompt to draw focus on the subtle differences among descendant classes. We model this prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). TransHP consists of three steps: 1) learning set of prompt tokens represent the coarse (ancestor) classes, 2) on-the- fly predicting the coarse class of the input image at intermediate block, and 3) injecting the prompt token of the predicted coarse class into the intermediate feature. Though the parameters of TransHP maintain the same for all input images, the injected coarse-class prompt conditions (modifies) the subsequent feature extraction and encourages a dynamic focus on relatively subtle differences among the descendant classes. Extensive experiments show that TransHP improves image classification on accuracy (e.g., improving ViT-B/16 by +2.83% ImageNet classification accuracy), training data efficiency (e.g., +12.69% improvement under 10% ImageNet training data), and model explainability. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information. The code is available at: https://github.com www.b/.oo////e/e.e.eoo Wenhao07 P/.
Introduction
Hierarchical image classification (HIC) aims to exploit the semantic hierarchy to improve prediction accuracy. More concretely, HIC provides additional coarse labels (e.g., Rose) which indicate the ancestors of the relatively fine labels (e.g., China Rose and Rose Peace). The coarse labels usually do not need manual annotation and can be automatically generated based on the fine labels, e.g. through WordNet [I] or word embeddings (2). Since it barely increases any annotation cost while bringing substantial benefit, HIC is of realistic value and has drawn great research interest
This paper explores novel hierarchical prompting mechanism that well imitates the human visual recognition for HIC. Specifically, person may confuse two close visual concepts (e.g.. China Rose and Rose Peace) when the scope interest is large (e.g., the whole Plantae). However, given a prompt narrowing down the category range (e.g., the rose family), the person can shift his/her focus to the subtle variations within the coarse class. We duplicate this procedure for deep visual recognition
Work done during Wenhao Wang's internship at Baidu Inc. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Actention
Attendion map
Transformer Block R.
Transformer Block
Transformer Block B,
Transformer Block B;
(the prompting block)
3) Absorbing prompt tokens
2) Predicting course casses
Mammal Bind
Fish
Leaminn promot tokeens
Transformer Block B,
Transformer Block B,
Class
Class token
Patch taken
Patch token
Prompt token
(+)
(D)
Figure 1: The comparison between Vision Transformer (ViT) and the proposed Transformer with Hierarchical Prompting (TransHP). In (a), VIT attends to the overall foreground region and recognizes the goldfish from the 1000 classes in ImageNet. In (b), TransHP uses an intermediate block to recognize the input image belonging to the fish family and then injects the corresponding prompt. Afterward, the last block attends to the face and crown, which are particularly informative for distinguishing the goldfish against other fish species. Please refer Fig. for more visualizations. Note that TransHP may have multiple prompting blocks corresponding to multi-level hierarchy.
based on the transformer prompting technique. The transformer prompting typically uses prompts (implemented as tokens vectors) to adapt pre-trained transformer for different downstream tasks 5. 7), domains [8, etc. In this paper, we inject coarse-class prompt into the intermediate stage of transformer. The injected coarse-class prompt will then modify the following feature extraction for this specific coarse class, yielding the so-called hierarchical prompting. To the best of our knowledge, explicitly injecting the coarse class information prompt has never been explored in the HIC community.
We model our hierarchical prompting mechanism into Transformer with Hierarchical Prompting (TransHP). commprrss our TransHP against popular transformer backbone VIT 99. TransHP consists of three steps: 1) TransHP learns set of prompt tokens to represent all the coarse classes and selects an intermediate block as the "prompting block" for injecting the prompts. 2) The prompting block on-the-fly predicts the coarse class of the input image. 3) The prompting block injects the prompt token of the predicted class (i.e., the target prompt token) into the intermediate feature. Specifically, TransHP concatenates the prompt tokens with feature tokens (i.e., the "class" token and the patch tokens) from the preceding block, and then feeds them into the prompting block, where the feature tokens absorb information from the target prompt through cross-attention!
Although TransHP based on the prompting mechanism of the transformer, has fundamental differences against prior transformer prompting techniques. A detailed comparison is in Section 2 We hypothesize this hierarchical prompting will encourage TransHP to dynamically focus on the subtle differences among the descendant classes. Fig. validates hypothesis by visualizing the attention map of the final-block class token. Fig. (a), given goldfish image, the baseline model (VIT) attends to the whole body for recognizing from the entire 1000 classes in ImageNet. In contrast, in Fig. (b), since the intermediate block has already received the prompt of "fish". TransHP mainly attends to the face and crown which particularly informative for distinguishing the goldfish against other fish species. Please refer to Section 4.4 for more visualization examples.
We conduct extensive experiments multiple image classification datasets (e.g., ImageNet 100 and iNaturalist 11 and show that the hierarchical prompting improves the accuracy, data efficiency
In practice, the absorption in "soft" manner which assigns all the prompt tokens with soft weights

---

and explainability of the transformer: (1) Accuracy. TransHP brings consistent improvement on multiple popular transformer backbones and five image classification datasets. For example, on ImageNet, TransHP improves ViT-B/16 [ by +2.83% top-1 accuracy. (2) Data efficiency. While reducing the training data inevitably compromises the accuracy. TransHP maintains better resistance against the insufficient data problem. For example, when we reduce the training data of ImageNet to 10%. TransHP enlarges its improvement over the baseline to +12.69%. (3) Explainability. Through visualization, we observe that the proposed TransHP shares some similar patterns with human visual recognition [2] [3], egg., taking an overview for coarse recognition and then focusing on some critical local regions for the subsequent recognition after prompting. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information.
2 Related Works
Prior HIC methods have never explored the prompting mechanism. We note that the prompting technique has not been introduced to the computer vision community until the very recent 2022 [14]
13 16 17 [8]. Two most recent and state-of-the-art HIC methods in 2022 are not related to the
prompting technique either. Specifically, Guided (3) integrates cost-matrix-defined metric into the supervision of a prototypical network. HiMulConE [4] builds an embedding space in which the distance between two classes is roughly consistent with the hierarchy (e.g., two sibling classes sharing the same ancestor are relatively close, and the classes with different ancestors are far away). Some earlier works [19 20 21 are valuable; however, they are also not directly related to the topic of prompting.
A deeper difference resulting from the prompting mechanism is how the mapping function of the deep model is learned. Specifically, deep visual recognition model can be viewed as a mapping function from the raw image space into the label space. All these prior methods learn shared mapping for all the images to be recognized. In contrast, the proposed TPansHP uses the coarse-class prompt to condition itself (from intermediate block). It can be viewed as specifying an individual mapping for different coarse classes, yielding a set of mapping functions. Importantly, TransHP makes all these mapping functions share the same transformer and conditions the single transformer into different mapping functions through the prompting mechanism.
Prompting was first proposed in NLP tasks 22, 23 24, and then has drawn research interest from the computer vision community, e.g. continual learning [4 image segmentation [16], and neural architecture search [17]. VPT (18) focuses how to fine-tune pre-trained ViT models to downstream tasks efficiently. Prompting can efficiently adapt transformers to different tasks or domains while keeping the transform's 's parameters untouched.
Based on the prompting mechanism, our hierarchical prompting makes some novel explorations. w.K.f the prompting objective, prompting structure, prompt selection manner, and training process. 1) Objective: previous methods usually prompt for different tasks or different domains. In contrast, TransHP prompts for coarse classes in the hierarchy, in analogy to the hierarchical prompting in human visual recognition. 2) Structure: previous methods usually inject prompt tokens to condition the whole model. In contrast, in TransHP, the bottom blocks is completely shared, and the prompt tokens are injected into the intermediate blocks to condition the subsequent inference. Therefore, the prompting follows a hierarchical structure in accordance to the semantic hierarchy under consideration. 3) Prompt selection: TransHP pre-pends all the prompt tokens for different coarse classes and autonomously selects the prompt of interest, which is also new (as to be detailed in Section 3.4). 4) Training process: The prompting technique usually consists of two stages, i.e., pre-training a base model and then learning the prompts for novel downstream tasks. When learning the prompt, the pre-trained model usually frozen. This pipeline is different from our end-to-end pipeline, í.e. no more fine-tuning after this training.
3 Transformer with Hierarchical Prompting
We first revisit a basic transformer for visual recognition (ViT (9) and the general prompting technique in Section 3.1 Afterward, we illustrate how to reshape an intermediate block of the
3

---

probabiiitt entsopy
SoftMax
similarity scores
coarse dass prototypes
[d
(8)
Transformer Block B, la prompting block)
ga12
20.1
2011
class token patch tokees
prompt tokens
(f)
(i)
(i)
Figure 2: (i) A prompting block in TransHP. Instead of manually selecting the prompt of the coarse class, the prompting block pre-pends the whole prompt pool consisting of M prompts (M is the number coarse classes) and performs autonomous selection. Specifically, learns to predict the coarse class (Section 3.2) and spontancously selects the corresponding prompt for absorption through soft weighting (Section 3.4p. i.e., the predicted class has the largest absorption weight. (ii) Autonomous prompt selection. We visualize the absorption weights of all the 20 coarse-class prompts for some CIFAR- 100 images. shows how TransHP selects the prompts when the coarse class prediction is correct (a and b), ambiguous (c and d), and incorrect (e and f), respectively. The red and green columns correspond to the ground-truth (GT) class and the false classes, respectively. The detailed investigation is in Section 3.4
Finally, we investigate how the prompting layer absorbs the prompt tokens into the feature tokens in Section
backbone into a hierarchical prompting block TransHP in Section 3.2
3.1 Preliminaries
Vision Transformer (ViT) first splits an image into N patches ((x; E R33xxP 1, N). where P x P is the patch size) and then embeds each patch into C-dimensional embedding by x; = Embed (r;)). Afterward, ViT concatenates a class token = E RC to the patch tokens and feed them into the stacked transformer blocks, which is formulated as:
[x x = Bi ([xa x 1 1 L
(1)
where X/sa and X are the class token and the patch tokens after the 1-th transformer block B respectively. After the total blocks, the final state of the class token (x/1,) viewed as the deep representation of the input image and is used for class prediction. In this paper, we call the concatenation of class token and patch tokens (i..,, cis x x
as the feature tokens.
Prompting was first introduced in Natural Language Processing switch the same transformer model for different tasks by inserting few hint words into the input sentences. More generally, conditions the transformer different tasks, different domains, etc, without changing the transformer parameters but only changing the prompts. To condition the model for the k-th task (or domain), a popular practice to select prompt Pa from prompt pool P (po: Pi, and pre-pend it to the first block. Correspondingly, Equ. turns into:
p Bi ([x x P
1.--'.PP-111 I1),
(2)
where Pk E P (the superscript is omitted) conditions the transformer for the k-th task.
3.2 The Prompting Block of TransHP
The proposed TransHP selects an intermediate transformer block B, and reshapes into a prompting block for injecting the coarse-class information. Let us assume that there are M coarse classes.

---

Table 1: The balance parameters used for Leearse of different levels (The last 1 is the balance parameter for the final classification.. tokens.
" denotes that this transformer layer does not have prompt
0
2
3
4
6
7
8
10
ImageNet
0.1
0.1
0.1
0.1
0.1
0.15
0.15
0.15
0.15
iNaturalist2200
INaturalist-2011
CIFAR-100
DeepFashion
0.5
Correspondingly, TransHP uses M learnable prompt tokens PM [oo Pi, rey Par-i represent these coarse classes. Our intention is to inject P& into the prompting layer, if the input image belongs the k-tth coarse class.
Instead manually selecting the k-th prompt Px (as in Eqn. 2 TransHP pre-pends the whole prompting pool P MM [Po, Po..PP....PP
to the prompting layer and makes the prompting layer automatically select Рı for absorption. Specifically, through our design, TransHP learns to automat- ically 1) predict the coarse class, 2) select the corresponding prompt for absorption through "soft weighting", i.e., high absorption on the target prompt and low absorption on the non-target prompts. The learning procedure is illustrated in Fig. 2 (i). The output of the prompting layer is derived by:
X X PM
Bi (1x x PM]))
where M is the output state of the prompt pool PM through the /-th transformer block B,. PM will not be further forwarded into the following block. Instead, we use PMs to predict the coarse classes of the input image. To this end, we compare As against a set of coarse-class prototypes and derive the corresponding similarity scores by:
,,, where w, is the learnable prototype of the i-th coarse class. We further use a softmax plus cross- entropy loss to supervise the similarity scores, which formulated as:
S i
py W, Di=1 exp w.:
Leoaree = -log
(5)
M
where y is the coarse label of the input image. We note there is difference between the above coarse classification and the popular classification: the popular classification usually compares a single representation against of prototypes. In contrast, coarse classification conducts set-to-set comparison (i.e., M tokens against M prototypes).
3.3 Overall Structure
Multiple transformer blocks for multi-level hierarchy. Some HIC datasets (e.g.. ImageNet- 1000) have a multi-level hierarchy. According to the coarse-to-fine multi-level hicrarchy, TransHP may stack multiple prompting blocks. Each prompting block is responsible for single level in the hierarchy, and the prompting block for the coarser level placed closer to the bottom of the transformer. The detailed structure is shown in Appendix Correspondingly. the overall loss function is formulated as:
L Lrine + Σ, Av CL Coarsse
where Crine is the final classification loss, C. is the coarse-level loss from the 1-th prompting block, and x is the balance parameter.
Through the above training, the prompting layer explicitly learns the coarse-class prompts, well as predicts the coarse class of the input image.
The position of the prompting block. Currently, we do not have exact position scheme for inserting the prompting block, given that the hierarchy different datasets varies lot. However, recommend qualitative principle to set the position for inserting the prompts: if the number of
5

---

0.10
4.0
0.08
3.5
0.06
0.04
2.0
0.02
train
train
50
100
150 spoch
200
250
300
50 100
150 opoch
200
250
300
(a)
(b)
Figure 3: TransHP gradually focuses on the predicted coarse class when absorbing the prompts. yielding an autonomous selection. (a) The absorption weight of the target prompt. (b) The ratio of the target prompt weight against the largest non-target prompt weight. The dataset is CIFAR-100. We visualize these statistics on both the training and validation sets.
coarse classes is small (large), the position of the corresponding prompting blocks should be close to the bottom (top). Based on this principle, we can obtain the roughly-optimal position scheme through Pross-verification. We empirically find that TransHP is robust the position scheme to extent (Fig. in the Appendix). Table summarizes the setting of the balance parameters and the position of prompting layers.
3.4 TransHP Spontaneously Selects the Target Prompt
We recall that we do not manually select the coarse-class prompt for TransHP. Instead, we concatenate entire prompt set, ie., PM = (Pi, P2. Pw) }. with the feature tokens. In this section, we will show that after TPansHP is trained to convergence, the prompting block will spontaneously select the target prompt p (k is the predicted coarse class) for absorption and largely neglect the non-target prompts Pičk-
Specifically, the self-attention the transformer make each token absorb information from all the tokens (i.e., the feature tokens and the prompt tokens). In Eqn. 3 given feature token x E [xats X] (the superscript is omitted for simplicity), we derive its absorption weights on the i-th prompt token from the self-attention, which is formulated as:
exp (Q(x) K(p.)/Va)
w((x f- p.) Σ exp (Q(x) ((X X, P Pw))/va)
where Q() and K() project the input tokens into query and keys, respectively. is the scale factor. Based on the absorption weights, we consider two statistics:
The absorption weight of the target prompt, i.e., w(x pk). It indicates the importance of the target prompt among all the tokens.
The absorption ratio between the target / largest non-target prompt, iee., R(T:N) = w(x <- pa)/ mmxx(w(x= + p- ) It measures the importance of the target prompt compared with the most prominent non-target prompt.
Fig. 3 visualizes these statistics at each training epoch on CIFAR-100 25 from which make two observations:
Remark 1: The importance of the target prompt gradually increases to high level. From Fig.
(a), is observed that the absorption weight on the prompt token undergoes a rapid increase and finally reaches about 0.09. We note that 0.09 is significantly larger than the averaged weight 1/217 (1 class token + 196 patch tokens 20 prompt tokens).
Remark 2: The target prompt gradually dominates among all the prompts. From Fig. 3 (b), it is
observed that the absorption weight on the target prompt gradually becomes much larger than the non-target prompt weight (about 4x).

---

Lyeee
Legerse
Lyine
11::
1:
fine
ins
Class token
Prompt token
Lcooree
Lemorce
(1) Baseline 76.21%
(2) No prompts
(3) No coarse labels
(4) Ours
77.58%
76.42%
78.65%
Figure 4:
Comparison between TransHP and its variants ImageNet. 1) A variant uses the coarse labels to supervise the class token in the intermediate layers (No prompts). 2) A variant injects additional tokens without supervision from the coarse-class labels (No coarse labels). 3) TransHP injects coarse-class information through prompt tokens and achieves the largest improvement (Ours).
Table 2: The top-1 accuracy of TransHP some other datasets (besides ImageNet). "w Pre" or "w/o Pre" denotes the models are trained from ImageNet pre-training or from scratch, respectively.
Accuracy (%)
iNaturalist-2018
iNaturalist-2019
CIFAR-100
DeepFashion
Baseline (w/o Pre)
51.07
57.33
61.77
83.42
TransHIP (w/o Pre)
53.22
59.24
67.09
85.72
Baseline (w Pre)
63.01
69.31
84.98
88.54
TransHP (w Prc)
64.21
71.62
86.85
89.93
Combining the above two observations, we infer that during training, the prompting block of TransHP learns focus on the larget prompt Рą: (within the entire prompt pool PM) ) for prompt absorption (Remark 2), yielding soft-weighted selection on the target prompt. This dynamic absorption on the target prompt largely impacts the self-attention in the prompting layer (Remark 1) and conditions the subsequent feature extraction.
2 (ii) further visualizes some instances from CIFAR-100 (20 coarse classes) for intuitively understanding the prompt absorption. We note that the coarse prediction may sometimes be incorrect. Therefore, use the red (green) column mark the prompts of the true (false) coarse class, respectively. In (a) and (b), TransHP correctly recognizes the coarse class of the input images and makes accurate prompt selection. The prompt of the true class has the largest absorption weight and thus dominates the prompt absorption. In (c) and (d), TransHP encounters some confusion for distinguishing two similar coarse classes (due to their inherent similarity or image blur), and thus makes ambiguous selection. In (e) and (f), TransHP makes incorrect coarse-class prediction and correspondingly selects the prompt of false class as the target prompt.
Experiments
4.1 Implementation Details
Datasets. We evaluate the proposed TransHP on five datasets with hierarchical labels, i.e., Ima-
geNet [10, iNtturalist-2018 (T]), |Naturalist-2019 [T], CIFAR-100 (25]. and DeepFashion-inshe
226. The hierarchical labels of ImageNet are from WordNet [1] with details illustrated on Mike's
website, Both the iNtturalist-20 8/2019 have two-level hierarchical annotations: a super-category
(14/6 classes) for the genus, and 8, 142/1, 010 categories for the species. CIFAR- also has two- level hierarchical annotations: the coarse level has 20 classes. and the fine level has 100 classes. DeepFashion-instee is retricval dataset with three-level hierarchy. To modify it for the classification task, we random select 1/2 images from each class for training, and the remaining 1/2 images for validation. Both the training and validation set contain 2 coarse classes, 17 middle classes, and 982 fine classes, respectively.
Training and inference details. Our TPa adopts an end-to-end training process. We use lightweight transformer as our major baseline, which has heads (half of ViT-B) and 12 blocks. The dimension of the embedding and the prompt token is 384 (half of ViT-B). We train it for 300 epochs on Nvidia A100 GPUs and Py PoTch. The base learning rate 0.001 with cosine learning rate. We set the batch size, the weight decay and the number of warming up epochs as
7

---

Table 3: TransHP brings consistent improvement on various backbones ImageNet.
Accuracy (%)
ViT-B/16
ViT-L/16
DeiT-S
DeiT-B
Baseline
76.68
76.37
79.82
81.80
TransIIP
79.51
78,80
80.55
82.35
The performance our reproduced ViT- and ViT-L/I6 are slightly worse than 77.91 and 76.53 in its original paper [9) respectively.
Table 4: Comparison between TransHP and two most recent state-of th-the-rrt methods. We replace their CNN backbones with the relatively strong transformer backbone for fair comparison.
Accuracy (%)
ImageNet
iNat-2018
iNat-2019
CIFAR- 100
DeepFashion
Baseline
76.21
63.01
69.31
84.98
88.54
Guided
76.05
63. 11
69.66
85.10
88, 38.32
HiMulConE
77.52
63.46
70.87
85.43
88.87
TransHP
78.65
64.21
71.62
86.85
89.93
1. 024, 0.05 and 5, respectively. Importantly, TransHP only adds small overhead to the baseline. Specifically, compared with the baseline (22.05 million parameters), our TransHP only adds 0.60 million parameters (about +2.7%) for ImageNet. When using ViT-B as the backbone, our TransHP only adds +1.4% parameters. Due to the increase of parameters and the extra cost of the backward of several Leverses. the training time increases by 15% our baseline and 12% VIT-B for ImageNet. For inference, the computation overhead is very light. The baseline and TP both use around 50 seconds finish the ImageNet validation with 8 A100 GPUs.
4.2 TransHP Improves the Accuracy
Improvement on ImageNet and the ablation study. We validate the effectiveness of TransHP on ImageNet and conduct the ablation study by comparing TransHP against two variants, well as the baseline. As illustrated in Fig. 4 the two variants are: 1) we do not inject any prompts, but use the coarse labels to supervise the class token in the intermediate layers: similar with the final fine-level classification, the class token is also used for coarse-level classification. 2) we inject learnable tokens, but do not use the coarse labels as their supervision signal. Therefore, these tokens do not contain any coarse class information. From Fig. 4 we draw three observations as below: 1) Comparing TPansHP against the baseline, we observe a clear improvement of 2..44% top-1 accuracy, confirming the effectiveness of TransHP on ImageNet classification. 2) Variant 1 ("No prompts"") achieves some improvement (+1.37%% ..7%) over the baseline as well, but is still lower than TransHP by 11007. It shows that using the hierarchical labels supervise the intermediate state of the class token is also beneficial. However, since it does not absorb the prompting information, the improvement is relatively small. We thus infer that the hierarchical prompting is superior approach for utilizing the hierarchical labels. 3) Variant 2 ("No coarse labels") barely achieves any improvement over the baseline, though it also increases the same amount of parameters as TransHP. It indicates that the benefit of TransHP is not due to the increase of some trainable tokens. Instead, the coarse class information injected through the prompt tokens matters.
TransHP gains consistent improvements on more datasets. Besides the most commonly used dataset ImageNet, we also conduct experiments some other datasets, i..., iNaturalist-201e 18, INaturalist-200 2001, CIFAR-100 and DecpFashion. For these datasets, we use two settings, i.e., training from scratch (w/o Pre) and finetuning from the prreeeee--rettraiee model (w Pre). The experimental results are shown in Table 2 from which we draw two observations. First, under both settings, TransHP brings consistent improvement over the baselines. Second, when there is no pre-training, the improvement is even larger, especially on small datasets. For example, we note that on the smallest CIFAR-100, the improvement under "w/o Pre" and "w Pre" are +5.32% and +1.77% respectively. We infer it is because TransHP considerably alleviates the huta-h problem of the transformer, which is further validated in Section 4.3
TPa improves various backbones. Besides the light transformer baseline used in all the other parts of this section, Table 3 the proposed TransHP some more backbones, i.e., ViT-B/16
8

---

Table 5: Comparison between TransHP and prior -fate-of-the-ae hierarchical classification methods
under the insufficient data scenario. "N% " means using N% ImageNet training data.
Accuracy (%)
100%
50%
20%
10%
Baseline
76.21
67.87
44.60
25.24
Guided
76.05
67.74
45.02
25.67
HiMulConE
77.52
69.23
48.50
30.76
TransHP
78.65
70.74
53.71
37,93
91. ViT-L/16 191 DeiT-S (27]. and DeiT-B [27]. We observe that for the ImageNet classification, TransHP gains 2.83%, 2.43%, 0.73%, and 0.55% improvement on these four backbones, respectively,
Comparison with state-of-the-a hierarchical classification methods. We compare the proposed TransHP with two most recent hierarchy-baan methods, i.e. Guided [3. HiMulConE [4. We do not include more competing methods because most prior works are based on the convolutional backbones and are thus not directly comparable with ours. Since the experiments on large-scale datasets is very time-consuming, we only select the most recent tfate-of-the-art methods and re-implement them on the same transformer backbone (based on their released code). The experimental results are shown in Table It is clearly observed that the proposed TransHP achieves higher improvement and surpasses the two competing methods. For example, on the five datasets, TransHP surpasses the most recent state-of-the-ar HiMulConE by +1.13% (ImageNet). +0.75% (iNat-2018). +0.75% (iNat-2019), +1.42% (CIFAR-100) and 1.06% (DeepFashion), respectively. We also notice that while Guided achieves considerable improvement on the CNN backbones, its improvement over our transformer backbone is trivial. This is reasonable because improvement over higher baseline (i.e., the transformer backbone) is relatively difficult. This observation is consistent with
4.3 TransHP Improves Data Efficiency
We investigate TransHP under the data-scarce scenario. To this end, we randomly select 1/10, 1 1/5, and 1/2 training data from each class in ImageNet (while keeping the validation set untouched). The results are summarized in Table from which we draw three observations as below:
First, as the training data decreases, all the methods undergo significant accuracy drop. This is reasonable because the deep learning method in its nature is data-hungry, and arguably this data- hungry problem is further underlined in transformer [9]. Second, compared with the baseline and two competing hierarchy-bane decrease. For example, when the training data is reduced from 100% -> 10%, the accuracy drop of the baseline and two competing methods are 50.97%, 50.38% and 46.76%, respectively. In contrast, the accuracy drop of the proposed TransHP (40.72%) is significantly smaller. Third, since TransHP undergoes relatively smaller accuracy decrease, its superiority under the low-data regime is even larger. For example, its surpasses the most competing HiMulConE by 1.13%. 1.51%, 5.21% and 7.17% under the 100%, 50%, 20% and 10% training data, respectively, Combining all these observations, we conclude that TransHP improves the data efficiency. The efficiency can be explained intuitively by drawing upon two perspectives, one philosophical and the other technical. Philosophical Perspective: Imagine knowledge as the essence of everything humans have summarized over time. When you possess knowledge, you have the distilled essence of myriad experiences and learnings. The proposed method leverages this accumulated knowledge. In scenarios where data limited, the power of such distilled knowledge becomes even more pronounced. Technical Perspective: Now, think of data not just as isolated pieces of information but categories. Even when the dataset might seem limited, there could still be ample samples within broader categories. This means that for these 'coarser' categories, accuracy can be achieved rapidly. Once the accuracy at this coarse level is established, the model can then use this foundation to prompt further. It's like planting tree you start with a strong base and then branch out.
methods, TransHP presents much higher resistance against the data
4.4 TransHP Improves Model Explainability
We analyze the receptive field of the class token to understand how TransHP reaches its prediction. Basically, the transformer integrates information across the entire image according to the attention map, yielding its receptive field. Therefore, we visualize the attention map of the class token in Fig.

---

insight for understanding Input image
(fine)
Input image
(fine)
tench
shepsherd
dog
German shepsherd
hen
bird
hen
bomese mountain
dog
berrese mountiin
bird
protective covering
dome
60 ANY
69 MNY
house finch
bird
house finch
doormat
covering
doormat
Figure 5: Visualization of the attention map for analyzing the receptive field. For TransHP, visualize a block before and after receiving the prompt (i.e., coarse and fine), respectively. The "coarse" block favors overview for coarse recognition, and the "fine" block further filters out the non-relevant regions after receiving the prompt.
For the proposed TransHP, we visualize the attention map at the prompting block (which handles the coarse-class information) and the last block (which handles the fine-class information). For the VIT baseline, we only visualize the attention score map of the the last block. We draw two observations
from Fig. 5
First, TransHP has a different attention pattern compared with the baseline. The baseline attention generally covers the entire foreground, which is consistent with the observation in (9]. In contrast, TransHP, although the coarse block attends to the overall foreground as well, the fine block concentrates its attention relatively small and critical regions, in pace with the "prompting predicting" procedure. For example, given the "hen" image on the second row (left), TransHP attends to the overall foreground before receiving the coarse-class prompt (i.e., the bird) and focuses to the eyes and bill for recognizing the "hen" out from the "bird".. Second, TransHP shows better capacity for ignoring the redundant and non-relevant regions. For example, given the "doormat" image on the fourth row (right), TransHP ignores the decoration of "GO AWAY" after receiving the coarse-class prompt of "covering", Similar observation is with the third row (right), where TransHP ignores the walls when recognizing the "dome" out from "protective covering".
Conclusion
This paper proposes novel Transformer with Hierarchical Prompting (TransHP) for image classi- fication. Before giving its final prediction, TransHP predicts the coarse class with an intermediate layer and correspondingly injects the coarse-class prompt to condition the subsequent inference. An intuitive effect of our hicrarchical prompting is: TransHP favors an overview the object for coarse prediction and then concentrates its attention to some critical local regions after receiving the prompt, which is similar to the human visual recognition. We validate the effectiveness of TransHP through extensive experiments and hope the hierarchical prompting reveals the transformers.
We presently focus on the image classification task in this paper, while there are some other tasks that are the potential to benefit from hierarchical annotations, e.g., semantic segmentation. Therefore, would like to extend TransHP for more visual recognition tasks in the future.
Limitation,
10

---

References
[1] George A Miller. WordNet: An electronic lexical database, MIT press, 1998.
[2] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In International Conference on Learning Representations Workshop, 2013.
[3] Loic Landrieu and Vivien Sainte Fare Gamnt. Leveraging class hierarchies with metric-guided prototype learning. In British Machine Vision Conference (BMVC), 2021.
[4] Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16660-16669, 2022.
[5] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
[6] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprini 100.::20090 2021.
[7] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. To- wards unified view of parameter-efficient transfer learning, arXiv preprint arX.v:2110.04366. 2021.
[8] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. arXiv preprin! urXiv: 2022.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image datahase. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[11] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7769-8778 2018.
[12] Omar Johannesson, Ian Thornoon, Irene Smith, Andrey Chetverikov, and Arni Kristjánsson. Visual foraging with fingers and eye gaze. i-Perception, 7(2):204 1666666769951667777, 2016.
[13] M Berk Mirza, Rick A Adams, Christoph Mathys, and Karl Friston. Human visual exploration reduces uncertainty about the sensed world. PloS one, 13(1):e0100429 2018.
[14] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 139-149, 2022.
[15] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su. Vincent Perot, Jennifer Dy, et Dualprompt: Complementary prompting for rehearsal-fiee continual learning. European Conference on Computer Vision, 2022.
[16] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086-7096, 2022.
[17] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. 2022.

---

[18] Menglin Jia, Luming Tang, Cor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision (ECCV). 2022.
[19] Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels. arXiv preprint arXiv:2111.11595, 2021.
[20] Kanishk Jain, Shyamgopal Karthik, and Vineet Gandhi. Test-time amendment with coarse classifier for fine-gra ane-ggrined classification. arXiv preprint arXiv:2302.00336, 2023.
[21] Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste, Wei Di, and Yizhou Yu. Hd-cmn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE international conference on computer vision, pages 2740 2748 2015.
[22] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[23] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL/UCNLP (1), 2021.
[24] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[27] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
12

---

Classify
Transformer Encoder Block Br
Transformer Encoder Block Bx+1
Transformer Encoder Block By (The prompting block)
Transformer Encoder Block B++1
Transformer Encoder Block B, (The prompting block)
Figure 6: The illustration of TransHP with multiple layers of hierarchy. I and are two insider layers, and L is the final layer.
79
78.50
70.57
78.70 78.85
78.00
78,50
78.55
accuracy
78
77.74
77
76,21
10
8
Adding prompt tokens from transformer layer
Figure 7: The top-1 accuracy on ImageNet w.r the transformer layer from which to add prompt tokens. The highest two transformer layers (which do notl have too coarse-level labels) play an important role.
Appendix
A Multiple layers of hierarchy
We illustrate the TransHP Fig. when dataset has multiple layers of hierarchy.
B Coarse-level classes of CIFAR-100
[0]: aquatic mammals, [1]: fish, [2]: flowers, [3]: food containers, [4]: fruit and vegetables, [5]: household electrical devices, [6]: household furniture, [7]: insects, [8]: large carnivores, [9]: large man-made outdoor things, [10]: large natural outdoor scenes, [11]: large omnivores and herbivores, [12]: medium mammals, [13]: non-insect invertebrates. [14]: people, [15]: reptiles, [16]: small mammals, [17]: trees, [18]: vehicles-1, and [19]: vehicles-2.
C Importance analysis of classification at different hierarchical levels
From Table D (Line 1), each transformer layer responsible for one level classification. We remove the prompt tokens from the coarsest level to the finest level. In Fig. 7 denotes that
13

---

Table 6: The analysis of the number of coarse-level classes the CIFAR-100 dataset. "N-class" denotes that there are N classes for the coarse-level classification.
Accuracy (%)
baseline
2-class
S-class
10-class
20-class
w/o Pre
61.77
63.34
63.12
64.47
67.09
w Pre
84.98
86.40
86.35
86,50
86,85
Table 7: Comparison between TransHP with the original baseline and the "No prompts" baseline.
Accuracy (%)
iNat-2018
iNar-2019
CIFAR-100
DeepFashion
Baseline (wfo Pre)
51.07
57.33
61.77
83,42
No prompts (w/o Pre)
51.88
58.45
63.78
84.23
TransHP (w/o Pre)
53.22
59.24
67.09
85,72
Baseline Pre)
63.01
69.31
84.98
88,54
No prompts (w Pre)
63.41
70.73
85.50
89,59
TransHP (w Pre)
64.21
71.62
86.85
89.93
the prompt tokens are added from the nith transformer layer. We conclude that only the last two coarse level classifications (arranged the 9th and 10th transformer layer) contribute most the final classification accuracy. That means: (I) it is not necessary that the number of hierarchy and transformer layers are equal. (2) no need to adjust any parameters from too coarse level hierarchy. (Note that: though the current balance parameter for the 8th transformer layer is 0.15, when it is enlarged further improvement achieved.)
D Analysis of the number of coarse-level classes
As shown in Appendix B the CIFAR-10W 100 dataset has 20 coarse-level classes. When we combine them into 10 coarse-level classes, we have ([O-1]), ([2-17]). ([3-4)), ([5-6]), ([12-16]), ([8-1]), ([14- 151), ([9-10]), ([7-13]), and ([18-19]), When combine them into 5 coarse-level classes, we have ([0-1-12-16)), ((2-17-3-4), ([5-6-9-10), (18-11-1--99), and (17-13114-55). When we combine them into 2 coarse-level classes, we have The experimental results are listed in Table 6
and (223445-.99.0011-111--11
We observe that: 1) Generally, using more coarse-level classes is better. 2) Using only coarse-level classes still brings over 1% accuracy improvement.
E
The comparison with the "No prompts" baseline
In this section, we provide experiments with the "No prompts" baseline. The detail of the "No prompts" baseline is shown in Fig. (2). The experimental results are shown in Table [ We find that though "No prompts" baseline surpasses the original baseline, our TransHP still shows significant superiority over this baseline.
Additional Leoorse with DeiT.
We introduce the experimental results by only adopting Leoarse in DeiT. Note that the Leourse is imposed on the class token as shown in Fig. (2). We find that the TransHP still shows performance improvement compared with only using Leoarse on DeiT-S and DeiT-B: compared with DeiT- S (79.82%) and DeiT-B (81 31180%). "only with Leoarse" achieves 79.98% and 81.76% while the TransHP achieves 80.55% and 82.35%, respectively.
14


Reviews:
[{"review_id": "K6JMxZCrUx", "reviewer": "Reviewer_9axF", "summary": "This works presents a novel hierarchical prompting mechanism for hierarchical image classification, named TransHP. In TransHP, a set of prompt tokens are learnt to represent coarse classes and was injected in the prompting block for coarse class prediction, and the injected prompt token can strengthen the feature. The proposed method was evaluated in several benchmarks and show consistently improvements compared with baseline model.\n", "questions": "I hope the author could address my questions in the weakness.", "limitations": "Yes. The author adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This work is well-written and well-organized, and the proposed method is simple and easy to implement.  \n- Good performance compared to previous methods and baseline model.\n- Visualization in Fig. 5 gives intuition to understand the proposed method.\n- Studies on data-scarce scenario is good.", "weaknesses": "- The ImageNet performance of HiMulConE used in this work is even worse than the official paper that adopt ResNet-50. \n\n- Lacking intuitive explanation on the data efficiency of the proposed method. \n\n- Table 4 shows that in data-scarce senario, the performance of all methods decreases significantly, and the results suggest that TransHP has better performance. However, I wonder if the settings used in this work is too weak? This is because recent semi-supervised work has shown that very high accuracy can be achieved with only 10% of ImageNet data, and some self-supervised pre-training methods (\\eg MIM) can also alleviate the problem of data scarcity.\n\n- The main experiments are conducts on a lightweight ViT, which is a weak baseline. It would be best to conduct more experiments on a stronger backbone & pre-train models.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we hope our response helps clear up your initial concerns/questions. We would be happy to provide further clarifications where necessary. We hope our paper is acceptable based on the clarifications and the point-to-point responses below.\n\n**1. The ImageNet performance of HiMulConE used in this work is even worse than the official paper that adopts ResNet-50.**\n\nAns: Thanks for the question. This phenomenon is reasonable because HiMulConE in our paper uses a lower baseline (ViT, 76.21 top-1 accuracy) instead of the original baseline (ResNet, 77.60 top-1 accuracy) in its official paper. Specifically, in its official paper, HiMulConE improves ResNet-50 from 77.60 to 79.14 (+1.54). In our paper, based on ViT (76.21), HiMulConE achieves a similar improvement (+1.31=>77.52) but is indeed lower than in its official paper. \n\n\n**2. Lacking intuitive explanation on the data efficiency of the proposed method.**\n\nAns: We apologize for the missing. The efficiency of the proposed method in terms of data efficiency can be explained intuitively by drawing upon two perspectives, one *philosophical* and the other *technical*. \n\n**Philosophical Perspective:**\n\nImagine knowledge as the essence of everything humans have summarized over time. When you possess knowledge, you have the distilled essence of myriad experiences and learnings. The proposed method leverages this accumulated knowledge. In scenarios where data is limited, the power of such distilled knowledge becomes even more pronounced. \n\n**Technical Perspective:**\n\nNow, think of data not just as isolated pieces of information but in categories. Even when the dataset might seem limited, there could still be ample samples within broader categories. This means that for these 'coarser' categories, accuracy can be achieved rapidly. Once the accuracy at this coarse level is established, the model can then use this foundation to prompt further. It's like planting a tree - you start with a strong base and then branch out. \n\n\n**3. Table 4 shows that in data-scarce scenario, the performance of all methods decreases significantly, and the results suggest that TransHP has better performance. However, I wonder if the settings used in this work is too weak? This is because recent semi-supervised work has shown that very high accuracy can be achieved with only 10% of ImageNet data, and some self-supervised pre-training methods (\\eg MIM) can also alleviate the problem of data scarcity.**\n\nAns: Thanks for the question. The data-scarce scenario in our setting differs from theirs: semi-supervised works leverage unlabeled data (100% of ImageNet data) as well as labeled data (such as 10% of ImageNet data) to increase classification performance; self-supervised pre-training models use 100% of unlabeled data. Ours are with N% of ImageNet training data and None of the unlabeled data. In conclusion, we focus on the **data-scarcity** problem, while theirs still use all of the data and focus on the **label-scarcity** problem.\n\n**4. The main experiments are conducted on a lightweight ViT, which is a weak baseline. It would be best to conduct more experiments on a stronger backbone & pre-train models.**\n\nAns: Thanks for the question. We also have provided experimental results for the main experiments on the stronger baselines: (1) Table 2 shows the performance on ImageNet with ViT-B/16, ViT-L/16, DeiT-S, and DeiT-B. (2) Table 8 in the supplementary shows the performance of our TransHP on ViT-B/16 with four other datasets: iNaturalist-2018, iNaturalist-2019, CIFAR-100, and DeepFashion. (3)  The experiments of Fig. 4 settings with DeiT-S and DeiT-B backbones are shown in Section I in the supplementary. All of these experiments show the effectiveness of TransHP with stronger baselines and different settings. According to your suggestion, we will also add more experiments on the stronger baselines to improve the paper further.\n"}, {"review_id": "1TyCmaN7Hm", "reviewer": "Reviewer_Y5zG", "summary": "This paper introduces a new approach called hierarchical prompting for hierarchical image classification (HIC). It's incorporated into a model named TransHP, which uses broader class 'prompts' to better distinguish between similar classes. The process improves image classification accuracy, data training efficiency, and model explainability. For example, it enhanced ViT-B/16's ImageNet classification accuracy by 2.83%.", "questions": "1.  In the method part, what does \"soft weighting\" mean? If I understand correctly, soft weighting happens without learning any special weighting parameters, instead it means the attention value toward \"correct\" course prompt increases along the training more epochs.", "limitations": "1. There appear to be certain limitations to the methods employed, particularly for datasets devoid of HIC structures. Understanding the process of implementing these methods on such datasets would be enlightening.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The innovative concept of utilizing additive tokens to garner coarse class data, aimed at enhancing fine visual recognition, holds great potential.\n2. The methodology proposed in this paper appears to be a valuable tool in understanding and interpreting the ViT backbone's functioning and structure.", "weaknesses": "1. The foundation established in this paper seems to lack strength. I recommend utilizing a more robust baseline within the High Interaction Control (HIC) domain. The chosen backbone, a lightweight Vision Transformer (ViT), does not exhibit wide applicability. It would enhance the argument's validity and strength if the authors considered employing the standard ViT-B as a baseline.\n   \n2. The diagram in Figure 4 implies that the bulk of the performance improvement arises from the coarse loss. This renders the necessity of prompt introductions potentially insignificant, as they contribute to less than a single point in improvements. Therefore, the authors should provide a deeper insight into the rationale behind incorporating prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we hope our response helps clear up your initial concerns/questions. We would be happy to provide further clarifications where necessary.\n\n**1. The foundation established in this paper seems to lack strength. I recommend utilizing a more robust baseline within the High Interaction Control (HIC) domain. The chosen backbone, a lightweight Vision Transformer (ViT), does not exhibit wide applicability. It would enhance the argument's validity and strength if the authors considered employing the standard ViT-B as a baseline.**\n\nAns: Thanks for the question. We also have provided experimental results for the main experiments on the stronger baselines: (1) Table 2 shows the performance on ImageNet with ViT-B/16, ViT-L/16, DeiT-S, and DeiT-B. (2) Table 8 in the supplementary shows the performance of our TransHP on ViT-B/16 with four other datasets: iNaturalist-2018, iNaturalist-2019, CIFAR-100, and DeepFashion. (3)  The experiments of Fig. 4 settings with DeiT-S and DeiT-B backbones are shown in Section I in the supplementary. All of these experiments show the effectiveness of TransHP with stronger baselines and different settings. According to your suggestion, we will also add more experiments on the stronger baselines to improve the paper further.\n\n**2. The diagram in Figure 4 implies that the bulk of the performance improvement arises from the coarse loss. This renders the necessity of prompt introductions potentially insignificant, as they contribute to less than a single point in improvements. Therefore, the authors should provide a deeper insight into the rationale behind incorporating prompts.**\n\nAns: Thanks for the question. Only with coarse labels, the performance is only 77.58% (See Fig. 4 (2)), while with coarse labels and prompts, the performance is 78.65% (See Fig. 4 (4)). When changing the backbone to a stronger one (DeiT, see Supplementary I L424~L426): no prompts bring NO observable improvements, while injecting prompts brings +0.73% (79.82% to 80.55%) improvements for DeiT-S and +0.55% (81.80% to 82.35%) improvements for DeiT-B.\n\n**3. In the method part, what does \"soft weighting\" mean? If I understand correctly, soft weighting happens without learning any special weighting parameters, instead it means the attention value toward \"correct\" course prompt increases along the training more epochs.**\n\nAns: Thanks for the question. Yes, you are right. We will illustrate “soft weighting” clearer in the revision.\n\n\n**4. There appear to be certain limitations to the methods employed, particularly for datasets devoid of HIC structures. Understanding the process of implementing these methods on such datasets would be enlightening.**\n\nAns: Thanks for this concern. We note that our method (as well as many other HIC methods) has the potential to benefit scenarios without hierarchical labels. It is because we can automatically extend the original annotations into hierarchical annotations, which is very economical. Specifically, as stated in Line 23 - Line 26 in the manuscript, given the fine-grained labels, one can autonomously obtain the coarse labels through taxonomy information (e.g., WordNet) or word embedding from language models. For example, the used coarse labels for ImageNet are generated from WordNet automatically. We believe the extra benefit from economic annotation cost is an important reason that the HIC task is highly regarded.\n"}, {"review_id": "ZQXzFoAK9H", "reviewer": "Reviewer_rPvW", "summary": "The paper proposes to use coarse token prompting for the task of image classification. The basic idea is to add coarse label tokens at intermediate layers of ViT and add additional coarse classification loss at intermediate level. It in some way correlates to convolutional networks based Hierarchical classification which incorporate additional branches for coarse level classification at intermediate levels. However, the idea is applied through prompting and the authors claim that this is the first paper to apply prompting for the task of hierarchical classification.", "questions": "- Please provide clarifications on missing details, as mentioned in limitations sections. Clarify on other points raised in the previous section as well.", "limitations": "Task level limitations are mentioned. However, there is nothing which reflects on the functioning of the proposed framework.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of coarse label prompting appears novel \n- Apart from some missing details, the paper is well written and the general idea is easy to comprehend\n- The results show that proposed hierarchical promoting leads to consistent gains across ImageNet, iNaturalist2018/19, CIFAR 100 datasets.", "weaknesses": "- Related work is not well covered. For instance, there are ideas from prior art which could be applied without much difficulty. For instance, [A] uses marginalization over the predicted fine grained labels to obtain coarse level predictions and applies an additional loss there. [B] simply trains two networks (one for each hierarchy and makes a post hoc correction). Comparison with atleast peer reviewer papers like [A] is warranted. Also a flavour of how these different papers have tackled the problem is worthwhile. Authors do mention that do not compare again convolutional methods, however, giving a general idea on branching and contrasting it with prompting will add value to the paper.\n\n\n- Many details are either not clear or missing. \n\n(a) For example, where exactly do they add the prompting, the exact location has to be given. Especially when you use multiple hierarchies, the locations of prompting blocks need to be clarified. Line 159, mentions that they do not have an exact position scheme and lower levels are better. However, they would have used some schema and the authors should mention that? \n\n(b) How many hierarchies are used in the main paper results. It should be clearly mentioned. I could gather that for iNaturalist only two levels are used, however, there are no detail regarding other datasets.\n\n(c) iNaturalist19 in my knowledge has 7 levels of hierarchy (3-kingdom, 4 phylum, 9-class, 34-order, 57-family, 72-genus, 1010 -species). In supplementary the authors mention to use 6 levels for genus and 1010 for species. Where did the 6 come from? \n\n(d) In Table5, supplementary material, how did you get level 11 for iNaturalist dataset? Is the the block number or level number in the Table?\n\n\n- Is it meaningful to do Ablation 2? Thats like sending some random vectors and checking if that helps. Not sure, if that is worthwhile to show.\n\n\n- I could not fully understand the use of absorption weights in equation 7. Is it just used for explainability and statistics (Figure 2)  for illustrating how the model is behaving or is it used while training? Please clarify. \n\n\n- Another variation of your method could be to predict something like P^hat_{cls} and use that for coarse prediction. Did authors try this variation, reducing the need of the learnable prototypes w_i. Please clarify.\n\n\n- Other ablations like changing the position of the prompting level or varying the number of hierarchies used, would be valuable. What if only the bottom two hierarchies are used (leaf and just above). Compare that against full hierarchy.\n\n\n\n[A] Jong-Chyi Su and Subhransu Maji. Semi-supervised learning with taxonomic labels. arXiv:2111.11595, 2021\n[B] Kanishk Jain, Shyamgopal Karthik and Vineet Gandhi. Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification arXiv 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we hope our response helps clear up your initial concerns/questions. We would be happy to provide further clarifications where necessary. We hope our paper is acceptable based on the clarifications and the point-to-point responses below.\n\n**1. Related work is not well covered. For instance, there are ideas from prior art which could be applied without much difficulty...\n[A] Semi-supervised learning with taxonomic labels\n[B] Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification**\n\nAns: We thank this valuable related work and suggestions. \n\nWe reimplement [A] with Transformer as the backbone using its official code on ImageNet, iNaturalist-2018, iNaturalist-2019, CIFAR-100, and DeepFashion datasets. The experimental results are shown in the below table. We also find [B] is a valuable work and will cite [A] in the experimental results and [B] in the related work part. We agree and acknowledge that learning how other papers solve the problem is worthwhile.\n\n\n| Accuracy| ImageNet| iNaturalist-2018 | iNaturalist-2019| CIFAR-100| DeepFashion| \n|------|------|------|------|------|------|\n| Baseline | 76.21 | 63.01 |69.31 | 84.98 |88.54 | \n| [A] | 77.08 | 63.21 | 70.97 | 85.01 | 89.70 | \n| **TransHP** | **78.65** | **64.21** | **71.62** | **86.85** |**89.93**| \n\n\n\nWe have already compared CNN-based methods with prompting in Related Works (Line 78~ Line 85): we find the fundamental difference between our prompting strategy and other general ideas on branching is how the mapping function of the deep model is learned. For more details, please refer to Line 78~ Line 85.\n\n\n**2. Many details are either not clear or missing.\n(a) For example, where exactly do they add the prompting, the exact location has to be given. Especially when you use multiple hierarchies, the locations of prompting blocks need to be clarified. Line 159, mentions that they do not have an exact position scheme and lower levels are better. However, they would have used some schema and the authors should mention that.\n(b) How many hierarchies are used in the main paper results. It should be clearly mentioned. I could gather that for iNaturalist only two levels are used, however, there are no detail regarding other datasets.\n(c) iNaturalist19 in my knowledge has 7 levels of hierarchy (3-kingdom, 4 phylum, 9-class, 34-order, 57-family, 72-genus, 1010 -species). In supplementary the authors mention to use 6 levels for genus and 1010 for species. Where did the 6 come from?\n(d) In Table5, supplementary material, how did you get level 11 for iNaturalist dataset? Is the the block number or level number in the Table?**\n\nAns: Thanks for the question. \n\n(a) The exact location of prompting blocks is in Table 5 of the supplementary material. Table 5 shows the balance parameters and the exact position of transformer layers to add the prompting. For example, for iNatualist-2019, we add the prompting on the 6th transformer block; and after the 11th transformer block, we perform the finest image classification.\n\n(b) The number of hierarchies is also shown in Table 5. We use 12 hierarchies for the ImageNet, 3 hierarchies for the DeepFashion, and 2 hierarchies for the other datasets.\n\n(c) Many thanks for pointing out this. The 6 is from the official dataset (train_val2019.tar.gz) downloaded from Kaggle, which contains the training and validation images in a directory structure following {iconic category name}/{category name}/{image id}.jpg. The 6 classes are amphibians, birds, fungi, insects, plants, and reptiles. We also notice that iNaturalist19 has another version of hierarchies, i.e., the 7 levels of hierarchies, as you mentioned. This kind of annotation may further benefit our TransHP due to its higher hierarchies.\n\n(d) Sorry for the confusion. It is the transformer block number: 11 means the final fine-grained classification after the final (11th) transformer block.\n\n**3. Is it meaningful to do Ablation 2? That's like sending some random vectors and checking if that helps. Not sure, if that is worthwhile to show.**\n\nAns: Thanks for the question. We think that ablation shows that the performance gain is from our prompting design rather than the extra parameters.\n\n**4. I could not fully understand the use of absorption weights in equation 7. Is it just used for explainability and statistics (Figure 2) for illustrating how the model is behaving or is it used while training? Please clarify.**\n\nAns: Sorry for the confusion. They are just used for explainability and statistics and NOT for training. We will clarify this in the paper.\n\n**5. Another variation of your method could be to predict something like P^hat_{cls} and use that for coarse prediction. Did authors try this variation, reducing the need of the learnable prototypes w_i. Please clarify.**\n\nAns: We thank you for the good suggestion. We add this variation and find its performance similar to ours (78.59% VS 78.65%). \n\n**6. Other ablations like changing the position of the prompting level or varying the number of hierarchies used, would be valuable. What if only the bottom two hierarchies are used (leaf and just above). Compare that against full hierarchy.**\n\nAns: Thanks. We have already included this kind of ablation in the supplementary: Fig. 7 shows using two hierarchies achieves 77.71% accuracy, and using three hierarchies achieves 78.50% accuracy, compared to 78.65% gained by all hierarchies.\n\n**7. Task level limitations are mentioned. However, there is nothing which reflects on the functioning of the proposed framework.**\n\nAns: Thanks. We think the functioning limitation of  our method is “we do not have an exact position scheme for inserting the prompting block.” To mitigate this, we suggest a qualitative principle to set the position for inserting the prompts: if the number of coarse classes is small (large), the position of the corresponding prompting blocks should be close to the bottom (top). For more details, please refer to Line 159~166. \n"}, {"review_id": "FqVoxoqyMn", "reviewer": "Reviewer_e5cs", "summary": "This paper aims to improve image classification accuracy by introducing hierarchical prompts. For a dataset with L hierarchy with M prompt at each level, the author proposed to inserting M prompt at each of the L randomly selected transformer layers. Each prompt is tasked to predict the intermediate coarse label in the hierarchy using a supervised manner. While the proposed idea is interesting, its scalability and novelty are limited. See weakness for more details.", "questions": "The author is suggested to address the concerns in the weakness section.", "limitations": "Limitation is mentioned in the end of the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tFigure 4 highlights the difference between the proposed method and the baselines.\n2.\tFigure 2 clearly summarizes the high-level idea of the paper.\n3.\tThe paper is simple to follow.", "weaknesses": "Major\n1.\tThe proposed method inserted M prompt in the lth prompt block (See Eq5 and Eq6). Since each prompt represents a coarse class (L129-L131) at the lth layer of the hierarchy,  the underlying assumption is that the hierarchy is a balanced tree (e.g. each level of the hierarchy has M coarse classes). What will happen if the hierarchy is imbalanced? Will the proposed method still work? The author is suggested to conduct experiments on a dataset that has an imbalanced hierarchy.\n2.\tL103-104 is confusing. Does the model optimize the entire backbone model (e.g. model parameter other than the prompt) + prompt? Or just the prompt?\n3.\tCaption of Figure 2. As shown in Eq2, the prompt is appended after the class and patch token. This contradicts the caption of Figure 2, which claims “the prompting block pre-pends the whole prompt pool consisting of M prompts”.\n4.\tIt is unclear how the proposed method TransHP selects an intermediate transformer block for inserting the prompt (L127). As mentioned by the author in L159-L160, there is no explicit guidance on how to select the block for prompt insertion. Furthermore, there is no discussion for the case where the level of dataset hierarchy exceeds that of the transformer blocks. This indicates that the proposed method is not applicable to deep hierarchy. \n5.\tFigure 4 is a bit misleading and there are some missing baselines (L221). For example, one could insert prompts (purple dots in figure 4) and do coarse classification using the [CLS] token. Is the proposed method better than this baseline?\n6.\tThe caption of Figure 3 mentioned that the prompt is autonomously selected. However, the Eq5 is nothing but a cross-entropy loss. There is nothing to be selected.\n7.\tHow is the proposed method different from a simple baseline that combines HD-CNN [a] into the transformer architecture?\n8.\tThe author is suggested to compare with VPT.\n\n[a] HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we hope our response helps clear up your initial concerns/questions. We would be happy to provide further clarifications where necessary. We hope our paper is acceptable based on the clarifications and the point-to-point responses below.\n\n**1. The proposed method inserted M prompt in the lth prompt block (See Eq5 and Eq6). Since each prompt represents a coarse class (L129-L131) at the lth layer of the hierarchy, the underlying assumption is that the hierarchy is a balanced tree (e.g. each level of the hierarchy has M coarse classes). What will happen if the hierarchy is imbalanced? Will the proposed method still work? The author is suggested to conduct experiments on a dataset that has an imbalanced hierarchy.**\n\nAns: Thanks for the question. We respectfully disagree with that. We do not assume that the hierarchy is a balanced tree: for ImageNet, the number of coarse classes in each hierarchy is 2, 2, 2, 4, 9, 16, 25, 49, 90, 170, 406, and 1000, respectively. The M in each level of the hierarchy is different. The classification in each level is across all coarse classes, such as we perform 170-classes classification in the third to finest layer. We do not meet training problems with such an imbalanced architecture.\n\n**2. L103-104 is confusing. Does the model optimize the entire backbone model (e.g. model parameter other than the prompt) + prompt? Or just the prompt?**\n\nAns: Sorry for the confusion. The entire backbone and prompt are optimized together. We will clarify it in L103-104.\n\n\n**3. Caption of Figure 2. As shown in Eq2, the prompt is appended after the class and patch token. This contradicts the caption of Figure 2, which claims “the prompting block pre-pends the whole prompt pool consisting of M prompts”.**\n\nAns: Sorry for the confusion. Eq. 2 shows a classical use of the prompts. The caption of Fig. 2  corresponds to Eq. 3.\n\n\n**4. It is unclear how the proposed method TransHP selects an intermediate transformer block for inserting the prompt (L127). As mentioned by the author in L159-L160, there is no explicit guidance on how to select the block for prompt insertion. Furthermore, there is no discussion for the case where the level of dataset hierarchy exceeds that of the transformer blocks. This indicates that the proposed method is not applicable to deep hierarchy.**\n\nAns: Thanks for the question. In Line 161-163, we give a qualitative principle to set the position for inserting the prompts. If the level of dataset hierarchy exceeds that of the transformer blocks, we suggest focusing on the last few coarse-level classifications. Supplementary D shows that too coarse level hierarchy does not contribute to the final result.\n \n**5. Figure 4 is a bit misleading and there are some missing baselines (L221). For example, one could insert prompts (purple dots in figure 4) and do coarse classification using the [CLS] token. Is the proposed method better than this baseline?**\n\nAns: We thank you for this suggestion and provide the result of the required experiment: 77.60%. This performance is similar to the “No prompts” setting and performs a litter better than the baseline while worse than our TransHP.\n\n\n**6. The caption of Figure 3 mentioned that the prompt is autonomously selected. However, the Eq5 is nothing but a cross-entropy loss. There is nothing to be selected.**\n\nAns: Sorry for the confusion. Figure 3 shows the attention value toward the \"correct\" coarse prompt increases along the training more epochs. We do not apply an explicit loss on this but observe this happening autonomously.\n\n\n**7. How is the proposed method different from a simple baseline that combines HD-CNN [a] into the transformer architecture?\n[a] HD-CNN: Hierarchical Deep Convolutional Neural Network for Large Scale Visual Recognition**\n\n\nAns: The differences between our method and other hierarchical image classification methods, such as HD-CNN (with the transformer architecture), lie in **how the mapping function of the deep model is learned**. Specifically, simply changing the backbone of a method does not change how the mapping function is learned. For more details, please refer to Line 78~85.\n \n**8. The author is suggested to compare with VPT.**\n\nAns: The differences between our TransHP and the VPT method are significant and fundamental. To be general, VPT duplicates the success of prompt-based efficient tuning from NLP to computer vision and still belongs to the efficient-tuning paradigm. In contrast, our TransHP has nothing to do with efficient tuning: it exploits the semantic hierarchy into prompts and improves image classification under the train-from-scratch paradigm. To be more concrete, our TransHP is significantly different from VPT (and previous efficient-tuning methods based on prompting) regarding four aspects, i.e., the objective, the structure, the prompt selection, and the training process of the prompts (See Line 91~104).\n"}, {"review_id": "z2ytHdg9XX", "reviewer": "Reviewer_7VFy", "summary": "This paper studies the problem of image classification leveraging the idea of hierarchical image classification (HIC), which exploits semantic relations across target classes to learn meaningful distinctive features.\n\nThe core idea of hierarchical classification is that if a model should classify plants and it knows that two species of flowers are from the “rose” family, it can focus on determining features that characterize the two classes instead of looking for more general features that help it associate the classes to the flowers category. \n\nThe authors consider visual transformers as models and introduce a new  “prompting block” that aims at modeling the coarse class to which target classes belong so that the features extracted by the transformer are dynamic with respect to this task-and-image-specific information.\n\nAt an intermediate layer of the transformer, a set of learnable vectors, i.e., prompt tokens which correspond to the number of coarse classes, is prepended to the feature tokens of an image (composed of the class token and the patch tokens). The transformed version of prompt tokens is used to compute the similarity between the tokens and learnable prototypes of the coarse classes via a cross-entropy loss. In this setting, the prompting block explicitly learns the coarse-class prompts while learning to predict the coarse class of the input image.\n\nExperiments demonstrate that training transformers with the prompting block(s) improve performances on tasks where a hierarchy can be defined (e.g., Imagenet, iNaturalist). An ablation study shows that embedding the information of the coarse classes without coarse class prompts already helps to improve the results, but prompting does it even more. Compared to the baselines, the methods are more data-efficient. Moreover, a qualitative study shows that the methods put attention to more distinctive features of the images.\n", "questions": "1. The authors explain that for tasks with multiple hierarchical levels, they add a prompting block for each of them. Do you think it is necessary to exploit all the hierarchical levels? Or maybe the more granular coarse levels are enough for better conditioning? \n2. By construction, the method it’s limited to cases where one is able to build a hierarchical relation across target tasks. However, training ViT on Imagenet with the hierarchical information added via prompting block might help in learning a space that generalizes better than standard ViT. Did you try to use TransHP pre-trained on Imagenet as the backbone to finetune with a few labels on other downstream tasks? Maybe fine-grained?\n", "limitations": "Limitations were mentioned and I asked follow-up questions in a previous section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow and supports the method explanation with good visualizations.\n2. The experiments support the validity of the method and more importantly the claims of the authors\n3. The results confirm the intuition of the authors about the capability of focusing on more discriminant features\n", "weaknesses": "1. Although the method consistently works on the tested tasks, what are the characteristics of the tasks that let us understand that the method generalizes? Number of classes? Number of added prompt blocks (determined by the number of hierarchical levels)? Or other characteristics?\n2. The application of the method is limited to the cases where we have access to a hierarchy among target tasks. In the question section, I ask if we can show that the method allows us to learn better pre-training embeddings that lead to improvements on other downstream tasks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we hope our response helps clear up your initial concerns/questions. We would be happy to provide further clarifications where necessary.\n\n**1. Although the method consistently works on the tested tasks, what are the characteristics of the tasks that let us understand that the method generalizes? Number of classes? Number of added prompt blocks (determined by the number of hierarchical levels)? Or other characteristics?**\n\nAns: Thanks for the question. The characteristics we conclude may benefit other tasks are: (1) The learning process of prompts. Most of the previous prompting methods usually consist of two stages, i.e., pre-training a base model and then learning the prompts for novel downstream tasks. When learning the prompt, the pre-trained model is usually frozen. We show that an end-to-end training scheme is easier and also works. (2) The meaning of prompts. The prepend prompts of the previous works have no specific meaning. In our work, we show each prompt represents a coarse class. In other works, exploring the meaning of prompts may also be helpful.\n\n**2. The application of the method is limited to the cases where we have access to a hierarchy among target tasks. In the question section, I ask if we can show that the method allows us to learn better pre-training embeddings that lead to improvements on other downstream tasks.**\n\nAns: Thanks for the question. Our method (as well as many other HIC methods) has the potential to benefit scenarios without hierarchical labels. It is because we can automatically extend the original annotations into hierarchical annotations, which is very economical. Specifically, as stated in Line 23 - Line 26 in the manuscript, given the fine-grained labels, one can autonomously obtain the coarse labels through taxonomy information (e.g., WordNet) or word embedding from language models. For example, the used coarse labels for ImageNet are generated from WordNet automatically. For the benefit of improvements on downstream tasks, please refer to the below answers.\n\n**3. The authors explain that for tasks with multiple hierarchical levels, they add a prompting block for each of them. Do you think it is necessary to exploit all the hierarchical levels? Or maybe the more granular coarse levels are enough for better conditioning?**\n\nAns: Thanks for the question. It is not necessary to exploit all the hierarchical levels, and the more granular coarse levels are enough. In the supplementary of Section E, we conclude that “​​only the last two coarse level classifications (arranged at the 9th and 10th transformer layer) contribute most to the final classification accuracy.” For more details, please refer to that section and Fig. 7.\n\n**4. By construction, the method it’s limited to cases where one is able to build a hierarchical relation across target tasks. However, training ViT on Imagenet with the hierarchical information added via prompting block might help in learning a space that generalizes better than standard ViT. Did you try to use TransHP pre-trained on Imagenet as the backbone to finetune with a few labels on other downstream tasks? Maybe fine-grained?**\n\nAns: Thanks for the question. We conclude that the TransHP-pre-trained backbone benefits the downstream task. In Table 1, TransHP (w Pre) denotes using the original pre-trained model; if we change it to the TransHP-pre-trained model, the performance on iNaturalist-2018, iNaturalist-2019, CIFAR-100, and DeepFashion can be further improved by +2.14% (64.21% to 66.35%), +1.27% (71.62% to 72.89%), +1.18% (86.85% to 88.03%), and +0.52% (89.93% to 90.45%), respectively.\t\t"}]

Author rebuttal:

Meta-review: Summary:
The paper introduces TransHP, a novel approach that leverages hierarchical prompting for image classification tasks. By injecting prompt tokens representing coarse classes into a prompting block, the method aims to enhance feature strength and improve classification accuracy. The proposed technique is evaluated on various benchmarks, showing consistent improvements over baseline models.


Strengths:
The concept of hierarchical prompting, especially in the context of image classification, is perceived as innovative by some reviewers.
The paper is generally well-written, organized, and easy to follow.
The visualization provided aids in understanding the proposed method.
The results demonstrate that the hierarchical prompting approach leads to consistent gains across multiple datasets.


Concerns:
Scalability and applicability: There are concerns about the method's scalability, especially when dealing with imbalanced hierarchies. The method's applicability is also questioned, particularly for datasets without hierarchical structures.
Technical clarity: Some reviewers have pointed out ambiguities in the paper, such as the exact location of prompt insertion, the number of hierarchies used, and the rationale behind certain design choices.
Experiments: While the method's performance on multiple datasets is acknowledged, there are concerns about the actual improvement brought about by the proposed techniques. The absence of comparisons with certain state-of-the-art methods also raises questions about the paper's thoroughness.
Novelty: The novelty of the proposed method is questioned by some reviewers, especially when compared to existing works in hierarchical classification. The paper's contribution in the context of existing literature is seen as incremental by some.
Baseline: The choice of a lightweight Vision Transformer (ViT) as the backbone is seen as a limitation. Reviewers suggest that a more robust baseline or a standard ViT might offer a more comprehensive evaluation.

After rebuttal, four out of five reviewers note that their major concerns have been addressed, and rate positive. The AC agrees with the majority of the reviewers and decides to accept the paper. The authors should well address the concerns in the final version, taking into account all reviewers' feedback.
Decision: Accept (poster)