Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a method to learn hierarchical information for image classification. The proposed method introduces prompt tokens into the intermediate layers in ViT, and these prompt tokens are used to make a prediction about the coarse class of the input image. This allows the prompts to learn coarse class information and transfer this information with other tokens in the self-attention layers. The method is evaluated on multiple datasets and backbones and shows better performance than baselines.
strength_and_weaknesses: Strength
1. The idea of using prompts to learn hierarchical information is interesting. 
2. The method is evaluated on multiple dataset and backbones and shows better performance than baselines.
3. The paper is easy to follow.

Weakness
1. The paper didn't explain how to determine the coarse classes. It seems that the datasets used in the paper all have defined hierachical labels, which could be directly used for the proposed method. What if there are no hierarchical labels? Do we have to manually label it? Also, what is the influence of the coarse labels on the final performance. For example, if the dataset can only be divided into very few coarse labels, how much improvement the method could get?
2. The paper didn't explain which layer to add the prompt and what is the effect of different layers.
3. The author mentioned multiple layers of hierarchy, it is better to explain this in the method and Fig. 1.
4. In Fig. 4, the no prompt variant can also improve over the baseline. (1) The author should explain more details about the implementation of this variant. (2) I think this variant should be used as the baseline which the proposed method should compare with in all the experiments. The naive baseline don't use hierarchical labels thus it is not fair.
5. In Table 1, some results seem not very strong. E.g., with ImageNet pre-training, the baseline ViT can only achieve 84.98% top-1 accuracy. The public available results of ViT-B with Imagenet pretraining on CIFAR-100 is around 94%. I understand the author used a smaller ViT here. Thus I suggest the author to use ViT-B as the baseline which makes the results stronger and more convincing.
6. In Fig.5, it is better to provide the coarse and fine label of each image.
7. The method seems to be limited to ViT backbone and image classification.
clarity,_quality,_novelty_and_reproducibility: There are some minor details missing in the paper but the paper is overall easy to follow. The proposed method is interesting and reproducible.
summary_of_the_review: I think the proposed idea of using prompts to learn hierarchical information is interesting. But I am not how generalized is the proposed method. I also think the experimental evaluation could be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Hierarchical Prompting that follows coarse-to-fine semantic structure. The authors predefine $M$ learnable prompts (coarse classes) and inject them into the Transformer layers. The $M$ learnable prompts are used to predict the coarse-class prototype $S$ and are optimized by a softmax loss $\mathcal{L}_{\text {coarse }}$. The authors also provide the visualization results to show that TransHP gradually focuses on the predicted coarse class during the training phase. Experimental results on five datasets demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: * Strength

1. The proposed method shows consistent improvements on five selected datasets, various backbones, and the data efficiency setting.

2. The authors provide the visualization results of the attention map to demonstrate how the proposed method improves model explainability.

* Weaknesses

First, I recommend the authors discuss the difference (may provide additional experimental results) with a previous related ECCV paper VPT [1]. Both VPT [1] and TransHP inject extra learning prompts into the visual Transformer encoder, which may weaken the contribution of this paper. This paper looks like just add extra intermediate loss supervision to VPT.

Besides, some implementation details of this paper are not clear to me. I have the following questions:

1. How to select the coarse label $y$ in Eq.(5) ? Does it bring additional annotating requirements?

2. What's the default value of the hyper-parameter $M$ (number of the prompts)? How to pick the value of $M$? Does the value of $M$ same for all the datasets?

3. In Eq.(3), the symbol $l$ means the $l$-th transformer block. What's the value of $l$? Does the authors apply the learnable prompts to one block, or all the blocks?

[1] Visual Prompt Tuning, Jia et al, ECCV 2022
clarity,_quality,_novelty_and_reproducibility: * Novelty

I am concerned about the comparison with one related work VPT.

* Clarity & Reproducibility

See the weaknesses section. The discussion of how to get the values of some hyper-parameters is not clear, which maybe affect the reproducibility.
summary_of_the_review: See the weaknesses section. I hope the authors can address my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to leverage the injetced prompt token to capture the coarse class feature to benefit the visual recognition.
With the additive corase class labels the proposed approach can outperform the baseline with less than 2% additional parameters.

strength_and_weaknesses: Strength:
1. The idea of using additive tokens to extract coarse class information to help the fine visual recognition is promising.
2. The visualization of the attention map in Fig.5 is interesting. Still, I wonder whether fine attention dominates the fine-grained classification, e.g., this attention is the key to distinguishing all classes in one coarse category.

Weakness:
1. For prompt tuning, the backbone is frozen while only the prompt token is learnable. However, I find no specific statement about which parameters need to be trained.
I think the proposed TransHP needs to fintune all the models and cannot be termed as 'prompt'.
In Fig.1, the fire symbol used in prompt tuning always means the only tuned parameter while the ice symbol means the frozen parameter.

2. This is not clear whether some additional paramters with coarse labels rather than the proposed hierarchical prompt tokens are the key to the performance gain.
As shown in Table 2, DeiT-S and DeiT-B is only a little inferior compared with the proposed approach. If we adpot the L_coarse in DeiT, whether the performance can be comparable with the proposed TransHP?

3. In "Remark 1" on page 6, the authors use all the tokens to calculate the absorption weight of the target prompt.
Since the absorption weight of the target prompt represents which coarse label the input image is from, then only use the prompt tokens are nature.
I wonder why use all the tokens including the feature tokens to calculate the absorption weight.
Is this a reweighting mechnasim for feature tokens?

4. How the coarse labels are selected and ablation studies on the coarse labels may be necessary, e.g., the number of the coarse labels (flowers and trees can be combined as one coarse label).

5. In this paper, all the prompt tokens are prepend after the feature tokens and trained with soft weighting.
This means the prompt token, which is sigificantly different from the target coarse label, is also used to guide the fine features.
I wonder whether a hard weighting mechaism can serve as a sparse Mixture of experts to help the fine-grained visual recognition and reduce the computation cost. 
6. I wonder why a new ViT baseline is created in the main experiment when ViT-small and ViT-base are widely adopted.
Actually, ViT-small has a 384-dimension embedding and 6 heads 12 blocks, which is the same as your proposed new lightweight transformer.

7. From my opinion, the proposed hierarchical prompt tokens can learn the coarse label and align the features.
It seems that this architecture is designed for fine-grained tasks.
However, all the experinments are conducted on datasets with several coarse labels. 
For me, the trivial increment on ImageNet (82.80->82.35) at the cost of expensive fintuing is not interesting.
If the proposed hierarchical architecture can achieve great performance gain on fine-grained tasks with tuning only the target prompt token, then this is more meaningful.
So, can the TransHP serves as a good prompt tuning approach for fine-grained tasks?

8. Lcoarse in Eq.5 is confusing, how coarse labels participate in the calculation of loss when only learnable prototypes and prompt tokens are used?
How are the prototypes learned?
clarity,_quality,_novelty_and_reproducibility: The clarity is good.
The Quality is fair.
The Novelty is good.
The Reproducibility is good.
summary_of_the_review: The idea of using additive tokens to extract coarse class information to help the fine visual recognition is promising.
However, I think finetuning all the parameters is not interesting and some experiments are missing and some statements are confusing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper makes the following improvements based on Vision Transformer (ViT).
1. The paper adds a Transformer Block to the existing ViT to predict coarse categories.
2. After obtaining the coarse category results, a simple self-attention mechanism is used to obtain the coarse category weights.
3. Finally, the coarse category is input into the final Transformer Block to get the final result.

The paper has been validated on datasets with hierarchical labels such as ImageNet, iNaturalist-2018, iNaturalist-2019, CIFAR-100, and DeepFashion-inshop.

The paper provides some target prompt weights. The paper argues that this is a manifestation of improving interpretability.
strength_and_weaknesses: Advantages:
1. The idea of introducing coarse class labels is straightforward and efficient. It has been verified in previous CNN work.
2. The idea of Prompt fully improved the results of the original baseline in the experiment.
3. The paper's proposed method seems helpful for interpreting the ViT backbone.

Disadvantages:
1. Lack of training details. I am confused about how to train the model proposed in the paper. Is it necessary to train 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐜𝐤 first? Or 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐜𝐤 and the final classification network are trained together?
2. Lack of analysis of general conditions. If there are no hierarchical labels, can the method proposed in the paper still be applicable? Which improvements are needed?
3. The paper structure and writing are a little confusing. Some of the details are not well explained. For example, the formula symbols are confusing, and the subscripts and subscripts are incorrect.
4. The authors did not discuss the limitations of the method.
clarity,_quality,_novelty_and_reproducibility: The attempt to improve the ViT is incremental innovation. I am concerned about not being able to reproduce the results of the paper. A lot of details in the paper are not clear enough. I have some questions about the implementation details as follows:

1. Did the authors compare whether the training time was increased? What about the inference time?

2. What are the limitations of the method proposed by the authors? How does it need to be modified for the case where there are no hierarchical labels? 

Also, I have some questions about the implementation details.

1. I understand the paper. But I am confused about how to train the model proposed in the paper. Do you need to train 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐜𝐤 first? Or are 𝐩𝐫𝐨𝐦𝐩𝐭𝐢𝐧𝐠 𝐛𝐥𝐨𝐜𝐤 trained together with the final classification network?

2. After obtaining the Absorbing prompt tokens, what are those tokens input to the final Transformer Block? Is there only one token or multiple tokens as input?


3. "The learning procedure is illustrated in Fig. 2 (a)." in front of Equation 3 is wrong here, right? So, where exactly does it refer to in Figure 2?

4. Are all predefined prompt tokens entered in Equation 3? Will the category of Prompt tokens change?

5. Are Equations (4) and (5) using a simple attention mechanism? The meaning of Equation (6) should not be simply adding softmax and cross-entropy loss. Does the author here just want to discover an Absorbing prompt token?

6. Does the author's interpretability mean the weight of the absorption weight and ratio of the target prompt score?
summary_of_the_review: I think there is some innovation in the paper. But there is a lot of ambiguity in organization and presentation. There is also a lack of comparisons of training details and efficiency. 

The paper lacks an analysis of the limitations of the proposed approach. Also, there seems to be only some weight for interpretability. A more detailed analysis of interpretability is lacking. 

So my opinion is marginally above the acceptance threshold.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four experts in the field. The reviewers raised many concerns regarding the paper: 1) Several important details regarding architecture selection, training paradigm, and hyper-parameter search are missing and unclear. 2) The additional annotation and training costs incurred by the requirement of hierarchical labels should be discussed. Considering the reviewers' concerns, we regret that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
justification_for_why_not_higher_score: The manuscript in its current form fails to demonstrate the advantage of hierarchical prompting over other alternatives through convincing comparisons and discussions.
Decision: Reject