LPT: Long-tailed Prompt Tuning  for Image Classification
Authors: Bowen Dong
Pan Zhou
Shuicheng YAN
Wangmeng Zuo
Venue: ICLR 2023 poster (2023)

Abstract: For long-tailed classification tasks, most works often pretrain a big model on a large-scale (unlabeled) dataset, and then fine-tune the whole pretrained  model for  adapting to long-tailed data. Though promising, fine-tuning the whole pretrained model tends to suffer from high cost in computation and deployment of different models for different tasks, as well as weakened generalization capability for overfitting to certain features of long-tailed data. To alleviate these issues, we propose an effective Long-tailed Prompt Tuning (LPT) method for long-tailed classification tasks. LPT introduces several trainable prompts into a frozen pretrained model to adapt it to long-tailed data. For better effectiveness, we divide prompts into two groups: 1) a shared prompt for the whole long-tailed dataset to learn general features and to adapt a pretrained model into the target long-tailed domain; and 2) group-specific prompts to gather group-specific features for the samples which have similar features and also to empower the pretrained model with fine-grained discrimination ability. Then we design a two-phase training paradigm to learn these prompts. In the first phase, we train the shared prompt via conventional supervised prompt tuning to adapt a pretrained model to the desired long-tailed domain. In the second phase, we use the learnt shared prompt as query to select a small best matched set for a group of similar samples from the group-specific prompt set to dig the common features of these similar samples, and then optimize these prompts with a dual sampling strategy and the asymmetric Gaussian Clouded Logit loss. By only fine-tuning a few prompts while fixing the pretrained model, LPT can reduce training cost and deployment cost by storing a few prompts, and enjoys a strong generalization ability of the pretrained model. Experiments show that on various long-tailed benchmarks, with only $\sim$1.1\% extra trainable parameters, LPT achieves comparable or higher performance than previous whole model fine-tuning methods, and is more robust to domain-shift.

Published as a conference paper at ICLR 2023
LPT: LONG-TAILED PROMPT TUNING FOR IMAGE CLASSIFICATION
Bowen Dong'
Pan Zhou²
Shuicheng Yan
Wangmeng Zuo 1,3
Harbin Institute of Technology
National University of Singapore 3 Peng Cheng Laboratory
{cndongsky, panzhou3, shuicheng yan] @mmil. com, wmzuo@hit edu, on
ABSTRACT
For long-tailed classification tasks, most works often pretrain a big model on a large-scale (unlabeled) dataset, and then fine-tune the whole pretrained model for adapting to long-tailed data. Though promising, fine-tuning the whole pre- trained model tends to suffer from high cost in computation and deployment of different models for different tasks, as well as weakened generalization capabil- ity for overfitting certain features of long-tailed data. To alleviate these issues, we propose an effective Long-tailes Prompt Tuning (LPT) method for long-tailed classification tasks. LPT introduces several trainable prompts into frozen pre- trained model to adapt it to long-tailed data. For better effectiveness, we divide prompts into two groups: 1) shared prompt for the whole long-tailed dataset learn general features and to adapt a pretrained model into the target long-tailed domain; and 2) group-specific prompts to gather group-specific features for the samples which have similar features and also to empower the pretrained model with fine-grained discrimination ability. Then we design a two-phase training paradigm learn these prompts. In the first phase, we train the shared prompt via conventional supervised prompt tuning to adapt a pretrained model to the de- sired long-tail tailed domain. In the second phase, query to select a small best matched set for group of similar samples from the group-specific prompt set to dig the common features of these similar samples, and then optimize these prompts with a dual sampling strategy and the asymmet- ric Gaussian Clouded Logit loss. By only fine-tuning few prompts while fixing the pretrained model, LPT can reduce training cost and deployment cost by storing few prompts, and enjoys strong generalization ability of the pretrained model. Experiments show that on various long-tailed benchmarks, with only ~ % extra trainable parameters, LPT achieves comparable or higher performance than previ- ous whole model fine-tuning methods, and is more robust to domain-shift. Code is publicly available at https: : / //git tithub. com/ wtt.cccmm//oo/eeee DongSky////////o/ /LPT.
use the learnt shared prompt
INTRODUCTION
Learning from long-tailed data (Cui al., 2019; Kang al., 2020; Zhang et al., 2021b) is very challenging in the deep learning era, since networks often excessively overfit to majority classes while ignoring the minority classes due the overwhelming training sample number of majority classes. To eliminate this negative effect, previous methods focus on three individual aspects: 1) re- sampling the long-tailed data distribution (Kang et al., 2020; Li et al., 2022: 2021a; Ren al., 2020) to achieve balance among all classes in each minibatch data, 2) re- re-weighting the training loss (Cui al., 2019: Li al., 2022: Menon et al., 2021) give heavier weights minority classes, and 3) specially-desinnee decoupled training (Kang et al., 2020), knowledge distillation (L.i et al., 2021b) or ensemble learning (Zhou et al., 2020; Wang et al., 2020).
Although alleviating the negative effect in long-tailed learning in some sense and achieving better overall performance, these methods generally need to train both feature extractors and linear clas- sifiers from scratch or from pretrained models large-scale datasets, e.g. ImageNet (Deng et al., 2009), thus suffering from three issues. Firstly, to adapt to long-tailed data, this whole model fine- tuning requires much higher extra training cost. Secondly, fine-tuning whole model also impairs the generalization ability of the pretrained model, since the pretrained model trained on a large-scale dataset often sees abundant data and enjoys strong discriminative ability various kinds of fea-

---

Published as a conference paper at ICLR 2023
LPT
VL-LTR
VL-LTR
Test Accuracy (%)
Accuracy (%)
LPT
PaCo
47-
RAC
75
RIDE
VPT
VIT-B
44
73
OLTR
Linear
PaCo
41
71
0
40
80
120
160
40 80 120 160
Finetuned Parameters (M)
Finetuned Parameters (M)
(a) Places-LT
(b) iNaturalist 2018
Figure I: Comparison among SoTA long-tailed methods on the Places-LT dataset and iNaturalist 2018 dataset, where the size of each spot indicates the model size of the overall network, including model backbone, classier and prompts. Our LPT only needs 11.1% additional trainable parameters while achieving comparable or higher accuracy on two highly long-tailed datasets.
tures, while fine-tuning often weaken this generalization ability caused by the overfitting to certain features of long-tailed data and hardly handle domain-shift or out-of-distribute data which occur frequently in long-tailed learning. Finally, fine-tuning also results in very different models for dif- ferent learning tasks, which destroys model compatibility and increases practical deployment cost.
Contributions. To alleviate the above issues, we propose a novel and effective Long-tailed Prompt Tuning (LPT) approach. Specifically, LPT builds on pretrained model, e.g. vision transformer (ViT) (Dosovitskiy et al., 2021), and introduces extra trainable prompts into this pretrained model, and finally only fine-tunes these prompts for adapting the pretrained model to long-tailed data at hand. For prompts, there are two kinds, 1) shared prompt for all classes to learn general features (knowledge) and adapt a pretrained model into the target domain; and 2) group-specific prompts to gather group-specific features for those samples which have similar features and also to empower the pretrained model with fine-grained distinguishing ability. For effective training, we design two-phase training framework to learn these two kinds of prompts. In the first phase, LPT optimizes the shared prompt and a classifier on a long-tailed training dataset of interest. For this phase, its target is to 1) adapt the pretrained model to the target domain of interest via prompt tuning, and 2) to empower the pretained model with the trained classifier the discriminative ability to the training data which is a basic to learn the group-specific prompts. During the second phase, we learn the newly added group- speuispe prompt set and further fine-tune classifier used in the first phase. Specifically, given an input, LPT feeds it into the pretrained model with the learnt shared prompt, and views the output class token as the query to select a small set of matched prompts via computing the cosine similarity between query and the corresponding keys from group-specific prompt set. Next, the trainable matched group-specific prompts are introduced into the pretrained model with shared prompts to help learn class-specific attributes, and is trained by asymmetric Gaussian Clouded Logit (A-GCL) loss (Li et al., 2022) with dual sampling strategy.
This LPT can well alleviate the above three issues in existing methods as aforementioned. For training cost, LPT only needs to fine-tune few prompts whose size is much smaller than the pre- trained model, and thus uses much less training cost than fine-tuning whole pretrained model for adaptation. As for generalization ability, LPT only fine-tunes prompt while fixing the pretraining model, and thus enjoys the strong generalization capacity of the pretrained model. On compatibility, LPT shares a pretrained model for different learning tasks, and only needs store the small-sized prompts, largely increasing the model compatibility and reducing practical deployment cost.
As shown in Fig. 1, on various long-tailed classification benchmarks, with only ~1. 1% additional parameters of prompts, LPT achieves comparable or higher performance than the previous methods which fine-tunes whole pretrained model. Especially, with only -siondbbsed data for training and testing, LPT achieves 50.1 % overall classification accuracy and 46.9% few-shot accuracy on Places- LT dataset (Zhou et al., 2017a), and makes 8.9% and 11.6% improvement trained on vision-only data. Besides, more experimental results shows the superiority of LPT and also its generalization and robustness on long-tailed data and also domain shifted data.
the previous methods
2 RELATED WORK
Longetailed Image Classification. To tackle negative effect from the highly imbalanced data dis- tribution, previous works mainly focus three different aspects, i.e.: data re-sampling (Kang al.,
2

---

Published as a conference paper at ICLR 2023
2020: Li et al., 2021a; Ren et al., 2020), which utilizes hand-crafted samplers (Kang et al., 2020), data augmentation (Li et al., 2021a) or meta-learning based sampler (Ren et al., 2020) to balance the training data among head and tail classes; loss re-weighting (Cui et al., 2019; Menon et al., 2021; Li et al., 2022; Jamal et al., 2020; Tan et al., 2020), which focuses on adding hand-crafted bias into the confidence scores (Menon et al., 2021; Li et al., 2022), re-scaling logits by hand-crafted weights (Cui et al., 2019; Tan et al., 2020), or meta-learning saasbeaarning- methods (Jamal et al., 2020); and decoupled training strategies (Kang et al., 2020; Li et al., 2021b) as well ensemble learning methods (Zhou et al., 2020; Wang et al., 2020). Recently, some Aision-language -ased methods (Ma et al., 2021; Tian et al., 2022: Long et al., 2022) have been proposed, which introduce extra language data (Ma et al., 2021: Tian et al., 2022) external database (Long et al., 2022) to generate auxil- iary confidence scores during training and testing, finally fine-tuning the whole CLiP-based model on long-tailed data. Different from above methods which fully fine-tune all parameters, we aim to leverage the powerful unbiased feature representation ability of pretrained models, and construct a prompt tuning method to obtain a flexible yet accurate classifier from long-tailed data.
Efficient Tuning. Efficient Tuning methods (including prompt (Lester et al., 2021; Jia et al., 2022), adapter (Houlsby et al., 2019; He et al., 2022; Nie et al., 2022; Chen et al., 2022), LoRA (Hu al., 2022) and others (Frankle et al., 2021; Touvron et al., 2022)) are designed to utilize representation ability from pretrained models, and fine-tune only few trainable parameters achieve better per- formance on downstream tasks (Zhai et al., 201 9;; Lin et al., 2014; Zhou et al., 2017b). In this paper we focus on prompt tuning (Zhou al., 2022a; Jía et al., 2022; Bahng et al., 2022). Specifically, Jia et al. (2022) introduced prompt into ImageNet (Deng et al.. 2009) pretrained ViT (Dosovitskiy et al., 2021); while Bahng et al. (2022) inserted the prompt on the edges of images and optimize prompts. Wang et al. (2022) also introduced prompt tuning method into continue learning framework, which used multiple learnable prompts to handle corresponding tasks. Different from above works, LPT focuses exploring the transfer ability of prompt tuning with large-scale while highly imbalanced training data, thus achieving comparable and accuracy.
3 PRELIMINARY STUDY
3.1 PERFORMANCE INVESTIGATION OF VPT
Prompt tuning in previous study (Zhou et al., 2022a; Jia et al., 2022) focuses on fine-tuning with limited data from balanced distribution (Zhai et al., 2019), while its transfer learning ability on large-scale long-tailed data (Zhou et al., 2017a; Van Horn et al., 2018) is not explored. To start our method, we first quantitatively evaluate whether prompt tuning benefits long-tailed learning or not. To this end, we investigate ViT-B (Dosovitskiy et al., 2021) pretrained on ImageNet-22 2009) by comparing the performance of linear probing and prompt tuning method, i.e. VPT (Jia et al.. 2022) because of its effectiveness, on the large-scale Places-LT dataset (Zhou et al., 2017a). Specifically, linear probing aims to fine-tune a linear classifier at the top of a pretrained and fixed feature extractor (e.g., ViT (Dosovitskiy et al., 2021)); while VPT often concatenates input tokens with learnable prompts (tokens) and linear classifier at top of pretrained model. During training, we use these two methods to independently optimize their learnable parameters for 20 epochs with well-tuned hyper- parametrrs,,, e.g., SGD with learning rate of 0.02 and weight decay of 1e-4.
(Deng et al.,
Table 1 summarizes the quantitative results of linear probing and VPT. Without class- balanced sampling. VPT achieves 37.52% overall accuracy and surpasses linear probing by 3.94%, 3.33%, 4.52% in terms of many/medium/ffw-shhe accuracy, respectively. Especially, after introducing class- balanced sampling (Kang et al., 2020) which first randomly sample classes from training set then randomly sample inputs with equal numbers in each iteration. VPT achieves 44.17% overall ac- curacy and even surpasses the counterpart by 8.67% in terms of few-shot accuracy. Based on the observation, we conclude that: a) prompt tuning can consistently improve the overall performance in long-tailed classification; and b) prompt tuning are more robust to long-tailed distribution and benefits more to tail categories. However, from Table 1, one can also observe that the performance of prompt tuning on long-tailed problem is not sufficient and yet far behind state-of-the-arts
3.2 ANALYSIS OF PROMPT TUNING
Nevertheless, the reason why prompt tuning improves the performance on long-tailed learning tasks is still unclear. To quantitatively and qualitatively analyze prompt tuning, we conduct a series of experiments Places-LT (Zhou al., 2017a). We first adopt Linear Discriminant Analysis (LDA)
3

---

Published as a conference paper at ICLR 2023
Table I: Prompt tuning results on Places-LT (Zhou al., 2017a). Prompt tuning achieves better
accuracy on all classes and tail classes (iee. "Few" in the table) with different training settings.
Method
Balanced Sampling
Tuned Params (w/o classifier)
Overall
Many
Medium
Few
Lincar VPT
O
33.29% 37.52%
46.48% 50.42% %
29.45% 32.78%
18.77% 23.29 %
92K
Lincar
41.33% 44.17% %
49.47 %
41.31 46.73 %
27.51%
VPT
92K
45.79%
36. %
Table 2: Quantitative analysis of features learned by pretrained ViT-B and VPT. Features from VPT obtain better discriminative ability in terms of cluster compactness and also KNN accuracy.
IN1K (ViT)
IN1k (VPT)
Places (ViT)
Places (VPT),
Method
VIT-B
VPT
Pretrain Data
IN21K
IN21K
Fine-tuned
Inner-class distance R
2.36+0.52
0.82:0443
Inner-class/ /inter-class K-NN Acc
0,171
0.128
30.80%
31.90%
Figure 2: LDA visualization of VPT.
to investigate the learned prompt from domain adaptation perspective. Specifically, we use the pretrained ViT-B and the ViT-B fine-tuned by VPT on Places-LT Sec. 3.1 extract features of ImageNet val set and Places-LT val set, and then employ the above features to obtain corresponding LDA vectors for visualization. From the qualitative result in Fig. 2, one can easily find that a) for pretrained ViT-B, its extracted features from ImageNet (red cluster) are far from its features from Places-LT (green cluster); b) for VPT fine-tuned ViT-B, its extracted features from ImageNet (yellow cluster) align with its features from Places-LT (blue cluster) and are close to each other. So these observations indicate that 1) the learned prompts in VPT could help align the fine-tuned data distribution (Places-LT) LT) with the pretrained data distribution (ImageNet), and thus can make the pretrained model adapt to the target domain for the long-tailed learning tasks.
Next we investigate the learned prompt from group-specific perspective. Specifically, for each class in Places-LT, we treat samples in this class as a group (cluster); then for each group (1 < is C with total C classes in dataset), calculate average distance between each sample and its corresponding group center, and views this average distance as inner-class distance R; of each group. Furthermore, also define the inter-class distance D as the average distance between any two group centers, and then calculate the ratio between the average of inner-class distance R; and the inter-class distance D, namely, y eD Ej Rg Intuitively, for a group, the smaller inner-class distance R, the more compact of the group. So if is smaller, then the groups are more discriminable. Thus, we use as a metric to measure whether the learnt features are distinguishable, and report the statistic results in Table 2. One can observe that features from VPT fine-tuned pretrained model achieves smaller average inner-class distance and also smaller ratio than those in the vanilla pretrained model, indicating that features of different classes in VPT are easier to be distinguished. Moreover, we also conduct K-NN evaluation between the pretrained VIT-B and VPT fine-tuned pretrained ViT-B. Table 2 shows that VPT surpasses vanilla pretrained ViT-B by 1.1% in terms of K-NN accuracy, indicating the higher discriminative ability of a VPT fine-tuned model. Therefore, one can conclude that 2) the learned prompt can further improve the discriminative ability of pretrained models, thus benefiting to long-tailed classification problems.
4 LONG-TAILES PROMPT TUNING
The observations in Sec. inspire us to design an efficient yet effective long-tailed learning method based on prompt tuning (Jia et al., 2022). Nevertheless, vanilla VPT in long-tailed learning still lags behind the state-of-the-arn methods (Tian et al., 2022; Long et al., 2022). To further improve the overall performance of prompt tuning in long-tailed learning, we propose an effective Long-tailed Prompt Tuning (LPT) method, whose framework and training procedure are illustrated in Fig. 3. Generally, LPT includes shared prompt for all classes to learn general features knowledge and to adapt a pretrained model into the target domain meanwhile empowering the discriminative ability to the training data; and group-specific prompts gather group-specific features and to further fine-
4

---

Published as a conference paper at ICLR 2023
Phase 1: Shared
14 Shared Prompt
n Shared
Shared Prompt
Prompt Tuning
Prompt
Group Promutt
Patch tokmn
[CL SJ token
Block
Closdffer
Codme
Patch
y
Prediction
Phase 2: Group
Match
Proup--Speific Prompts
Prompts Tuning
Shared
Blckk
Bock
[CLS]
Prompt
Key
Prompt
Prompt
(1)
Prompt
ppend Prompt
Potch
Black K+
Choosffee
Cosiaa
Dual
-
Sampling
Prodiction
Training data
Figure 3: Pipeline of Long-tailes tailed Prompt Tuning, where snow means freezed parameters and fire means trainable parameters. For Phase 1, LPT learns shared prompt to capture general knowledge for all classes. For Phase LPT uses fixed shared prompt with ViT generate query, then select best matched prompt from group-specific prompts, finally adopts prompt in the last L-K blocks.
tune classifier used in the first phase for higher performance. Two sets of prompts are optimized by shared prompt tuning and group prompt tuning, respectively. We introduce our LPT as follows.
4.1 PHASE 1: SHARED PROMPT TUNING
For Shared Prompt Tuning phase in Fig. 3, with given pretrained VIT (Dosovitskiy et al., 2021) with L layers, we aim optimize the shared prompt where u follows VPT-Deep (Jia et al., 2022) and consists of L. individual learnable token sequences. Specifically, with given input image LPT obtains the initial patch tokens Zo the pretrained patch embedding layer. Then, with given class token ([CLS)) co and pretrained transformer encoder, for the i-th layer in ViT, where and corresponding key and value keitm vi [G.1, 17 u.), then update (c,, 2,) with u by,
[ui, [u,,... Ut and cosine classifier ( 05),
< L., we define the query used i-th block as qi [c 1.1..
(c,, 2, FFN, (Attn; (q."", KK"h, ))"
where [ ......,] denotes token concatenation operation along the token number direction, Attn; and FFN; are the self-attention layer and feed-forward network in the i-th pretrained ViT block (Vaswani al., 2017). Then, the final class token e, are fed into the cosine classifier calculate per-class confidence scores s = f(CL: fy). Finally, with given ground-truit minimize Lp, = Les (s, during the training of phase I to optimize and 0y. where Lels the classification loss used in both phases and will be discussed in Sec. 4.3.
of corresponding input I, we
4.2 PIIAS 2: GROUP PROMPTS TUNING
A straightforward solution to reduce the difficulty of tailed learning is dividing the training data into multiple groups via the similarity of features, thus sharing group-specific knowledge in each group and reducing the recognition difficulty. Based on this motivation, to gather group-specific features for those samples which have similar features and also to empower the pretrained model with fine-gre greingraindd discriminative ability, we aim to use different group prompts to handle samples from different classes, thus gathering group- specifie features via each group prompt, benefiting to long-tailed classification. Therefore, we introduce group- spou-speccfiic prompts with m individual learn- able prompts R = {(k), r'),.... (km, r""),, where k; is the key of the corresponding i-th group prompt I² and each r' has L - K trainable token sequences. To reduce the computational cost and number of additional parameters, we keep using only shared prompt the first blocks and intro- duce group- speccicc prompt set R into the last L-K blocks. In this subsection, group prompts tuning. Specifically. based on our observation (2) in Sec. 3.2, select the query q C₁ from Phase rather than using output class token from pretrained ViT like Wang et al. (2022), since the class token e₁ often enjoys stronger discriminative ability. Given the query q, we select the best-matched prompts adaptively from R by:
mainly discuss the
training procedure
Wia..., W&] top-k((q. [k,...., km));k)
(2)
5

---

Published as a conference paper at ICLR 2023
where top-k(;,k) returns the indices of prompts w =[ [Wi..... wi] with the largest i cosine simi- larities, and (---) means the cosine similarity operator. Here we discuss the optimization of keys. Intuitively, straightforward method to optimize keys is enforcing queries from the same class to match certain keys. However, this method infeasible since it is difficult to interpret which classes could be matched into some certain prompts exactly. Instead, we prefer to simply minimize the distance between the matched queries and keys, thus optimizing these keys adaptively. We design such query function from this perspective. As observed in Sec. 3.2, the feature cluster of each class generated by the fine-tuned phase 1 is compact. Therefore, for queries from the same class, if we randomly select query qi and key k then minimize 1 - (q kk) ) distance between k and other queries are naturally minimized since these queries are fixed and compact enough. Thus during training, each key are learnt close one multiple nearby clusters, finally guiding correspond- ing group prompt to gather group-specific feature. Moreover, since 1) VPT (Jia et al., 2022) benefits from prompt ensembling, and 2) introducing more group-specific knowledge may benefit to recog- nize samples for tail classes. Instead of using only one matched group prompt from R, LPT also conduct prompt ensembling with multiple selected prompts. which is shown as:
(3) thus resulting an ensembled group prompt r. With given r, LPT reuses the feature (CK, Zk) from Phase 1 as (Ōx, 2x) to save computational cost, then define the query used in i-th block as q.m [C.., 1. and key with value hnth 9.ttm [ 24.1, , U, r.k, finally update (c,, 2:) as:
sum([r")....... rrl))kk
(c, = FFN, (Attn; att katin, ),ttt)),
where K+1 < i S L indicates the index of the last L - K pretrained blocks in VIT. Next, the output class token & are fed into the cosine classifier f and calculate per-class confidence scores by $ = Oy) Finally, with given ground-truth y of corresponding input I, we minimize Lp, including both classification loss Lets as well as the cosine similarity between query q and corresponding matched keys kw, kw which is shown as Eqn. 5:
Σ Eiew (q.k)))
Cp. = (S. y) +
(5)
where B is scale factor of Las and will be discussed in the following.
Note that naively using pling (Kang et al., 2020) may lead to severe overfitting on tail classes or head classes (Zhang al., 2021b) respectively. To balance the performance between head classes and tail classes and avoid overfitting, we introduce dual sampling strategy. Specifically, for each training iteration in Phase 2, LPT randomly samples a mini-batch ) from instance-balance sampler as well as another mini- batch [Thbal from class-balanced sampler. For samples in (T) we simply set B 1 to calculate Lp,; and for samples in {T)m.. we set B = (EE e)/E, where 17 is the initialized weight for (Tiim. iiss denotes the maximum number of epochs, and is the current epoch number.
sampling (Kang et al., 2020) or instance-balanced sam-
4.3 LOSs FUNCTION
Finally, we introduce the classification loss Lats used in our two-phase training. Though LPT can use multiple classification losses to further improve the performance of LPT, we adopt asymmetric GCL loss LA-OCL for both adjusting logits based statistic label frequency from training data and re- re-weighting gradient between positve and negative classes. Without loss of generality, we use $ f 0)) calculated in the Phase of LPT as example to demonstrate GA. Following (Li et al., 2022), we re-scale the confidence score of i-th class by:
Vi (s: (log Amax - log ||e|])
where a is the scaling factor, e is the random variable from gaussian distribution, n, and Mmxa mean the label frequency of i-th class and the maximum label frequency in the training set, respectively. Then, we calculate per-class probability = [p....... Pel by:
P . Pcl = softmax (Vvia.. vcl).
Next, we use asymmetric re- re-weighting (Ridnik et al., 2021) to eliminate the effect from negative gradient in long-tailed learning. Suppose j is ground-truth class of I, we calculate LA-GCL as:
LA-GCL = (1 p,) *+ log(p,) + Σ Eisisc (p.)* log(p;),
(8)
where X and A_ is the focusing parameter (Lin et al., 2017) for ground-truth class and negative classes respectively. Finally, we choose Las L. during two-phase training of L.T..
6

---

Published as a conference paper at ICLR 2023
Comparison with state-of-the-art long-tailed classification methods on Places-LT dataset (Zhou et al., 2017a). Our LPT achieves state-of-the-art performance among vision-only pretrained methods and achieves the same performance with of-te-of-the-ar VL-based methods
Table 3:
Method
Backbone
Params
Tuned
Total Params
Extra Data (Inference)
Overall
Many Medium
Few
Vision-only Pretrained
OLTR (Liu et al., 2019)
Res152
60.34M
60.34M
35.9
44.7
37.0
25.3
TADE (Zhang ct al., 10)
Res152
60.34M
60.34M
38.8
42.8
39.0
31.2
LWS (Kang al., 2020)
Res152
60.34M
60.34M
37.6
40.6
39.1
28.6
MisLAS (Zhong et al., 2021)
Res152
60.34M
60.34M
40.4
39.6
43.3
36.
ALA (Zhao al., 2022)
Res152
60.34M
60.34M
40.1
43.9
40.1
32.9
PaCo (Cui et al., 2021)
Res152
60.34M
60.34M
41.2
36.1
47.9
35.3
VPT (Jia et al., 2022)
WT-B
0.09M
86.66M
37.5
50.4
33.8
23.3
LPT (Ours)
VIT-B
1.01M
87.58M
50.1
49.3
52.3
46.9
Vision-Languge Pretrained with Extra Data
RAC (Long et al., 2022)
VIT-B VIT-B VIT-B
86.57M
236.19M
IN21k Feat
47.2
48.7
48.3
41.8
BALLAD (Ma et al., 2021)
149.62M
149.62M
49.5
49.3
50.2
48.4
VL-LTR (Tian et al., 2022)
149.62M
149.62M
Wiki Text
50.1
54.2
48.5
42.0
Table 4: Comparison with state-of of-tth--rtttttttt art meth- ods CIFAR 100-LT with various imbalanced ra- tio T. LPT performs best among all methods.
Table 5: Comparison with state-of-the offtt-----hhearrt
methods iNaturalist 2018. LPT performs best among vision-only pretrained methods.
Imb Ratio
200
100
10
Method
Overall
Few-shot
Training from Scratch
Vision-only Pretrained
PaCo
$2.0
56.0
64.2
TADE
72.9
Zhu al. (2022) Li et al. (2022)
$1.9
56.6
64.9
PaCo
75.2
74.7
44.9
48.7
53.6
VIT-B/16
73.2
Iseen al. (202l)
75.3
73.2
Vision-only Pretrained
VIT-1/16
75.9
VPT
72.8 87,9
81.0
84.8
89.6
LPT (Ours)
76.1
79.3
LPT (Ours)
89.1
90.0
91.0
VI. Pretrained with Extra Data
VL Pretrained with Extra Data
VL-LTR RAC
76.8
Ma (2021)
77.8
80.2
81.0
EXPERIMENTS
5.1
COMPARISON WITH STATE-OF-THE-AN METHODS
Here we show essential comparison and analysis. More details are shown in Appendix and
Comparison on Places-LT. Generally, these methods can be roughly divided into two groups, i.e., vision-only pretrained methods and vision-language (VL) pretrained methods. Note that VL-based methods (Tian et al., 2022; Long et al., 2022; Ma et al., 2021) may introduce extra data (i.e., Wiki text data or external ImageNet-21k database) during training and testing. Our LPT belongs to the first group and does not rely extra data. Table 3 lists the evaluation results of competing methods. Compared to other vision-only pretrained methods, with only 1.01M (1.1%) additional trainable parameters, LPT achieves 50.1% 1% and 46.9% in terms of overall accuracy and few-shot accuracy, PaCo (Cui et al., 2021) by 8.9% and 11.6%. Even
and respectively surpasses the state-of-the-ee compared with VL-LTR (Tian al., 2022) which VL-based method with extra data in training and testing, our LPT achieves the same overall accuracy while obtaining higher few-shot accuracy.
Comparison on CIFARI00-LT. Then we evaluate LPT on CIFARI00-LY (Krizhevsky, 2009; Cui et al.. 2019) with imbalanced ratio r = 10, 50, 100, 200. Evaluation results are given in Table 4. outperfonm LPT surpasses the CI.P--retrained BALLAD by 11.3% in terms of accuracy with 100. These results indicate the effectiveness of LPT common object-centric data with long-tailed distribution.
Our LPT outperforms all the competing methods on four different imbalanced ratios. Especially,
Comparison on iNaturalist 2018. Finally, we explore LPT on large-scale and fine-graind iNatu- ralist 2018 (Van Horn et al., 2018). Quantitative results are given Table 5. LPT achieves 76. % overall and 79.3% few-shot accuracy and surpasses all other state-of-the-art methods with vision- only pretrained models. Especially, LPT also surpasses fully fine-tuned ViT-L/16 (Touvron et al., 2022) by 0.2%. These results demonstrate that LPT also handle large-scale long-tailed data with only prompt tuning and achieve comparable accuracy. Note that since VL-based methods (Tian al., 2022; Long al., 2022) leverage both CLIP pretrained models and extra testing data, mean-

---

Published as a conference paper at ICLR 2023
Table 6: ImageNet-Sketch evaluation results from different fine-tuning methods.
Table 7: Ablation study of LPT with different pretrained model sizes.
Method
Backbone
Overall
Backbone
Phase Acc
LPT Ace
Linear Probe
VIT-B ViT-B VIT-B
31.55% 32.25%
VIT-T
32.55% 40.50%
37.40%
Full fine-tune
ViT-S
44,66 %
LPT (Ours)
36.22% %
ViT-B
49.41%
50.07 %
Table 8: Ablation study of each phase in LPT on Places-I.T benchmark (Zhou et al., 20)
Method
Prompt
Phase
CAGCL
Phase 2
Overall
Many
Medium
Few
Linear VPT
33. 29% 37.52%
46. 46.48 50.42%
29.45% 33.78%
18. 18.77%
23.29%
41.33%
49.47%
41.31%
27.51%
(6)
49. 10%
19.62%
51.53%
43.25%
49,41 %
46.89%
52.54%
47.32%
50.07 %
49.27%
52.31
46.88%
while the extra data benefits more to object-centric long-tailed scenarios, such that LPT still has small gap with these methods. How to climinate this gap is worth to be further explored.
5.2 ROBUSTNESS WITH DOMAIN SHIFT
To evaluate the robustness of LPT with domain shift LPT on ImageNet-LT (Deng et al., 2009; Liu et al., 2019) train set, and then evaluate the fine-tuned LPT ImageNNet-Sketch (Wang al., 2019) val set which includes multiple sketches from 1 1,000 classes and far different from natural images. Here we select linear probing and fully fine-tuning from the same pretrained ViT-B baseline methods. Evaluation results are shown as Table 6. LPT achieves 36.22% accuracy on ImageNet-Sketee dataset, surpassing linear probing as well as fully fine-tuning by 4.67% and 3.97%. One possible explanation that our LPT gather the domain- specific knowledge during training and adapt the pretrained model to the target domain of interest via prompt tuning, thus with given input images from other domains, LPT can transfer the feature into the domain learned in LPT, then reducing the effect from domain shifting in long-tailed learning.
data, we optimize
5.3 ABLATION STUDY
Effect of Each Phase. First we analyze the effect of each training phase and component in LPT. Table 8 demonstrates the evaluation results of per-phase ablation study. Here we select linear prob- ing (Dosovitskiy et al., 2021) and VPT (Jia et al., 2022) as baseline methods. After training with phase 1, type (a) and type (b) surpass their corresponding baselines by 8.04% and 1 1.588 in terms of overall accuracy. Meanwhile, compared to type (a), after introducing prompt for fine-tuning, type (b) surpasses type (a) by 7.77% and 15.74% terms of overall accuracy and few-shot accuracy. These results indicate that: 1) introducing prompt for fine-tuning benefits to the overall performance and tail classes accuracy in long-tailed learning; and 2) the proposed phase of LPT can fully release the representation ability of learnable prompt, thus leading to better classification results. Further- more, when replacing cross entropy loss in type (b) by CA-GCL. type (c) achieves 49.41 % overall accuracy and obtains 4.07% improvement in terms of few-shot accuracy. Finally, after introducing the group-specific prompts as well as phase 2 in LPTT, type (d) achieves 50.07% overall accuracy on Places-LT, which indicate that using different group prompts handle different input samples can reduce the difficulty of long-tailed learning and further improve the classification performance.
Different Model Size and Pretrained Models. To verify the compatibility of LPT, we eval- uate LPT with ViT-Tiny/SmallBBase (Dosovitskiy et al., 2021), and all ViTs are pretrained with ImageNet-21k (Deng et al., 2009). Quantitative results are shown as Table 7. LPT achieves 37.40% 44.66% and 50.07% accuracy with three pretrained models, which surpasses models with only shared prompt by 4.85% 4.16% and 0.66%, respectively. These results demonstrate the compat- ibility of LPT. Besides, for ViT-T and ViT-S which has less parameters than ViT-B, the combination of shared prompt and group-specific prompts can supply abundant domain-specific knowledge and group-specific features, thus these models benefit more from LPT and obtain larger accuracy im- provement. Furthermore, we also keep using ViT-B and analyze the effect of LPT on various pretrained models (Touvron et al., 2021; Caron et al., 2021; Zhou et al., 2022b; Dosovitskiy et al., 2021). Evaluation results indicate that better supf--uper learning methods and more pretraining data lead to better accuracy, ie.. 50.07% after LPTT. More details are shown in Appendix C.5.
8

---

Published as a conference paper at ICLR 2023
Table 9: Ablation Study of Decoupled or Joint Training, Decoupled Training is better.
Table 10: Ablation study of query function in
phase 2. Using phase as query is better.
Method
Epochs
Overall
Query Fune
K-NN Acc
Phase 2 Ace
Joint 1&2
80 40+40
47 48%
ViT-B Phase
32. 32.11%
49.81% 50.07
Decoupled 1&2
50.07% %
36.16%
0.90
0.90
0.8
0.6
0.54
0.62
0.53
0.4
0.45
0.34
0.39
0.2
0.04 0.06
0.01
0.06
0.02
0.04 0.10
0.01
0.08
Airfield (many)
Canyon (many) Best-matched Prompt
Engine Room (medium)
Racecourse (medium)
Auto Factory (few) Others
Market Indoor (few)
Second mest-hatched Prompt
Figure 4: Statistic results visualization of prompt matching proportion for classes in Places-Ia
Decoupled Training. To verify the effect of decoupled training, we conduct ablation study which jointly optimizes shared prompt and group-specific prompts. For fairness, optimize the joint- trained model with 80 epochs. The results are shown as Table 9, LPT with decoupled training achieves 50.07% overall accuracy and surpasses that with joint training by 2.59%. These results indicate that, during joint training, the shared prompt is still updated simultaneously. thus the query function is sub-optimal during training, resulting in worse matching results. Nevertheless, decoupled training leverages a fixed yet optimal shared prompt query function, thus obtaining better results.
Query Function and Group Size m. We further analyze query function and group size m. Since Wang et al. (2022) used pretrained ViT as query function, to validate whether using our Phase 1 of LPT as query function is better or not, we conduct ablation study about query function. Specifically, we follow the design of LPT to use cosine similarity as distance metric and evaluate the K-NN accuracy of two query functions, then evaluate Phase accuracy with different query functions, which is given in Table 10. Phase 1 achieves 36.16% K-NN accuracy and surpasses the pretrained ViT-B by 4.05%, which indicates that queries from Phase obtain higher quality for matching prompts. Meanwhile, compared LPT with ViT-B query, LPT with phase query achieves 50.07% overall accuracy, which also demonstrates the effectiveness of using phase as query function. Besides, we also vary the size of R m from 5 to 40, and find that LPT is robust to m and achieves the best accuracy 50.07% when m 20. More details are presented in Appendix C.6.
Statistic of Prompt Matching. To verify that keys in group-specific prompts can adaptively learn to match samples from the same class, we count the matching results for samples in each class. And for better visualization, we randomly select two classes from many/medium/few-shee classes respectively, and then demonstrate the proportion of best-matched prompt as well as the second best-matched prompt, which is shown as Fig. We notice that, for each class, samples matched by prompts with top-2 cosine similarity consists of the majority of proportion. This result is consistent with the adaptive prompt matching and prompt ensembling with & = 2 mentioned Sec. 4.2, and demonstrate the effectiveness of group-specific prompts.
CONCLUSION
We proposed Long-tailed Prompt Tuning (LPT) to tackle long-tailed learning, which consists of 1) shared prompt for all classes to learn general features or knowledge and to adapt a pretrained model into the target domain; and 2) groui-specific prompts gather group-specific features for those samples which have similar features and also to empower the pretrained model with fine- grained discriminative ability. For effective training, we propose a two-phase training framework, in which the first phase optimizes shared prompt to adapt the pretrained model to the target domain of interest, and the second phase optimizes group-specific prompts with dual-sampling strategy and asymmetric GCL loss to dig the common features of these similar samples. Experimental results demonstrate the effectiveness and efficiency of our LPT with ~1.1% extra trainable parameters, meanwhile illustrating its robustness against domain shift.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported by the National Key RD Program of China under Grant No. 121000
REFERENCES
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6897-6907, June 2022.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models, 2022.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlles dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL http:: http:::://c/ceeeeeee ngs neurips cc/ pdf.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020..
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aubel-distrbuution-aware margin loss. In Advances in Neural Information Processing Systems, 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Hao Chen, Ran Tao, Han Zhang, Yidong Wang, Wei Ye, Jindong Wang, Guosheng Hu, and Marios Savvides. Conv-adapter: Exploring parameter efficient transfer learning for convnets, 2022.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference computer vision, pp. 715-724, 2021.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. 2019 IEEF/CVE Conference on Computer Vision and Patlern Recognition (CVPR), Jun 2019, doi: 10.1 (0.11099cv::221990094499 URL http:/ /xxx/xx/// http::/dddd.i.o//g///.. 10. 1111//11PPPR.. 2019 .9.199940 00949.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training batchnorm and only batchnorm: On the expressive power of random features {cnn]s. In International Conference on Learning Representations, 2021. URL https: htwp//peeeeeiieeeeeeeeeeeeeeeeee net forum? id=vve@Q2 9Tbvx.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrice and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022. URL https: ///p/oppeeeiiieeeeeeeeeeeee net forum? in htt.://rerrr.reeoooooooooooddo.. ORDcd5Axo..
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu. Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
10

---

Published as a conference paper at ICLR 2023
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Charles Herrmann, Kyle Sargent, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu, Dilip Krishnan, and Deqing Sun. Pyramid adversarial training improves vit performance. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 33419-13429, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- lernational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799, PMLR, 09-15 Jun 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https::/ /ttp:///e/eeeeeeee///eee/ee/e.ee net / forum? id-nZevKeeFYfs,
Ahmet Iscen, André Araujo. Boqing Gong, and Cordelia Schmid. Class-balanced distillation for long-tailed visual recognition. arXiv preprint urXiv:2104.05277 2021.
Muhammad Abdullah Jamal, Matthew Brown, Ming- Huug- Yang. Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from domain adaptation perspective. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition (CVPR), 2020.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), 2022.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailes recognition. In International Conference Learning Representations, 2020. URL /tttttt::///pereeve/e http///inneeeeeeeeeeeee/e. net forum? id=r1gRIcVEvB.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059, 2021.
Mengke Li, Yiu-ming Cheung, and Yang Lu. Long-tailed visual recognition via gaussian clouded logit adjustment. In CVPR, pp. 6929-6938, 2022.
Shuang Li, Kaixiong Gong, Chi Harold Liu, Yulin Wang. Feng Qiao, and Xinjing Cheng. Metasaug: Meta semantic augmentation for long-tailed visual recognition. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5212-5221, 2021a.
Tianhao Li, Limin Wang, and Gangshan Wu. Self supervision to distillation for long-tailed visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 630-639, 2021b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision. pp. 2980-2988, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X. Yu. Large- scale long-tailed recognition an open world. IEEE Conference on Computer Vision and Patlern Recognition (CVPR), 2019.

---

Published as a conference paper at ICLR 2023
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Chun- hua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. In CVPR 2022, 2022.
Teli Ma, Shijie Geng, Mengmeng Wang. Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. simple long- tailed recognition baseline via vision-language model, 2021.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021. URL https::/ opw.reveee.e..eeoooeee net/ forum?
Xing Nie, Bolin Ni, Jianlong Chang, Gaomeng Meng. Chunlei Huo, Zhaoxiang Zhang, Shiming Xiang, Qi Tian, and Chunhong Pan. Pro-tuning: Unified prompt tuning for vision tasks, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Jiawei Ren, Cunjun Yu, shuman sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li. Balanced meta-softmax for long-taiie tailed visual recognition. H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.
4175-4186. Curran Associates, Inc., 2020.
Tal Ridnik, Emanuel Ben- -arrhh, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 82-91, 2021.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1662--11771, 2020.
Changyao Tian, Wenhai Wang, Xizhou Zhu, Xiaogang Wang. Jifeng Dai, and Yu Qiao. Vi-Itr: Learning class-wise visual-linguistie representation for long-tailed visual recognition. ECCV, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers and distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357, July 2021.
Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers, 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song,
Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8779-8778, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998- 6008, 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518, 2019.
Xudong Wang. Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distributton-awaree experts. arXiv preprint 1r 809, 2020.
12

---

Published as a conference paper at ICLR 2023
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi. Ali Farhadi, Hongseok Namkoong, al. Robust fine-tuning of zero-shot models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959-7971, 2022.
Yue Xu, Yong-Lu Li, Jiefeng Li, and Cewu Lu. Constructing balance from imbalance for long-tailed image recognition, 2022.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2019.
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv preprint arXiv:2107.09249, 2021a.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04590 2021b.
Yan Zhao, Weicong Chen, Xu Tan, Kai Huang, and Jihong Zhu. Adaptive logit adjustment loss for long-tailed visual recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3472 3480, 2022.
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16489-16498, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017a
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017b.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Eilateral-brande network with cumulative learning for long-tailed visual recognition. 2020 IEEE/CVE Conference on Computer Vision and Patern Recognition (CVPR), Jun 2020. doi: 10.1109/cyp426002220.0009999 URI. 2020. 00974.
http::/ httt:://xx..oo.ooee org/ 10. httrg//...109///P/.. 11111110/////11110
Kaiyang Zhou, Jingkang Yang, Chen Change Loy. and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, pp. 1-12. 2022a.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi-granular self-supervised learning framework, 2022b.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long- tailed visual recognition. In Proceedings of the IEFE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 6908-6917, June 2022.
13

---

Published as a conference paper at ICLR 2023
The content of the Appendix is summarized as follows: 1) in Sec. A, we demonstrate the details of datasets we use in experiments of LPT: 2) in Sec. B. we demonstrate the implementation and training detail of LPT; in Sec. C, we illustrate more detailed ablation studies of LPT, including effect of shared and group prompts in C.I, pretrained models C.5, group sizes C.6, effect of K C.7, prompt ensembling C.8, dual sampling strategy C.9 and asymmetric GCL loss C.10. Meanwhile, we also illustrate more comparison results, including comparison with state-of-the-ar methods with same backbone in C.2, comparison with different efficient tuning methods in C.3, comparison with multi-task learning based method in C.1 I and more detailed robustness evaluation in C.4.
A DATASET DETAILS
CIFARIO0-LT is subset of the original CIFAR- 100 (Krizhevsky. 2009). which includes 100 different categories and 10.000 test images. With given imbalanced ratio max max(n;)/min(nn) where 1 S i < 100 is the class index of CIFAR-100, we follow the exponential down-sampling setting from (Cao et al., 2019; Cui et al., 2019) to generate corresponding long-tailed training data, and utilize CIFAR 100-LT with = 10, 50, 100, 200 to evaluate our L.T..
Places-LT (Liu et al., 2019) is the subset of Places-366 365 dataset Zhou et al. (2017a), which includes 62500 training images from 365 individual scene categories with per-class samples from 5 to 4980. Following previous works (Liu et al., 2019; Kang et al., 2020), we optimize our LPT on train set and then select the best model val set, finally report the test accuracy on test set.
iNaturalist 2018 (Van Horn et al., 2018) is a large-scale fine mme-grained classification dataset, which exists extremely high imbalanced distribution with 7 = 996. This dataset includes 4437K training images as well as 24.4K validation images. Following previous works (Kang et al., 2020; Cui et al., 2019), we use train set to optimize our L.PT and evaluate I.PT on val set.
ImageNet-Sketch (Wang et al., 2022) includes 50,000 individual sketch images from the same 1,000 ImageNet categories. Since the backbone of our LPT is pretrained on ImageNet (Deng et al., 2009), different from previous methods, we only use ImageNet-LT (Liu et al., 2019) train set to fine-tune the prompt and classifier. Instead, we evaluate LPT on ImageNet-Sketch val set to verify the robustness of our LPT with domain shift scenarios.
A.1 EVALUATION PROTOCOL
Following Cui et al. (2021); Tian et al. (2022); Long et al. (2022), for both Places-LT (Zhou et al., 2017a) and iNaturalist 2018 Van Horn et al. (2018), we first divide the whole dataset by the number of training images in each class, i.e., many-shot (> 100 images), medium-shot (20~100 images) and few-shot (=20 images); then we report the overall accuracy as well as many/medium/feww accuracy, respectively. Meanwhile, for CIFAR 100-LT dataset, following Cui et al. (2021); Li et al. (2022), we directly report the overall accuracy with = 10, 50, 100, 200. And for ImageNet-Sketch dataset, we also directly report the overall accuracy for simplicity.
B IMPLEMENTATION DETAILS
Following Jia al. (2022); Wang al. (2022), use VIT-B/16 (Dosovitskiy al., 2021) with ImageNet-21k pretrained model as the backbone of LPT. For shared prompt, we simply set the default length of prompt as 10 and adopt prompts on all transformer blocks in the ViT. And for group-specific prompts, we set shared layer number K = 6 and the size of prompt size m = 20; for each prompt the set, the prompt length is also set as 10. Note that setting K may lead to 1.5x inference cost (i.e., use VIT with shared prompt to generate output class token as query, then reuse features from the 6-th block and inference with both shared prompt and group prompt for the last 6 blocks) compared to VPT (Jia et al., 2022), but can achieve better accuracy, and is more efficient than inference twice (i.e., inference once with pretrained ViT-B to generate the output class token as query, and then inference the second time with prompt to calculate the final confidence scores), e.g., Wang et al. (2022). During training and testing, we set the prompt ensemble number k = 2. Finally, the number of additional parameters (linear classifiers omitted for simplicity) is 1.01 Both
14

---

Published as a conference paper at ICLR 2023
Table 13: Quantitative analysis of features learned by Phase 1 of LPT and Phase 2 of LPT. Features from phase 2 LPT obtains the similar 7 value while obtaining larger inter-class distance.
Method
LPT Phase
LPT Phase 2
INLK (ViT)
INLK (Phase 1)
Inter-class distance
15.8
16.1
Places (VIT)
Places (Phase
Inner-class/ inter-class
0.237
0.242
Places (Phase 2)
Figure 5: LDA visualization of LPT.
prompts are initialized with truncated normal distribution. During training in both phase one and phase two, we use SGD optimizer with momentum of 0.9 to optimize the shared prompt and group- specific prompts, respectively. During training, the initial learning rate is set to be 0.002 256 where B indicates batch size. Different from Alshammari et al. (2022), we set the weight decay le-4 since large weight decay does not lead significant performance improvement in LPT. During training, we use cosine learning rate scheduler to control the learning rate with 5 warmup epochs. For Places-LT, CIFAR 100-LT and ImageNet-LT, we optimize phase and phase of LPT for E = 40 epochs, respectively; and for iNaturalist 2018, since this dataset includes much more training data, set epoch number E 80 for phase and phase respectively. For asymmetric Goss we set X4 and A_ as O and 4, respectively. And for phase 2, we set the initialized weight Y used in im) as 0.5. all experiments, the training and testing images are resized to x24 224. During both two training phases of LPT, we only introduce random crop and resize operation and Mixup technique as data augmentation; and during evaluation, we All programs are implemented by PyTorch toolkit (Paszke et al., 2019), and all experiments are conducted single RTX A6000 GPU.
Table 11: Ablation Study of different pretrained models. Both more pretraining data and better pretraining algorithms benefit to LPT.
Table 12: Ablation Study of the size of group- specific prompts. Prompt set with 20 individual group prompts performs better.
Method
Data
Phase
LPT
Group Size
Parameters
Phase
DINO Mugs
40.65%
42.18%
5
0.23M
49.81%
INIK
41.92%
43.66%
10
0.46M
49 19.90%
Supervised
INIK
41.33%
42.71%
0.92M
50.07 %
Supervised
IN21K
49.41
50.07%
40
1.84M
49.87%
C MORE ABLATION STUDY AND DISCUSSION
C.1 THE EFFECT SHARED PROMPT AND GROUP PROMPTS
To further investigating the effect of shared prompt and group prompts, in addition to the ablation study in Sec. 5.3, we also conduct more quantitative and qualitative analysis on Places-LT dataset. As discussed Sec. shared prompt aims to learn the domain-specific knowledge from training data while group prompts aim to gather group-specific features from training data in corresponding groups. To verify this point, we also adopt LDA among features extracted from LPT (Phase 1) and LPT (Phase 2), as well as features from pretrained VIT without any prompts. The visualization results are shown in Fig. 5. According to the visualization, we find that the distribution of feature from LPT (Phase 1) and LPT (Phase 2) are highly overlapped, this observation proves that shared prompts in phase 1 obtains the domain-speciffe knowledge. Moreover, we also investigate the y value and inter-class distance of features from LPT (Phase 1) and LPT (Phase 2), which are shown in Table 13. We find that, both of them have the similar 7, while LPT (Phase 2) has larger inter-class distance, which indicates that group prompts introduce more class discriminative ability.
15

---

Published as a conference paper at ICLR 2023
Table 14: Fair comparison with state-of-the-art methods on Places-LT dataset. All methods start from the same IN21K pretrained ViT-B feature extractor to conduct fully fine-tuning or prompt tuning. Quantitative results show that LPT still largely surpasses previous methods by 7.25%.
Method
Backbone
Pretrained Data
Overall Acc
Kang et (Kang et al., 2020)
ViT-B
IN21K
40.45%
PaCo (Cui al., 2021)
VIT-B
IN21K
37.00%
GCT (I.i et al., 2022)
ViT-B
IN21K
42.82%
VIT-B
IN21K
50.07%
Table 15: Fair comparison with different efficient tuning methods on Places-LT dataset. All methods start from the same IN21K pretrained ViT-B feature extractor. Quantitative results show that LPT achieves the best accuracy.
Method
Backbone
Pretrained Data
Overall Acc
VPT (Jia al., 2022)
ViT-B
IN21K
44.2%
Visual Prompting (Bahng et al., 2022)
ViT-B
IN21K
13.8%
Pro- Pro-Tuning (Nic et al., 2022)
VIT-B
IN21K
48.3%
LPT
VIT-B
IN2IK
50.1%
UPT+FT-Nomm (Frankle et al., 2021)
VIT-B
IN21K
49.9%
Adaptrr (Nic et al., 2022)
VIT-B
IN21K
49.9%
C.2 FAIR COMPARISON WITH STATE-OF -EE------TEEEEEEE METHODS
Since previous methods mainly leverage INIk pretrained ResNet to conduct experiments on long- tailed datasets, fairly compare LPT with previous methods via using the same backbone, we reimplement three state-of-the-an 2022) with the same IN21K pretrained VIT-B as feature extractor to conduct fully fine-tuning on Places-LT. Experimental results are shown in Table 14. The state-of-the-e GCL (Li et al., 2022) with IN21K K pretrained VIT-B backbone achieves 42.82% overall accuracy on Places-LT, while still lower than LPT by 7.25%. These results also indicate that LPT surpasses previous state-of-the-art methods under the fair comparison settings.
tailed methods (Kang et al., 2020: Cui et al., 2021: Li et al.,
C.3 COMPARISON WITH DIFFERENT EFFICIENT TUNING METHODS
Without loss of generality, conduct the comparison between LPT and more efficient tuming meth- ods on Places-LD LT dataset, including different prompt tuning methods (Visual Prompting (Bahng et al., 2022) and VPT (Jia et al., 2022)) and adapter tuning methods (c.g., Pro-Tuning (Nic et al., 2022)). Furthermore, we also additionally adopt two efficient tuning methods, i.e., nor- malization tuning and adapter tuning into our LPT (i.e., LPT+FT-Norm (Frankle et al., 2021) and LPT+Adapter (Nie et al., 2022)). Note that for previous efficient tuning methods, we adopt balanced sampling and use grid search find the best training parameters to optimize the models. Without loss of generality, we keep using the same IN21K pretrained ViT as backbone for all methods and evaluate all methods on Places-LT dataset. The corresponding evaluation results are shown in Ta- ble 15. Compared to other efficient tuning methods, our L.T surpasses the state-of-the-ar VPT and Pro-Tuning by 1.8% and 5.9%, respectively. These results demonstrate the effectiveness of LPT in handling long-tailed classification. And note that Bahng et al. (2022) only inserts learnable prompt on the edges input images, and fixes all the other pretrained parameters (including the linear clas- sifier), which may explain its relatively lower performance. And meanwhile, after adopting other efficient tuning methods into LPT, the final performance has no significant improvement. Thus we choose the current LPT structure the final framework.
C.4 MORE DETAILED EVALUATION FOR ROBUSTNESS WITH DOMAIN SHIFT
To further compare our LPT with the baseline methods (linear probe, fully fine-tune. VPT (Jia al., 2022), WISE-FT (Wortsman et al., 2022)), evaluate all these models six different OOD
16

---

Published as a conference paper at ICLR 2023
Table 16: Full comparison with different fine-tuning methods on six different OOD dataset. All methods start from the same IN21K pretrained ViT-B feature extractor. Quantitative results show that LPT achieves the best accuracy.
Method
ImageNet-Sketch
Image
ImageNet-V2
ImageNet-A
ImageNet-R
ObjectNet
Linear Prohe
31.55%
81.43%
63.54%
29.20%
45.72%
6.61%
Fully Fine-tune
32.25%
80.10%
62.31% 1%
30.12%
43.14%
7.13%
VPT
34.64%
85.82%
68.51% %
35.17%
47.06%
8.03%
WISE-FT
34.79%
82.20%
65.76%
36.75%
47.32%
8.00%
LPT
36.22%
87.22%
70.71%
39.65%
50.47%
8.22%
Table 17: Ablation Study of the number blocks with only shared blocks, i.e. K. Quantitative results show that K 6 performs better for phase 2 of LPT.
K
Param Num Overall Acc
0.61M 49.77%
0.92M 50.07% %
1,23M 49.92%
datasets (ImageNet-Sketch (Wang al., 2019), ImageNet-Real. (Beyer al., 2020), ImageNet- V2 (Recht et al., 2019), ImageNet-A (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a), ObjectNet (Barbu et al., 2019)). For fairness, all these methods start from the same IN21K pretrained ViT-B and fine-tune on the same ImageNet-LT training set. The evaluation results are shown in Table 16. Our LPT surpasses all baselines on six OOD datasets. Note that Herrmann et al. (2022) mentioned that the pretrained VIT (Dosovitskiy et al., 2021) we used in this experiment performs bad on ObjectNet benchmark (i.e., 17.36% aceuracy after ImageNet- 1k training), all cor- responding results ObjectNet in the table are relatively low because the training is performed on the subset of ImageNet- 1k (i.c., ImageNet-LT).
C.5 PRETRAINED MODELS
We also evaluate our LPT on various pretrained models from different pretraining algorithms and different pretraining data scale. Specifically, we keep using ViT-B/16 structure and select DeiT (Tou- vron et al., 2021), DINO (Caron et al., 2021) and Mugs (Zhou et al., 2022b) pretrained on ImageNet- Ik (Deng al., 2009), meanwhile using VYT-B (Dosovitskiy et al., 2021) pretrained on ImageNet- 21k as baseline model. Evaluation results are shown as Table 11. With the same ImageNet- Ik pretraining data, LPT with Mugs achieves 41.92% accuracy after phase and 43.66% accuracy af- ter phase 2, which surpasses other two pretrained models. Meanwhile, VPT with ImageNet-211 pretrained model achieves 50.07% accuracy and largely surpasses LPT with DeiT-B, These results indicate that pretrained models from better algorithms or larger pretraining data lead to better effi- cient tuning results with long-tailed target data.
C.6 SIZE OF GROUP-SPECIRG PROMPTS
We also conduct ablation study about the size of group-specific prompts. Generally, we start each phase training from the same phase I model, and only change the group size of the group-specific prompts. The corresponding evaluation results are shown as Table 12. When the size of group- specific prompts increasing 20, the accuracy LPT increases from 49,8 % 50.07%. However, when we further increase the size to 40, the final accuracy declines to 49.87%. A possible reason is that, some classes in the dataset may share some similar group-specific feature or knowledge, such that features from instances in corresponding classes may be similar. Thus instances from these classes can be seen as cluster and can be matched into the same prompt. Since the number of different attributes is limited, only need fixed number (e.g., 20) of group prompts to handle the whole dataset and achieve better accuracy. Besides, using too many group prompts may increase the difficulty of clustering and optimization, which affects the final accuracy.
17

---

Published as a conference paper at ICLR 2023
Ablation Study of ensemble prompt number k. Quantitative results show that k 2 and 3 achieve better results, thus we select & = 2 in LPT.
Table 18:
Ensemble Num
3
4
Overall Acc Few-shot Acc
49.93% 46.32%
50.07% 46.88 %
50.00% 46.87%
49.93%
46.84%
Table 19: Ablation Study of dual sampling in phase 2. Adding dual sampling strategy with proper small y (e.g.. 0.5).
Method
Dual
Y
Overall
Many-shot
(a) balanced sampling dual w/ small"? y (c) dual w/ large
47.67% 49.27% 49.62%
49. 90.90%%
0.5
50.07% %
1.0
50.06% %
C.7 EFFECT OF K
Next further analyze the effect of K, which stands for the number of blocks with only the shared prompt u. Intuitively, the less K means more parameters are inserted into more transformer blocks conduct group prompts tuning and should lead better performance. To verify this hypothesis, we set K 4/6/8 and conduct corresponding ablation study. Evaluation results are shown in achieves the best performance and surpasses LPT with K 8 by 0.3% in
Table 17. LPT with terms of overall accuracy. Meanwhile, LPT with K 4 achieves similar performance with K = 6 counterpart, but does not lead to significant performance improvement with more parameters. The possible reason is that: 1) too large K could restrict the group-specific knowledge gathering ability of group-specific prompts, since only a few layers are utilized to extract group-specific features from maigttilldd data; and 2) compared to K = 4, LPT with K 6 fully leverages the adapted feature representation from phase 1, thus reducing the difficulty of optimization and achieving better accuracy. Therefore, choose K for final LPT during experiments.
C.8 EFFECT OF ENSEMBLE NUMBER
We also analyze the effect of ensemble token number l in phase 2. Intuitively, introducing can lead better accuracy tail classes since more class-specific knowledge are utilized for recognition. Therefore we set A: = /2 and conduct corresponding ablation study. Evaluation results are shown as Table 18. Based on the results, we find that: 1) the overall accuracy are robust with different & values, and 2) introducing prompt ensembling benefits to tail classes (+0.56% in terms of few-shot accuracy), meanwhile using top-2 best matched prompts for ensembling achieves the best results. Therefore, choose & = during training and testing.
C.9 EFFECT OF DUAL SAMPLING IN PHASE 2
To evaluate the effect of dual sampling in phase 2, we conduct a series of experiments, which is shown as Table 19. Compared type (a). type (b) achieves ~0.2% accuracy improvement in terms of overall accuracy, meanwhile surpasses 1.6% improvement in terms of many-shot accuracy. These results indicate that: introducing dual sampling strategy with proper y can lead to better overall performance and reduce the overfitting from balanced sampling only, but dual sampling with too large weight may lead to negative effect on overall accuracy.
Table 20: Ablation Study of asymmetric GCL Loss. Introducing gradient re- weightittng into GCL Loss can further improve overall accuracy in long-tailed classification.
Loss Function
Overall
Many
Medium
Few
GCL Li al. (2022)
49.58% 50.07%
48.19% 49.27%
$2.62% 52.31%
45.75 75% 46.88 %
LA.GE.
18

---

Published as a conference paper at ICLR 2023
Table 21: Fair comparison with multi-task learning methods Places-LT dataset. All methods start from the same IN21K pretrained ViT-B feature extractor to conduct fully fine-tuning or prompt tuning. Quantitative results show that LPT still surpasses multi-task learning method by a large margin.
Method
Backbone
Pretrained Data
Places-LT Ace
CIFARI00-LT (IF=200) Acc
Multi-task
VIT-B
IN21K IN2IK IN2IK
40.45%
LPT (Places-LTT
VIT-B
50.1%
LPT (CIFARIOO-LT, IF=200)
VIT-B
87.9%
C.10 EFFECT OF ASYMMETRIC GCL Loss
To evaluate the effeet of adding asymmetric gradient re- re-weighting design into GCL loss, we conduct ablation study between LA and standard GCL loss Li et al. (2022). Without loss of generality, we conduct both experiments on phase The quantitative results are shown as Table 20, LPT with LA-GCL surpasses the counterpart with GCL loss by 0.49% and 1.03% in terms of overall accuracy and few-shot accuracy. These results further demonstrate the effect of LA-GCL.
C.11 COMPARISON WITH MULTI-TASK LEARNING METHOD
Without loss of generality, we compare our LPT with a multi-task training based method from IN21K pretrained VIT-B to optimize both Places-LT and CIFARI00-LE with imbalanced factor of 200 (IF=200). Generally, in the multi-task training method, for each task (corresponding to a specific dataset), initialize linear classifier, and then optimizing all linear classifiers and the pretrained backbone by end-to-end fine- tuning. The corresponding results are shown in Table 21. LPT surpasses multi-task training method by 10.6% Places-I.T and 14.8% CIFAR 100-LT (IF=200). Benefiting from the two merits mentioned in the introduction, LPT can achieve high performance meanwhile easy to deployment with different scenarios.
C.12 BROADER IMPACT
LPT is based on previous large-scale pretrained models, and fine-tunes only as few extra trainable parameters to adapt real-world long-tailed scenarios. Compared to previou methods, LPT is more efficient for saving training cost and storing additional parameters, which is economic for real-world application. However, LPT still fully data-driven, and should be cautious with potential negative impact from biased data.
C.13 LIMITATION
The performance of head classes from LPT is still lower than that from the VL-based state-of- the-art method (Tian et al., 2022). A possible solution proposing novel head-tail separation algorithm (Xu et al., 2022) to further reduce the difficulty of prompt tuning with divide-and-congee strategy, thus improving the accuracy for both head and tail classes. This part leaves for further exploration in the future.
D MORE STATISTIC OF PROMPT MATCHING
To verify that keys in group-specific prompts can adaptively learn to match samples from the same class, we count the matching results for samples each class. And for better visualization, we provide more results from many/mediun/ffw-shoe classes, and then demonstrate the proportion of best-matched prompt as well as the second best-matched prompt, which is shown as Fig. 6. We notice that, for each class, samples matched by prompts with top-2 cosine similarity consists of the majority of proportion. This result is consistent with the adaptive prompt matching and prompt ensembling with & = 2 mentioned Sec. 4.2, and demonstrate the effectiveness of group-specific prompts.
19

---

Published as a conference paper at ICLR 2023
0.90
0.90
0.86
0.62
0.6
0.54
0.53
0.45
0.34
0.39
0.2
0.04 0.06
0.01
0.08
0.02
0.04
0.10
0.04
0.08
Airfield (many)
Canyon (many) Best-matched Prompt
Engine Room (medium)
Racecourse (medium) Second Best-matcher Prompt
Auto Factory (few) Others
Market Indoor (few)
0.85
0.8
0.76
0.72
0.64
0.60
0.6
0.53
0.4
0.39
0,29
0.36
0.24
0.2
0.08
0.07
0.09
0.15
0.04
0.05
0.04
Market Outdoor (many)
Skyscraper (many) Best-matched Prompt
Picnic Area (medium)
heat Field (medium)
Volleyball Court (few) Others
Utility Room (few)
Second Best-matched Prompt
0.90
0.92
0.8
0.69
0.65
0.68
0.67
0.6
0.4
0.22
0.30
0.20
0.24
0.2
0.05 0.05
0.09
0.05
0.12
0.09
0.03
Cean Deep (many)
Tundra (many) Best-matched Prompt
Tree Farm (medium)
Television Studio (medlum) Second Best-matched Prompt
Tree House (few) Others
Toyshop (few)
0.8
0.75
0.72
0.55
0.55
0.59
0.41
0.2
0.4
0.04
0.27
0,18
0.25
0.05
0.17
0:11
0.23 0.18
Snow field (many)
Shed (many) Best-matched Prompt
Rock Arch (medium)
Rice Paddy (mediuu)) Second Best-matched Prompt
Server Room (few) Others
Sandbox (few)
0.84
0.82
0.8
0.74
0.65
0.60
0.62
0.4
0.32
0.15
0.15
0.09
0.06 0.12 0.12
0,23
0.15
Restaurant (many)
Rainforest
Raceway (medium)
Pub Indoor (medium)
Physics Lab (few) 0 thers
Pharmacy (few)
(many) Best-matched Prompt
Second Best-malched Prompt
0.90
0.83
0.8
0.63
0.51
0.46
0.4
0.29
0.20
0.26
0.34
0.20
0.060.04
0.07 0.10
0.100.09
0.11
off Course (many)
Harbor (many) Best-matrhed Prompt
Music Studio (medium)
Moutain Path (medium) Second Best-matched Prompt
Moat W Water (few) Others
Galley (few)
Figure 6: More statistic results visualization of prompt matching proportion for classes in Places-LT.
20


Reviews:
[{"review_id": "jb5bv29sN6", "reviewer": "Reviewer_qoSe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an effective prompt-tuning method for long-tailed recognition. The proposed method consists of two phases. They first learn the shared prompt and then consider group prompts. They conduct experiments on Places-LT, Cifar100-LT and iNaturalist 2018. The results are good and comprehensive, which validates the efficacy of the proposed method. \n", "summary_of_the_review": "This paper proposes a new prompt-tuning based method for tackling with long-tailed datasets. The results and the analysis are good. However, there are some points which need further clarification. ", "strength_and_weaknesses": "Strength:\n\n1. It's a good try to explore prompt tuning for long-tailed recognition. \n2. The experiments and related analyses are comprehensive and clear.\n3. The paper is well-written and easy to follow. \n\nWeaknesses: \n\n1. The biggest concern is the leverage of pretrained models, which are based on much larger datasets. This will make it unfair to compare with previous baselines. This point should be discussed. \n\n2. Since most of previous methods are based on resnet. The difference between different backbones should also be discussed. The difference should be shown clearly in the table.\n\nThus, in all tables in the experiments, it is suggested to provide the corresponding backbone and valid parameters of the proposed method and baselines for a fair comparison. \n\n3. Why the many-shot accuracy of VPT is substantially below than linear probing under balanced sampling in table 1?\n4. The adoption of pretrained model leads to the case that the method is naturally more advantageous than previous methods.  Thus, a more comprehensive comparison between different prompt learning methods is needed in this setting. These methods can be those discussed in the related works. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X9JumSQd9D", "reviewer": "Reviewer_q9x4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces visual prompt tuning (VPT) methods in long-tailed recognition and proposes a modification to VPT to adapt to long-tailed scenarios with so-called long-tailed prompt tuning (LPT). The method is simple and effective. The results of compared benchmarks show improvements from the proposed LPT method. ", "summary_of_the_review": "Overall, I think this paper is good.", "strength_and_weaknesses": "The proposed method is simple yet effective and achieves general improvements over compared baselines in the reported experiments. I think it is a good contribution to the community to introduce prompt tuning to long-tailed recognition because it sparks thinking on how to effectively utilize pre-learned knowledge (pre-trained weights in this case). However, prompt tuning has a prerequisite, a \"pre-trained weights\" or a foundation model, which is not always approachable. And the existence of pre-trained weights shifts the goal of long-tailed methods from \"balanced learning from imbalanced data\" to \"effectively transfer pre-trained knowledge to imbalanced data\". This shift makes evaluating the model's pure recognition and generalization ability harder, and generally changes the setting of long-tail recognition. Although this might not be a bad thing, I think this shift is worth some discussion. Moreover, since the basis of the method is to solve challenges like \"fine-tuning whole model impairs the generalization ability of pre-trained models\", the discussion on why we need to preserve the generalization ability of pre-trained models is necessary.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QyWhXlvMDO", "reviewer": "Reviewer_ferv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission focuses on the long-tailed classification tasks, and proposes the LPT method to alleviate the negative problems of long-tailed learning. The training of LPT consists of two stages. The first stage trains shared prompts to learn general features of the dataset, and the second stage trains group-specific prompts to learn specific features for samples with similar features. The contribution of this paper is to apply the idea of VPT to the long-tailed classification problem and achieves effective improvement.", "summary_of_the_review": "The authors are inspired by VPT, like using prompts to fine tune the network, and apply this kind of problem-solving idea to the study of long-tailed data. Although this idea exists in previous research, it is undeniable that this method has achieved some effective improvement in experimental performance, so I think it is a meaningful work.", "strength_and_weaknesses": "Strengths:  \n1.\tCompared with previous work, the LPT method can effectively handle various long-tailed learning scenarios by fine-tuning fewer parameters, and the model has strong generalization ability and compatibility.\n2.\tThe experimental settings are reasonable, and the authors have done relatively complete ablation experiments.\n\nWeaknesses: \n1.\tCompared with the VPT method, the LPT method only adds a small number of fine-tuning parameters, but the manuscript does not mention whether the 2-stage method will increase the training time.\n2.\tThe LPT method is robust to the domain shift problem. This paper proposes a possible explanation, but does not specify that domain-specific knowledge is obtained by what kind of prompt and at which stage.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JpxJtWFtCg", "reviewer": "Reviewer_mA6r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method, Long-Tailed Prompt Tuning (LPT), for light-weight finetuning of ViT's in the long-tailed image recognition regime. They identify three key issues with current methods for finetuning for long-tailed recognition:\n\n1. Full finetuning is expensive\n2. Full finetuning reduces generalization performance\n3. Model features are not compatible between different finetuned models, (increased deployment cost?)\n\nLPT attempts to fix these issues by adding a stage to the VPT training procedure, which introduces group (class) specific prompts at intermediate layers of the vision transformer. They claim that this approach makes their model have better:\n\n1. Generalization (evaluated wrt ImageNet-Sketch)\n2. Better performance in the long-tailed regime\n3. Light-weight finetuning", "summary_of_the_review": "**Update post-rebuttal:** With the promised changes to the draft, I update my review to an _accept_ (score of 8). Thank you to the authors for their thoughtful reply and overall effort.\n\n\nMy overall rating is _weak reject_. \n\nThe results in the long-tailed setting looks strong and the method is very interesting, however I don't think they are fleshed out enough yet for acceptance by themselves. In particular, I would like to see experiments in other long-tailed settings (e.g. the iWildCams dataset), and also for all the methods in Table 3 to be replicated in the iNaturalist setting. The multi-task claim does not appear to be fleshed out. Need results on VTAB/DomainNet for that claim to be in the paper. The generalization results could also be extended to more distribution shifts, which could provide a better understanding of the methods/improvements. You also seem to be comparing with other methods on # of tuned params. Either this is a central claim for your paper, in which case you need comparison to light-weight finetuning methods (besides VPT), or you should clarify that this is a side-benefit of your method.\n\nMy general suggestion would be to focus on the long-tailed regime and present the other experiments as side-benefits.", "strength_and_weaknesses": "##### Strengths\n\n- Very strong results in the long-tailed regime. In particular, the Places-LT were convincing. The problem of representation learning in long-tailed scenarios is relevant quite broadly, as most natural data sources exhibit long-tailed behavior.\n- Method seems intuitive (e.g. better clustering for class labels). Unclear why this leads to better generalization (see weaknesses), but the results speak for themselves.\n- The method is straightforward to understand but shows creativity, in particular the strategy for integrating the group level prompt-embeddings was interesting.\n\n\n##### Weaknesses\n\n- Would appreciate more generalization results. In particular I would like to see results on all 5 ImageNet distribution shifts in Taori et al. (ImageNet-R, ImageNet-A, ImageNet-Sketch, ImageNet-V2, ObjectNet). Also how do these robustness claims fair when you pretrain on ImageNet-LT?\n- In Section 5.2 (robustness), you should compare with methods which focus on robust finetuning (e.g. [1] and [2] which show good results on CLIP-pretrained backbones). It's known that full-finetuning reduces robustness but there's methods to mitigate those effects for certain classes of pretrained models. Also, does VPT exhibit similar robustness capabilities to your method? Since your LDA experiments show the features remain roughly similar, I would think it does.\n- Would like to see ablation results on adding the group embeddings early vs late in layers, instead of shared prompt -> group. Would also like to see what happens if you just used group embeddings.\n- You claim that \"LPT shares a pretrained model for different learning tasks, and only needs to store the small-sized prompts, largely increasing the model compatibility and reducing practical deployment cost,\" however you don't compare to any multi-task methods. This claim is weak.\n- You also make claims about the parameter efficiency of this method, however that doesn't seem to be the main selling point as you don't compare to any lightweight finetuning methods besides VPT.\n- Would like to see more comparisons with CLIP-pretrained methods on iNaturalist and Places-LT. What is the result when you use your method with a CLIP pretrained backbone? I find CIFAR results to be not convincing in general, would appreciate some clarity at larger scales. \n\n[1] Robust fine-tuning of zero-shot models, Wortsman et al. 2020\n[2] Overcoming Catastrophic Forgetting by Incremental Moment Matching, Lee et al. 2017", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster