Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models
Authors: Shuai Fu
Xiequn Wang
Qiushi Huang
Yu Zhang
Venue: ICLR 2024 spotlight (2024)
Keywords: Vision-language models; Soft-prompt tuning; Low-norm effect; Normalizing soft prompts
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper first uncovers a Low-Norm Effect phenomenon that occurs in soft prompt-tuning vision-language models (VLMs) and proposes a method for normalizing the soft prompts of VLMs to achieve better performance.

Abstract: With the prevalence of large-scale pretrained vision-language models (VLMs), such as CLIP, soft-prompt tuning has become a popular method for adapting these models to various downstream tasks. However, few works delve into the inherent properties of learnable soft-prompt vectors, specifically the impact of their norms to the performance of VLMs. This motivates us to pose an unexplored research question: ``Do we need to normalize the soft prompts in VLMs?'' To fill this research gap, we first uncover a phenomenon, called the $\textbf{Low-Norm Effect}$ by performing extensive corruption experiments, suggesting that reducing the norms of certain learned prompts occasionally enhances the performance of VLMs, while increasing them often degrades it. To harness this effect, we propose a novel method named $\textbf{N}$ormalizing th$\textbf{e}$ soft-pro$\textbf{m}$pt v$\textbf{e}$ctors of vi$\textbf{si}$on-language model$\textbf{s}$ ($\textbf{Nemesis}$) to normalize soft-prompt vectors in VLMs. To the best of our knowledge, our work is the first to systematically investigate the role of norms of soft-prompt vector in VLMs, offering valuable insights for future research in soft-prompt tuning.

Published as a conference paper at ICLR 2024
NEMESIS: NORMALIZING THE SOFT-PROMPT VECTORS OF
MODELS
Shuai Fu', Xiequn Wang's Qiushi Huang 12 Yu Zhang
DDpparmeent of Computer Science and Engineering, Southern University of Science and Technology Coomuteer Science Research Centre, University of Surrey {fus. jayce, wangxiequn, yu. zhang ust @ 1 com qiushi huangssurrey. ac.............
ABSTRACT
With the prevalence of large-scale pretrained vision-language models (VLMs), such as CLIP, soft-prompt tuning has become a popular method for adapting these models to various downstream tasks. However, few works delve into the inherent properties of learnable soft-prompt vectors, specifically the impact pose
of their norms the performance of VLMs. This motivates unexplored research question: "Do we need to normalize the soft prompts in VLMs?" To fill this research gap. we first uncover a phenomenon, called the Low-Norm Effect by performing extensive corruption experiments, suggest- ing that reducing the norms of certain learned prompts occasionally enhances the performance of VLMs, while increasing them often degrades it. To har- ness this effect, we propose novel method named Normalizing the soft-prompt vectors of vision-language models (Nemesis) to normalize soft-prompt vectors in VLMs. To the best of our knowledge, our work the first to systemati- cally investigate the role of norms of soft- offt-prompt vector in VLMs, offering valu- able insights for future research in soft soft-prompt tuning. The code is available at https: / thub. com/ ShyFoo Nemess
INTRODUCTION
In the age of large-scale pretrained vision-language models (VLMs), such as CLIP (Radford al., 2021), Flamingo (Alayrac et al., 2022), and BLIP (Li et al., 2022), soft-prompt-based methods, also known as prompt-tuning, have emerged wide range of downstream tasks. For instance, Zhou et al. (2022b) propose a Context Optimization (CoOp) method to learn soft prompts in a continuous space of CLIP for image classification tasks. Additionally, Rao et al. (2022) and Du et al. (2022) also employ prompt-tuning to address dense prediction and open-vocabulary object detection tasks, respectively.
dominant approach for adapting these models to a
Recent research in the field of VLMs has been primarily focused on enhancing model performance through the aligmment of visual and textual features. For instance, in (Lu et al., 2022), the weight distribution of output embeddings is estimated, while Zang al. (2022) propose a joint optimization approach for prompts across multiple modalities. Additionally, Chen et al. (2023) employs optimal transport techniques. To interpret learned soft-prompt vectors, Zhou al. (2022b) and Chen et al. (2023) map them to the nearest words within the embedding space. More recently, Oymak et al. (2023) delves into the role of attention mechanisms in prompt-tuning. specifically within the context of a one-layer attention network.
While considerable advancements have been made in soft-prompt-based techniques for VLMs, scant attention has been paid to their intrinsic properties, specifically the norms of learnable soft oft--rrompt vectors. We argue that the norms of soft prompts are crucial but overlooked attribute that sig- nificantly influences the performance of VLMs. This paper addresses an overlooked aspect and presents a research question "Do we need to normalize the soft prompts in VLMs?" To the best of our knowledge, there is no work to study this question.
"oorreponniigg author.

---

Published as a conference paper at ICLR 2024
Leamable Solt Phonptt
Class
Classlictic Bandt
66
Imagetet
JEASE
Cuttechol
Oafordhees
StanfordCars
Deceoose Norwn
Regiossse
Food102
Fresuerryy
1066
Ps
Pg
VLM
SUN397
DTD
EUrOBAT
UCF 101
10
166
Shots
(a) Top: corrupted soft prompts with increased norms (b) The occurrence frequency of the Low-Norm leading to decreased performance; Middle; soft prompts Effect across 11 datasets. Each distinct color learned by CoOp; Bottom: corrupted soft prompts with geometrical shape represents a different dataset. reduced norms resulting in enhanced performance.
Figure 1:
A schematic diagram of the Low-Norm Effect. (a) The occurrence of the Low-Norm Effect in soft-prompt tuning VLMs. (b) The occurrence frequency of the Low-Norm Effect across 11 datasets commonly used in soft-prompt tuning VLMs.
To thoroughly investigate the role of soft-prompt vector norms VLM performance, we introduce two corruption operations, REPLACE and RESCALE, to alter the norms in vectors learned by CoOp (Zhou et al., 2022b). Through the corruption of learned soft prompts, an intriguing phenomenon emerges: the reduction of norms at specific positions within these prompts enhances performance, whereas an increase in norms typically results in performance deterioration, as illustrated in Figure (a)). We term this previously uncovered phenomenon the Low-Norm Effect.
Figure 1(b) explores the prevalence of the Low-Norm Effect across 11 widely-used prompt-tuning VLM datasets. Notably, the Imagenet (Deng al., 2009), OxfordPets (Parkhi et al., 2012), and Food101 (Bossard et al., 2014) datasets show higher frequency of this effect compared to datasets like Flowers102 (Nilsback Zisserman, 2008). Moreover, we observe a negative correlation be- tween the occurrence frequency of the Low-Norm Effect and the number of shots, indicating that less training data tends to induce the effect. These discrepancies might be linked to potential degra- dation in soft-prompl methods with limited data resources, thereby affecting model performance. Addressing the Low-Norm Effect remains challenging due to its inconsistent manifestation across datasets.
To exploit the Low-Norm Effect for enhancing the performance of VLMs, we propose a method called Normalizing the soft-prompt vectors of vision-language models (Nemesis). We employ Position-Uniform Normalization (PUN) loss to regulate the norms of all prompt vectors. This ap- proach can be easily integrated into existing soft-prompt methods with negligible computation costs. However, the PUN loss may degrade the performance since it may normalize soft-prompt vectors that are unaffected by the Low-Norm Effect.
To handle this, the Position-Aware Normalization (PAN) loss is proposed as a refined substitute for the PUN loss. Specifically, pre-inference step introduced before each training batch to identify positions that are likely to induce the Low-Norm Effect. The pre-inference step applies corruption operations generate multiple sets of corrupted prompts at distinet positions and then evaluates these corrupted prompts against their non-corrupted counterparts. This allows for the identification of positions that induce the Low-Norm Bffect, followed by selective normalization of those positions, ultimately improving the performance. Extensive experiments demonstrate that the proposed Nemesis method could help boost the performance of soft-prompt-bas based VLMs in various tasks.
LOW-NORM EFFECT
In this section, we examine how the norms of learned prompt vectors influence the performance of VLMs and identify the Low-Norm Effect. To achieve that, conduct extensive corruption
2

---

Published as a conference paper at ICLR 2024
experiments by altering the norms of learned prompt vectors and applying two corruption operations (i.e., REPLACE and RESCALE) proposed in Section 3.2.
Specifically, following (Zhou et al., 2022b), we train the CoOp model and obtain the learned prompt vectors with a length of L. Then we corrupt these soft-prompt vectors at a single position each time and record the change of the performance and norms of soft prompts. To reduce the effect of experimental randomness, we average outcomes over five distinct runs using different seeds. Implementation details and results can be found in Appendix A.2.3.
The corruption experiments reveal a interesting phenomenon: reducing the norms of soft prompts at specific positions enhances the performance, while increasing them could degrade the performance, which is termed Low-Norm Effect. This phenomenon responds to the research question posed in Section 1 by uncovering the previously unexplored Low-Norm Effect in prompt-tuning VLMs. also motivates the proposed Nemesis method for soft prompt normalization. Moreover, Section 4.6 offers plausible explanation for the occurrence of the Low-Norm Effect and provides insights into the effectiveness of the proposed method. We believe our discovery can offer valuable insights for future research soft-prompt tuning, laying the groundwork for potential advancements.
METHODOLOGY
In this section, introduce the proposed Nemesis method. We begin with a review of the CoOp method (Zhou et al., 2022b) and subsequently introduce two key corruption operations, REPLACE and RESCALE. Finally, present the entire method.
3.1 A REVISIT OF PROMPT-TUNING VISION-LANGUAGE MODELS
Over the years, pretrained VLMs have demonstrated impressive generalization performance in zero- shot open-world visual recognition, wherein the model can perform task without undergoing plicit training. One typical paradigm is CLIP (Radford et al., 2021). which consists of an image en- coder and text encoder. CLIP is trained on approximately 400 million image-text pairs, contribut- ing to its remarkable performance. Nevertheless, effectively fine-tuning these VLMS for downstream tasks remains a challenge, particularly when dealing with few-shot data, due to their massive param- eters. The CoOp method addresses this issue by setting the templated context prompts (e.g. This is a photo of (class- naass--ammee.. as learnable vectors, which only requires fine-tuning these learnable vectors while keeping the pretrained VLMs frozen. For a downstream visual recogni- tion task consisting of C' categories, the classification weights of one image can be defined by the similarity between the visual feature and the text features of all categories.
Formally, the image encoder and text encoder can be denoted by I and g. respectively. Given an image x along with its classification label y. the visual feature can be formulated as f()), while the textual prompt of i-th class can be formulated as t, = {Vi, U2 Uj...., UL ). where vy and c, denote the j-th soft-promp pott-prompt vector and the word embedding of the class name, respectively. Then the i-th class textual feature can be denoted as g; gt.).. Given few-shot data, CoOp can learn the soft prompts xLxD dimension of prompt vectors, respectively, by minimizing the negative log- bog-likeldhood between the image feature f and its ground-truth textual feature By as
{U1, U2 .... } where L and D denote the length of soft prompts and the
exp(sim(f. .gy)/A) LCE= -- Σ log E 2 (x.)sim(ff.::/A))
ZEX
where À is temperature parameter and sim(..) .) denotes the cosine similarity function. After the training process, the text encoder g encodes both the learned prompts V and the class embeddings to produce textual features for all classes.
3.2 CORRUPTION OPERATIONS
In this section, we introduce two corruption operations: REPLACE and RESCALE, which can be employed to corrupt the learned soft-prompt vectors.
For the REPLACE operation, we replace learned prompt vectors at a single position with a zero- mean Gaussian-distributee vector with fixed variance. Then, obtain set of corrupted soft
3

---

Published as a conference paper at ICLR 2024
prompts t {u1 22 ry..... UL, C,), where the j-th prompt vector is replaced with random Gaussian vector ry ~ Mu1,, I²),, where denotes vector of all ones with an appropriate size, I denotes an identity matrix with an appropriate size, μ denotes the mean for each dimension, and denotes the standard deviation for each dimension.
For the RESCALE operation, we rescale the value of j-th prompt vector using various rescaling factors. Then, we can obtain set of corrupted soft prompts t.e {w) U2 U2, x Vj.... Uz, Ci). where s a rescaling factor.
In essence, both corruption operations be regarded as strategies to modify the soft prompts, subsequently altering their norms. By changing soft prompts, the textual features generated by g are influenced, consequently impacting final predictions. Specifically, given corrupted soft prompts, the prediction probability can be calculated as
exp(sim (f. //A)
Pesp (v)p) = Ei= exp(sim(( f. 9, _;/r)/)
where geep can be either g({T)) g(,)"). Note that the two corruption operations proposed can be applied to any learned soft prompts, thereby facilitating research in the field of investigating the impact of soft-prompt norms the model performance.
3.3 THE NEMESIS METHOD
To handle the Low-Norm Effect during prompt-tuning VLMs, propose two losses for normal- izing the norms of soft prompts: Position-Uniform Normalization (PUN) loss and Position-Aware Normalization (PAN) loss. In the experiments, they separated as individual regularization item, which is added to the standard soft-prompt tuning process.
Generally, given set of soft prompts VLXD {vi, 2,, )r we can calculate their norms as
M ay1|ey1lp where M denotes the number of non-zero values in the set {ar ........ ., at} and - Pp denotes the norm vector. Unless otherwise specified, we use the & norm by default.
For the PUN loss, all elements of the set {ai, .2..... } are set to the same value, imposing an equal weight on the norms of soft prompts at all positions. Hence, this loss can be formulated as
1 LPUN
Eaglluyle. M
where aj = w j 1,.j=,,.., I,...,,. L. Here w scaling coefficient that controls the normalization strength. However, normalizing prompt vectors at positions unaffected by the Low-Norm Effect may not yield performance improvement. This is because could potentially restrict the weight updates of soft prompts at these positions. Hence, it is necessary tackle the Low-Norm Effect at each prompting position and dynamically adjust oj during training.
On the other hand, if the Low-Norm Effect be explicitly recognized during the training process, we can effectively address this issue and enhance the efficacy of soft-prompt learning. To achieve this, we incorporate an additional inference process prior to each batch training iteration identify the prompting positions that induce the Low-Norm Effect.
rescaling factor, denoted by т, induce the Low-Norm Effect, where 7 is positive real number less than 1. Then we apply the RESCALE oper- ation on normal soft prompt
Similar corruption experiments, initially
generate N of corrupted prompts distinct prompting posi- tions (VL, Viy...,, ..,,..., Vix J. where l, denote corrupted positions. Note that for each training batch, we randomly select N distinct positions from the set of L positions L = {1, 2.... L}. For- mally, the conditions < l # l2... # I.... F iN < L and N < L ensure that the positions of rescaled prompt vectors for each set of corrupted prompts are distinct from each other.
By having a set of images 2.' with batch size of B as well as their ground-truth labels w (Vi,..., (B), and hybrid prompt set (N++))XLDD = [,, Vi, Vig...., Viy J. where V is the orig- inal prompt and others corrupted prompts, we obtain set of label predictions (N+1)xx
4

---

Published as a conference paper at ICLR 2024
of VLMs, where the first corresponds to the batch predictions for the original prompt and the next N rows represent the batch predictions for its corrupted counterparts.
By calculating prediction scores for all prompts and comparing them, prompts at specific positions that yield more accurate predictions than their original counterparts. This observation indicates the occurrence of the Low-Norm Effect during training. As result, only those soft prompts at the prompting positions that induce the Low-Norm Effect will be normalized during this training batch. Hence, the PAN loss can be defined
identify corrupted
IN
LPAN =
1
Σ
a
(4)
M
k=1
1ij yo) and otherwise for A 11, 12, .2.... Iv. Vo.k refers to the prediction for the b-th sample in the batch, using the soft prompts which have been corrupted at k-th position, while ye represents the prediction for the b-th sample in the batch, using the original, uncorrupted soft prompts. The detailed algorithm by adopting LPAN can be found in Section A.
where og equals w Σ>=1
[(hn, = 1b) Σ
To up, for a training batch, can optimize soft prompts by minimizing the following total objective as
L= Lcn + 3CuuN + (1 - BANN
(5)
where which equals either corresponds two variants of the proposed Nemesis method.
4 EXPERIMENTS
In this section, extensive experiments are conducted to evaluate the proposed Nemesis method, including comparison with CoOp (Zhou et al., 2022b) on few-shot image classification tasks and domain generalization tasks, comparison with CoCoOp (Zhou et al., 2022a) in the base-to-new generalization setting. Additionally, we conduct an in-depth impact analysis on VLM performance due the norms of soft prompts, explore the method's extensibility other soft-prompt tuning approaches, and assess the computational efficiency.
4.1 DATASETS
For few-shot image classification experiments and base-to-new generalization tasks, follow the experimental setting CoOp and CoCoOp. respectively, and conduct experiments on 11 visual classification datasets, including Caltech 101 (Fei-Fei al., 2004) and Image (Deng al., 2009) for object recognition, AT (Helber et al., 2019) for satellite image recognition, DTD (Cimpoi al., 2014) for texture recognition, UCF101 (Soomro al., 2012) for action recognition. SUN397 (Xiao et al., 2010) for scene recognition, OxfordPets (Parkhi et al., 2012), FGVCAircraft (Maji al., 2013), Food101 (Bossard al., 2014), Flowers 102 (Nilsback & Zisserman, 2008), and Stan- fordCars (Krause et al., 2013) for fine-grained recognition. Besides, ImageNet (Deng et al., 2009) and its variants, including ImageNet-A (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a) 1a), ImageNetV2 (Recht et al., 2019), and Neeeehht (Wang et al., 2019), are used for the evaluation of domain generalization. Detailed descriptions of each dataset can be found in Appendix A.
4.2 IMPLEMENTATION DETAILS
For few-shot image classification experiments and domain generalization tasks, we compare our method with the baseline method CoOp, while CoCoOp is chosen as our baseline model in base- to-new generalization tasks. Following the few-shot evaluation protocol used in CoOp, we use a fixed number of training samples from each category (i.e. 1, 2, 4, 8, 16 shots per class), Besides, we follow the same training configurations as these baseline models, including training epochs, learning rate, and batch size, etc. All reported results Bold denotes the best performance on each comparison setting. More implementation details and hyper-parametter settings can be found in Section A.2.2.
based the average of five different seed runs.
5

---

Published as a conference paper at ICLR 2024
Average over 11 datasets
Imagellet
Caliech102
Oxforotats
12.
72.5
1-2
753
Cabty Canye-erm-soiiig cady=-een--coPp
coop-theneeccom (PSN) cadpeteeees (800
Eotyy-eeeeei yun
Ex0p-teered JeHt
21.
Eony-Semeed jeai
Svenpercos
St00 sercom
Shotrper den
sieea com
StanforsCars
F Fowers102
Food103
FGVCAICroft
12.3
12.0-
25.5
750
CoOp
ChTp
-4- Do0p
53.
Codgp Memeaau
Maress
Carpp theress (PUN)
Mereed 2000
12.5
SUN397
EurosAl
UCF101
10.0
163
12.3
14.3
Ex0p-leeres
Eo0g- pun
450
120
Chop-hemee (RR1))
2:2
Svenbercos
Stot
Sharppr des
5tea
Figure 2: The few-shot recognition results of CoOp and CoOp+Nemesis (ours) on 11 datasets.
4.3 FEW-SHOT IMAGE RECOGNITION RESULTS
The experimental results of few-shot recognition are summarised in Figure 2. The blue, orange, and green lines represent CoOp, CoOp+Nemesis with the PUN loss, and CoOp+Nemesis with the PAN loss, respectively. In terms of average performance, both Nemesis methods outperform CoOp. Particularly, they achieved a large improvement over CoOp on the ImageNet, OxfordPets, Food101, and SUN397 datasets. This indicates that normalizing the soft prompts in VLMs can lead to bet- ter performance on these datasets that exhibit a more pronounced Low-Norm Effect. Taking the ImageNet dataset 0.38% performance boost over CoOp at 1, 2, 4, 8, 16 shots. Similarly. Nemesis with the PUN loss also shows performance improvements of 0.46%, 1.56%. 1.74%, 0.80%, and 0.44%. Moreover, it is evident that CoOp+Nemesis demonstrates enhanced robustness and superior performance on the Food101 and OxfordPets, compared with CoOp. Additionally, comparing Nemesis with the PUN loss, Nemesis with the PAN loss shows more robust performance at larger shot settings. All these performance comparisons demonstrate normalizing the soft prompts in VLMS can facilitate the ef- fective learning of soft prompts for few-shot recognition. More detailed data and analysis of training process can be found in Appendix A.2.6.
example, Nemesis with the PUN loss gains 2.06%, 3.84%, 2.6%, 1.16%,
4.4 EVALUATION OF GENERALIZATION PERFORMANCE
In this subsection, we conduct experiments to assess the generalization performance of the proposed method. All methods are trained on the ImageNet dataset with 16 shots per class and tested on four different ImageNet- datasets. Table 1 reports the results of CoOp, CoOp+Nemesis (PUN), and CoOp+Nemesis (PAN). It is clear that CoOpsNemesis outperforms CoOp consistently on both source and target domains, whether adopting the PUN loss or PAN loss, which suggests that Nemesis can improve CoOp's domain generalization abilities by normalizing the soft prompts in VLMs. Furthermore, we can observe that Nemesis using larger w can achieve better transfer performance, implying that a stronger normalization of soft prompts could enhance the robustness of soft prompts to domain shifts. Comparing Nemesis using the PUN loss and Nemesis using the PAN loss, despite that the latter achieves better performance on the source domain, its performance on target domains is inferior to the former. We argue that this may arise due to the PAN loss excessively prioritizing to identify and address the Low-Norm Effect within intra-domain data, which could compromise its generalization capability. The results of base-to-new experiments can be found in Appendix A.2.4.

---

Published as a conference paper at ICLR 2024
Table Comparison of CoOp and CoOp+Nemesis (ours) in the domain generalization setting.
Source
Target
Method
ImageNet
-V2
-Sketch
-A
-R
CoOp
62.70
55.04
32.64
22.44
54.60
CoOp+Nemesis (PUN, 1)
62.96
55.48
33.64
22.96
55.72
CoOp+Nemesis (PUN, 10)
63.08
55.50
34.34
23.40
56.78
CoOp+Nemesis (PAN,
63.28
55.16
32.70
22.48
54.72
CoOp+Nemesis (PAN, w = 10)
63.14
55.20
33.50
23.30
56.68
145
56.5
Neess
105
36.0
-1.00
56.0
1.00
55.5
0.95
Soore
55.0
Eecenay Rarmseed
00.00
545
54.5
CoCp Goord
$4.0
540
CoOp-Nemesk: (cord)
CoOp (score)
CoCp (hemm value)
53.5
CoOp (horm vallee
0.80
C50g-Nemesk mom valuet
(a) The occurrence of the Low-Norm Effect in CoOp. (b) Comparison of CoOp and CoOp+Nemesis (ours).
Figure 3: Analysis of the impact of soft prompt norms on the model's performance using the Stan- fordCars dataset as an example. The X-axis, Yl-axis, and Y2-axis represent training epochs, test accuracy, and average norm value of all soft-prompt vectors, respectively.
It also be found that the generalization performance from base classes unseen classes be improved by normalizing the soft prompts in VLMs.
4.5 IN-DEPTH STUDIES ON THE LOw-NORM EFFECT IN VLMS
In this section, we aim to provide plausible explanations for the occurrence of the Low-Norm Effect and the effectiveness of the proposed method Nemesis.
From Figure 3(a), apparent that the norms of soft prompts in CoOp first increase and then level off, while test accuracy falls into degradation as norms slowly flatten out. By performing corruption operations that decrease the norms of prompt vectors, the last green circle may be pushed away from the degradation area and get closer those small green circles that demonstrate superior performance. This could regarded as a plausible explanation for the occurrence of the Low- Norm Effect: those corrupted soft prompts that demonstrate superior performance than their original counterparts may be precisely one of those small circles. Moreover, this figure may unveil a potential correlation between the time when prompt learning starts to degrade and the time when the norm of soft prompts begins to stabilize. We leave this to future research.
From Figure 3(b), different from the observed norm variation pattern in CoOp, CoOp+Nemesis (ours) exhibits a distinct trend where norms initially increase, followed by subsequent decrease, and eventually stabilize. Furthermore, the test accuracy exhibits consistent upward trend before reaching a plateau, whereas declining trend is observed in CoOp. This implies that our method can delay the time point where soft prompts tend to plateau during the learning process, thereby reducing the probability of learning degradation.
4.6 EXTENDIBILITY ANALYSIS
To analyze the extensibility of the proposed approach Nemesis, we apply the proposed method Nemesis to other soft prompt-tuning methods few-shot recognition experiments. PLOTT (Chen

---

Published as a conference paper at ICLR 2024
Table 2: The few-shot recognition results PLOT and PLOT+Nemesis (ours) on several datasets.
Dataset
Method
1 shot
shots
shots
shots
16 shots
PLOT
88.80
87.94
89.44
90.46
92.16
Caltech101
+Nemssss (PUN)
89.14
88.02
89.74
90.38
92.10
PLOT+Nemesis (PAN)
89.16
87.98
89.86
90.80
92.30
PLOT
46.42
51.52
56.10
60.66
64.16
DTD
+Nemess (PUN)
46.64
51.58
56.22
60.94
64.46
PLOT+Nemesis (PAN)
46.65
51.40
56.08
61.56
64.68
PLOT
71.06
81.38
87.68
92.58
94,92
Flowers
P (PUN)
70.94
81,44
88.42
92.86
95.32
+Nemesiss (PAN)
71.40
81.96
88.24
93.06
95.46
PLOTT
68.76
73.61
77.74
81.23
83.75
Average
PLOT+Nemesis (PUN)
68.91
73.68
78.13
81.39
83,96
PeOT+Nemssis (PAN)
69.07
73.78
78.05
81.81
84.15
Table 3: Ablation studies about the PAN loss on the FGVCAireraft dataset. N denotes the number
of corruption positions, while w/o selection means the prompting positions where the soft prompts directly normalized, without the selection process within the PAN loss.
Method
1 shot shots 4 shots 8 shots 16 shots
CoOp
9.56
18.30
21.04
26.66
31.64
NoOpsNemesis (N 1, www selection)
9.60
19.34
21.64
26.98
30.92
CoOp+Nemesis (N 1)*
9.88
18.52
20.94
27.50
32.32
CompsNemesis (N 2, w/o selection)
9.50
18.72
21.72
27.22
30.90
CoOp+Nemessi (N 2)
9.74
18.38
21.18
27.36
32.46
CoOp+Nemesis (N = 4, w/o wo selection)
10.04
19.20
22.10
27.36
31.26
CoOp+Nemesss (N 4)
9,44
19.02
21.26
27.52
32.10
CoOp+NNmmesis (N 8, wtwe selection)
9.62
19.28
21.58
26.92
30.88
CoOp+Nemesis (N 8)
9.30
19.32
21.66
27.36
32.02
CoOp+Nemesis (N 16, w/o selection)
9.82
19.24
21.52
26.98
31.12
CoOpsNemesis (N 16)
10.44
19.72
22.13
27.83
31.44
et al., 2023) is ensemble-based prompt-tuning method and leverages optimal transport distance to achieve better alignment between multiple sets of soft prompts and image features. In this ex- periment, We set 0.1 by default. Table 2 provides partial results for comparing PLOT and PeosiNemesis (our approach). In comparison to PLOT, NeOTss demonstrates enhanced recognition performance for all shots. This verifies that ensemble-based prompt-tuning methods can also benefit from normalizing the soft prompts in VI.Ms. Furthermore, we discuss other applicable scenarios of the proposed method. More details can be found in Appendix A.2.8.
4.7 ABLATION STUDIES AND HYPER-PARAMIN PARAMETER ANALYSIS
The results of ablation studies about the PAN loss are shown in Table 3, where w/o selection is analogous to utilizing the PUN loss to normalize soft prompts at N random positions and means the default setting in few-shot recognition tasks. Overall, increasing N typically leads to improved performance, except for the case of 16 shots, where performance initially increases and then decreases with N. Besides, comparing the PAN loss without considering the selection process, the proposed selection of prompting positions that are used for normalization based on performance variation demonstrates a more robust performance.
Furthermore, Table 4 provides the results of combining the two proposed losses. We can observe that (3003 and 3=0.1 achieve the best results under 1-shot and 16-shots, respectively. This suggests that the PAN loss plays dominant role, while the PUN loss provides assistance, which can lead to improved performance. Besides, the discussion about the computation costs raised by the two proposed losses can be found in Appendix A.2.7.
Moreover, the hyper parameteret analysis about w. т, and norm types used in the normalization losses can be found in Tables A5, A6, and A7, respectively. The outcome data demonstrate that the pro-
8

---

Published as a conference paper at ICLR 2024
Table Ablation studies about B.
3=0
B=0.1
3=0.3
3=0.5
$=0.7
3=0.9
B=1
I-shot
59.33
59.45
59.54
59.29
59.26
59.37
59.42
16-shots
74.20
74.39
73.37
74.02
73.93
73.79
73.47
posed method Nemesis, resilient when different norm types and values of used. Additionally, we observe that larger value of w generally yields better performance for small shots, whereas a smaller value of w performs well for large shots. This finding combined with the results of corruption experiments presented in Section 2, implies that soft prompts with higher occurrence frequency necessitate stronger normalization. Therefore, determining the appropriate size of is also one of the potential research directions for the future.
RELATED WORK
5.1 VISION-LANGUAGE MODELS PRE- TRAINING
Vision-language models (VLMs), usually consisting of visual module and language module, expected to explore the semantic connections between images and texts. With large-scale image- text pairs which are available on the internet, an increasing number of pre-trained VLMs (Radford et al., 2021: Cui et al., 2022; Yao et al., 2021) have been proposed. These pre-trained VLMs can capture deep langual semantic correspondence by using various vision-languaa objectives, including contrastive objective (Radford et al., 2021; Cui et al., 2022; Singh et al., 2022; Yao et al., 2021; Zhong et al., 2022), generative objective (Cui al., 2022: Singh et al., 2022), and alignment objective (Yao et al., 2021; Zhong et al., 2022). Moreover, these pre-trained VLMs also show strong capacities for generalization and can perform zero-shot predictions (without fine-tum tu-iinnin models) on wide range of downstream tasks, such image classification (Radford et al., 202 I), visual question answering (Alayrac et al., 2022), and text-guided image generation (Avrahami et al., 2022).
5.2 EARAMETER-EPPICIENT RIN-TUNING
Parameter-eeffiiien fine-tuning (PEFT) methods serve as a crucial approach for adapting pretrained models, particularly within the Natural Language Processing (NLP) domain. Among these, prompt- tuning (Lester et al., 2021; Jiang et al., 2023) has gained attention for optimizing task-specifi prompt embeddings. providing performance similar full parametric fine-tuning but with fewer tunable pa- rameters. Similarly, prefix-tuning (Li & Liang, 2021) extends this concept by optimizing sequence of prefixes at each transformer layer, thereby augmenting the set of tunable parameters marginally, while P-tuming (Liu et al., 2022) incorporates manually designed patterns to intersperse learned prompts within the input embeddings. Inspired by these PEFT methods NLP, this technique has been successfully extended to VL.Ms. For instance, CoOp (Zhou al., 2022b) and its variants (Zhou et al., 2022a) apply CLIP (Radford et al., 2021) to few-shot visual recognition tasks by replacing hard-crafted prompts with learnable soft-prompt vectors. 2023), which allows for fine-tuning a part of the network or fine-tuning an extra network, are an- other research direction of PEFT method of VLMs. Distinctively, the proposed method, Nemesis, first provides empirical evidence that normalizing poft-prompt vectors of VLMs can help improve performance.
addition, adapter-tuning (Gao et al.,
6 CONCLUSION
In this paper, we are the first to examine the impact of soft prompts" norms on the performance of VLMs. We conduct extensive corruption experiments using two specially designed operations and discover the Low-Norm Effect. To harness this phenomenon, we introduce Nemesis, method for normalizing soft prompts during soft-prompt tuning. In general, Nemesis can be incorporated into any fett-rrompt-bssed methods, other PEFT methods, such prefix-tuning, and P-tuning. We hope our findings and proposed method can provide new insights and facilitate future research on these fields.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work is supported by NSFC key grant under grant no. 62136005, NSFC general grant under grant no. 62076118, and Shenzhen fundamental research program JCYI20220032410500000.
REFERENCES
San--Batistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: A visual language model for few-shot learning. In NeurlPS, 2022.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- mod-100-mining discriminative compo- nents with random forests. In ECCV, 2014.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vasion-language models. In ICLR, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Quan Cui, Boyan Zhou, Yu Guo, Weidong Yin, Hao Wu, Osamu Yoshie, and Yubo Chen. Con- trastive vision-language pre-training with limited resources. In ECCV, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: A large-scale
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, 2005.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2020.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. CVPR, 2022.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, 2004.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, pp. 1-15, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-22) 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
10

---

Published as a conference paper at ICLR 2024
Weisen Jiang, Yu Zhang. and James Kwok. Effective structured prompting by meta-learning and representative verbalizer. In ICML, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, 2013.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In EMNLP, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In ICML, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL (Long Paper). 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning be comparable to fine-tuning across scales and tasks. In ACL (Short Paper), 2022.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. CVPR, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint vrXiv:/306.511 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008.
Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis, On the role of attention in prompt-tuning. ICML, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICMI., 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In ICML, 2019.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12/2.0402, 2012.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. NeurIPS, 2019.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language- aanguage-image image pre-training. In ICLR, 2021.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arX1:22210.07225, 2022.

---

Published as a conference paper at ICLR 2024
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 190.:4990 2019.
Yiwu Zhong, Jianwei Yang. Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, al. Regionclip: Region- basedd language-image pretraining. In CVPR, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, 2022a.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2345 2022b.
12

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 METHOD DETAILS
During the training process, the proposed method Nemesis adopting the Position-Aware Normal- ization (PAN) loss would involve an additional inference process to determine those prompting positions inducing the Low-Norm Effect. On the other side, Nemesis does not change the inference process original voft-prompt-bsee methods. Hence, here we only provide more details of the training process for Nemesis adopting the PAN loss in this subsection. Its iterative optimization procedure is summarized in Algorithm
Algorithm 1:
The training process of Nemesis adopting the PAN loss.
Input:
The training data fa, y) E (X, Y), pretrained CLIP model f and g. the threshold т inducing the Low-Norm Effect, the number of corrupted positions N for soft prompts, the scaling coefficient w for the norm of soft-prompt vectors, the total training epochs E, and the training batch size B.
Output: The parameters of soft prompts V = {11, U2..... ..... UL}.
Function updat e _alpha (V, 2, V/ T, w, N)
Randomly select N distinct positions {41, l1,,,.....44 (vn from the set of L positions
{1,2...., L} to generate the corrupted prompt set;
Obtain the corrupted prompt set (N++)xxXDD = (V, Vi....,, Vinh where the original prompt;
// Enabl ing the process of inference
Generate a visual feature set BBC of with the visual encoder f(22);
Generate a corrupted textual feature set BBB(N+))xC of each class with textual encoder g(concat (V,c,)) (,)),, where c; denotes the word embedding of i-th class
name;
Calculate the similarity between visual and textual feature S, = 8,F of each class; // Terminating the process inference
Ú1
9n. 82
ÚB
U1,1
02.1
Obtain the classification predictions z
:
01.xx
U2,1
ÚB.I/
Calculate a set of prediction performance:
(> 1 = yb) Σ 1(0.11, = 3/b)
D B
100..N = yo));
for ln to (11 12.2........ IN do
if b= 1(bb.1. = ybb > ΣB= 1(9: = vbb) then
10
11
01
12
else
13
al =
14
return {ai, :ig....;
Initialize V for to E do
17
for each training mini-batch 2 2/ do Obtain (ot, Olty Oiw using update alpha (V, 2, 3 w, N);
18
IN
Calculate the PAN loss LPAN コ
Σ aj
20
Generate the visual feature set & and textual feature set S, of each class;
exp(sim( F -2,/A)
21
Calculate the cross-entropy loss LCE =
Σ
log
2EX
Σ exp(siml 8))/2)
22
Update by minimizing LeB and LPAN
13

---

Published as a conference paper at ICLR 2024
Table Al: The detailed statistics of datasets.
Dataset
Classes
Train
Test
Task
Caltech101 (Fei-Fei et al., 2004)
100
4,128
2,465
Object recognition
ImageNet (Deng et al., 2009)
1,000
1.28M
50,000
Object recognition
EuroSAT (Helber et al., 2019)
10
13,500 8,100
Satellite image recognition
DTD (Cimpoi al., 2014)
47
2,820
1,692
Texture recognition
UCF101 (Soomro et al., 2012)
101
7.639 3,783
Action recognition
SUN397 (Xiao et al., 2010)
397
15,880
19,850
Scene recognition
OxfordPets (Parkhi et al., 2012)
37
2,944 3,669
Fine time-grained pets recognition
FGVCAircraft (Maji et al., 2013)
100
3,334
3,333
Fine- line-griined aircraft recognition
Food101 (Bossard et al., 2014)
101
50,500
30,300
Fine-grained food recognition
Flowers 102 (Nilsback & Zisserman, 2008)
102
4,093
2,463
ammeggrained flowers recognition
StanfordCars (Krause et al., 2013)
196
6,509
8,041
Fine-grained cars recognition
ImageNet-A (Hendrycks et al., 202
200
N/A 7,500
Robustness of adversarial attack
ImageNet-R (Hendrycks et al., 2021a)
200
N/A 30,000
Robustness of multi-domains
ImageNetV2 (Recht al., 2019)
1,000
N/A
10,000
Robustness of collocation
ImageNet-Skee -kettthhhh (Wang et al., 2019)
1,000
N/A 50,889
Robustness of sketch domain
A.2 EXPERIMENTAL DETAILS
A.2.1 DATASETS
Following CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a), the datasets we used in- clude 11 datasets for few-shot visual recognition and basc-to-new generalization tasks, well as ImageNet-baad datasets for the evaluation of domain generalization. The details each dataset can be found in Table AI, including dataset name, the number of classes, the number of training and testing samples, as well as the type of visual task.
A.2.2 HYPER- PARAMETER SETTINGS
To normalize the soft prompts in VLMs, two types of normalization losses are proposed: the Position-Unifone Normalization (PUN) loss and the Position-Aware Normalization (PAN) loss. Both losses involve a crucial hyper-parameter w. which controls the extent of normalization for soft prompts. Unless specified otherwise, is where it is set to 10, the OxfordPets dataset, where it is set to 50, and the Food101 dataset where it is also well on these three datasets when w is relatively large. This observation aligns with our discovery of a pronounced Low-Norm Effect in these datasets, providing evidence that our method is indeed capable of addressing the Low-Norm Effect. At the same time, provide decreasing schedule of w for a better balance between Loe and the PUN or PAN loss. To be specific, it is varied based on logistic function Wg 1 #-ezp(-k(EE 0.5 m) where E and maxg denote current training epoch and maximum training epoch, respectively. le represents the attenuation rate, and it fixed as 0.2.
for all datasets, except for the ImageNet dataset
50. Based on our experimental findings, we observed that our approach performs
In addition to w, the PAN loss incorporates two important hyper-parametert the number of corruption positions N and the pre-defined threshold 7 inducing the Low-Norm Effect. It should be noted that the size of the additional inference cost incurred by the PAN loss positive correlation with N. Technically, the text encoder needs to perform inferences computational efficiency experiments, we set the default value of N to and to 0.5. These settings chosen to minimize computational costs while maintaining the desired performance.
a batch of B x (N + 1). In the
Furthermore, the baseline model CoOp has multiple variations, including multiple backbones (e.g. ResNet-50 (He et al., 2016) and VIT-B/16 (Dosovitskiy et al., 2020)), different positions for the class token (e.g.. "front", "middle", " and "end"), various lengths for the soft-prompt (e.g.. and 16), and multiple parameter initialization strategies. For a clear comparison, here choose of them as our baseline with ResNet-50 backbone, the class token at the "end" position, 16 coft-prompt tokens, and "random" initialization.
14

---

Published as a conference paper at ICLR 2024
A.2.3 CORRUPTION EXPERIMENTS
This subsection provides more details of corruption experiments implemented by the two corruption operations we proposed.
The REPLACE operation involves replacing the prompt vector at a single position with randomly generated vector from a Gaussian distribution, which is characterized by zero mean and fixed variance. By modifying the variance of the Gaussian distribution, we can roughly control the norms of generated Gaussian vectors. To be specific, increasing the variance leads to higher norms of the generated Gaussian vectors while decreasing the variance results in lower norms. We employ five different variance values: O, 0.001, 0.01, 0.1, and 0.5.
Furthermore, the RESCALE operation utilized to rescale the prompt vector at single position by applying various sizes of rescaling factors, including 0.001, 0.01, 0.1, 0.5, and 2. The first four rescaling factors are employed to reduce the norms, while the last one is utilized to increase the norms. As result, for soft prompts of length L and corrpption operation, experiments I. times. Then, we can calculate the occurrence frequency with which the performance of corrupted prompts exceeds that of their original counterparts. Tables A2 and A3 provide de- tailed record of this occurrence frequency under different corruption operations, respectively. It is noteworthy that the results of occurrence frequency of the Low-Norm Effect across 11 datasets (i.e. the results of Figure I) are calculated by the sum of four rescaling factors, including 0.001, 0.01, 0.1, and 0.S.
conduct corruption
For better intuitive understanding, we present part of corruption experiments in the form of bar graphs. Take the ImageNet dataset under I shot setting as an example, as shown in Figure Al. From Figure Al(c), we can perceive that the replacement of prompt vectors at positions 1-4 with random Gaussian vector having a zero mean and a variance of 0.001 barely changes the model's performance. Surprisingly,
improvement in performance is observed for positions 5, 7, 8, and 9. Additionally, a similar pattern can be found in Figure (l))), where we observe varying degrees of performance improvement when the prompt vector at positions 1-14 is rescaled to half of its original magnitude (i.e. reducing the norms). On the contrary, the model's performance for all positions experiences varying degrees of decline when the norms of soft prompts increase, whether the Replace or Rescale operation, as shown in Figure (g))), (i and A Al)).
A.2.4 BASE-TO-NEW RESULTS
In this subsection, we compare the baseline model CoCoOp and CoCoOp+-Nmmesis (ours) in the tase-to-new setting. CoCoOp inputs batch of image features into additional neural network to generate instance-conditional prompts, which are added to soft prompts to produce a batch of final prompts. Following CoCoOp, all methods are implemented with VIT-B/16 backbone and evaluated with 16 shots. We report the performance on 11 datasets, including the base classes (Base), new classes (New), and the harmonic mean (H) of both. We present comprehensive results of base-to- new experiments conducted on all datasets, as illustrated in Table A4. We can observe that increasing the strength of normalization of soft prompts (i.e. larger w) would have a slight negative effect on the performance of base classes but better enhance the performance of new classes. This suggests that the generalization performance from base classes to unseen classes can be improved by normalizing the soft prompts of VLMs. In particular, CoCoOp+Nemesis (PAN, 20) achieve performance improvement of 1.3% compared to CoCoOp for new classes.
A.2.5 RESULTS OF ABLATION STUDY AND HYPER-PARAMETES ANALYSIS
This subsection presents the ablation results for the normalized strength w. The outcomes different sizes of w sew-shot recognition tasks across 11 datasets are illustrated in Table A5, Additionally, Table A7 showcases the results using different norm types for the PUN loss, including 1-norm, 2-norm, and Inf-norm.
A.2.6 RESULTS DURING TRAINING PROCESS
This subsection provides results during soft-prompt tuning VLMs, including training loss, test curacy, norms soft prompts at various positions, and the occurrence frequency of Low-Norm Effect for different prompting positions. The data comparison between Figure A2 and Figure A3
15

---

Published as a conference paper at ICLR 2024
leads to several conclusions. Firstly, regardless of whether it is in the 1-shot 16-shot setting, the PUN loss demonstrates a stronger level of normalization compared to the PAN loss. On the other hand, despite the stronger normalization of the PUN loss, it does not outperform the PAN loss in the 16-shot setting. This implies that simply reducing the norm of soft prompts does not always lead to an improvement in model performance. Secondly, there is notable occurrence frequency of the Low-Norm Effect in the 16-shot setting during training process, indicating that an increased number of training samples can facilitate the identification of prompting positions that induce the Low-Norm Effect by the PAN loss. This observation may explain why the PAN loss performs better in the 16-shot setting but is inferior to the PUN loss in the 1-shot setting. Lastly, the addition of two normalization losses does not seem to have a substantial influence on the original cross-entropy loss, indicating that they do not contribute to model performance primarily by reducing the cross-entropy loss. Instead, their benefits are likely derived from harnessing the Low-Norm Effect..
A.2.7 COMPUTATION COST ANALYSIS
The data in Table A8 represents the average computation costs of multiple training batches, based on the benchmark of CoOp's 's training time. The numbers in parentheses indicate the corresponding running times in seconds. Firstly, we observe that incorporating the PUN or PAN loss increases computation costs compared to using CoOp alone. The PAN loss introduces an additional pre- inference step before each training batch, resulting in higher computational burden compared to the PUN loss. However, combining both losses does not significantly impact the running time compared to using the PAN loss alone. Furthermore, there is clear and consistent trend across all methods, including CoOp, PLOT,, and our two losses, where computation costs increase with the number of classes in the datasets. We speculate this is caused by the fact that CoOp-based methods optimize the similarity scores between text features of all categories and image features of a mini- batch. The increase of number of classes will result in significant increase in the gradients required for calculation. Lastly, it should be noted that the running time of the PAN loss and PLOT is almost comparable.
The data in Table A9 demonstrate that increasing the value of N leads to increased computational costs, particularly when dealing with large datasets, resulting longer computation time and higher memory consumption. This is because the model requires generating a greater number of corrupted textual features for prediction within each training batch. Although is generally observed that increasing N has positive impact on the model's performance, finding ways to mitigate the com- putational burden is a valuable area for future research.
A.2.8 OTHER APPLICABLE SCENARIOS ANALYSIS
While our proposed method primarily focuses on benchmarking soft prompt-tuning VLMs, it should be noted that the benefits of Nemesis may be not limited to it alone. We speculate that they can also be applied to other parameter-efficient tuning (PEFT) methods, such as visual prompt-tuning (Jia et al., 2022) and prefix-tuning (Li & Liang, 2021), as well as their various downstream task.
To verify this, we conducted preliminary experiments on few PEFT methods and their applicable scenarios, including visual prompt-tuning (Jia et al., 2022) for image classification and prefix-tuning (Li & Liang. 2021) for paraphrase classification. Given that the proposed PAN loss may involve designing specific corruption experiments and adopting distinct performance comparison metrics depending on different methods and tasks. To be more specific, we should adopt various perfor- mance metrics when identifying the positions that induce the Low-Norm Effect,, such classifica- tion accuracy for classification tasks, mAP (mean Average Precision) for object detection tasks, loU (Intersection over Union) for segmentation tasks, and BLEU (Bilingual. Evaluation Understudy) for text generation. We leave it for the future work. Here, we only incorporate the PUN loss into these methods and obtain the results, shown in Tables A10 and All.
Based on the presented results, the proposed PUN loss can enhance the performance in all conducted experiments. However, determining the ideal weight for the proposed loss requires further investi- gation. which is potential direction for future work. In summary, these preliminary results support our research prospects discussed in the Conelusion Section (i.e. Section 6), and we hope that our findings and approaches will provide new insights into PEFT methods and inspire future research in these fields.
16

---

Published as a conference paper at ICLR 2024
205
19.0
16.3
$3.0
25.3
140
-15
Pastion
(a) Replaced by (r; 0,0).
(b) Rescaling factor 0.001.
15.0
422
141
12
30
02:30
Fastion
(c) Replaced by r ~ N(;; 0, 0.001
(d) Rescaling factor 0.01.
55.00
$10
52.0
10
Pautbon
Peolaan
(c) Replaced by (rr;0, 0.01
Rescaling factor = 0.1.
165
$1.03
515
555
35
14
16.
16
12 10
(g) Replaced by r (r; 0.1
(h) Rescaling factor 0.5.
6.8
120
56.2
Nr.om. $140
300
545
15.5
10
Replaced by ((r;; 0,0.3).
Rescaling factor 2.
Figure A1: Comparison of the model's performance and norms of soft prompts before and after corrupting the prompt vector various positions. The blue and green bars denote the model's performance and the norms of soft prompts, respectively. The lines represent the benchmarks of without corruption. Take the ImageNet dataset under shot setting example.
17

---

Published as a conference paper at ICLR 2024
w Manh
Onty
Figure A2: The results the Caltech101 dataset at the 1-shot setting during the training process.
18

---

Published as a conference paper at ICLR 2024
20003
41
51
200
Figure A3: The results of the Caltech101 dataset the 16-shots setting during the training process.
19

---

Published as a conference paper at ICLR 2024
Table A2: Corruption experiments using the Replace operation: the detailed occurrence frequency with which the performance of corrupted prompts exceeds that of their original counterparts. The upward and downward arrows indicate an increase and decrease in norms, respectively.
Dataset
Variance
shot
2 shots
4 shots
8 shots
16 shots
Norms variation
0
0
0
0
0.001
0
0
0.01
ImageNet
0.5
0
3
0
0
0.001
4
0
0
0.01
0
Caltech101
0.1
O
0
0.5
O
12
0.001
12
0.01
12
5
3
OxfordPets
0.1
8
0
0.5
3
0
0.001
0.01
StanfordCars
0
0
0
0
0.001
0
0.01
Flowers 102
0.1
0.5
0
5
5
0.001
5
0.01
3
5
Food101
0.1
3
0
0.5
0
0
0.001
FGVCAireraft
0.5
0
0.001
0.01
SUN397
0.1
0.5
0
0.001
0
0.01
0
0
DTD
0.1
0
0
0.5
0.001
EuroSAT
0.1
0
0.5
0
0
0
0
0
0.001
0
0
0.01
0
UCFL01
0.1
0.5
20

---

Published as a conference paper at ICLR 2024
Table A3: Corruption experiments using the Rescale operation: the detailed occurrence frequency with which the performance corrupted prompts exceeds that of their original counterparts. The upward and downward arrows indicate increase and decrease in norms, respectively.
Dataset
Resclaing factor
shot
2 shots
4 shots
shots
16 shots
Norms variation
0.001
0
0
0.01
0
0
0
0.1
3
0
0
0
ImageNet
0.5
14
15
13
11
9
2.0
0
1
0
0.001
3
0
0
0.01
O
0
0.1
5
O
0
C1
0.5
10
8
11
4
2
2.0
2
0
0.001
12
8
6
0.01
10
13
7
6
0.1
10
13
12
6
OxfordPets
0.5
10
15
16
15
7
2.0
0.001
0
0.01
0
0
0.1
0
0
0
StanfordCars
0.5
16
5
0
0
2.0
0
0.001
0
0.01
0
0
0.1
0
0
Flowers
0.5
2.0
0.001
7
0.01
5
7
7
0.1
13
9
1
Food101
0.5
14
12
11
15
13
2.0
0
0
0.001
0
0.01
0
0
0.1
0
0
FGVCAireraft
0.5
0
2.0
0
0
0.001
0
0.01
0
0
0.1
0
SUN397
0.5
12
11
5
2.0
0
0.001
0.01
0
0
0.1
0
DTD
0.5
13
11
2
2.0
2
0.001
0
0
0.01
0
0
0.1
0
EuroSAT
0.5
5
2
2
2.0
2
0.001
0
0
0
0
0.01
00
0
0.1
0
UCF101
0.5
11
12
2
2.0
0
0
0
21

---

Published as a conference paper at ICLR 2024
Table A4: Comparison of CoCoOp and CoCoOp+Neemesis (ours) in the base-to-new generalization setting. Following CoCoOp, all methods implemented with VIT-B/16 backbone and evaluated with 16 shots. We report the performance on datasets, including the base classes (Base), new (H) of both.
classes (New), and the harmonic
(a) Average.
(b) ImageNet.
Base
New
H
Base
New
H
CoCoOp
80.8
72.6
76.5
CoCoOp
76.3
70.6
73.3
CoCoOP=-Nemsis (PUN,
80.7
72.6
76.4
CoCnOp+ Nemesis (PUN, =
76.2
70.7
73.3
CoCoop+Nemesis (PUN, w = 10)
80.4
73.2
76.6
CuCnOp+ Nemesis (PUN, = 10)
76.2
70.6
73.3
CoCoOp+*Nemest (PUN, 20)
73.3
JoCuOOp+Nementt (PUN,
73.3
(PAN, Ceemeps
80.4
72.6
76.3
CoCoOp++Nemest (PAN,
76.2
70.5
73.2
A.Coopp-Nemesis (PAN, 10)
80.5
73.3
76.7
CoCoOp+Nemmeii (PAN, 10)
76.2
70.5
73.3
CoCoOp=Nemesis (PAN, = 20)
80.3
73.9
77.0
CoCoOp++Nemesis (PAN, = 20)
76.2
70.5
73.2
(c) Caltech101.
(d) OxfordPets.
Basc
New
Base
New
CoCoOp
97.9
93.2
95.5
CoCnOp
95.0
97.6
96.3
&Nemesss (PUN,
93.6
CuCuOp+ Nemesis (PUN, w
95.2
96.5
CoCoOp+Nemmsis (PUN, 10)
97.7
93.7
95.6
CoCoOp+Neemsse (PUN,
95.0
97.6
96.3
CoCoOp+Nemesti (PUN, 20)
98.0
93.7
95.8
CoCoOp+-Nemesis (PUN, 20)
94.8
97.7
96.2
CoCoOp=Nemeste (PAN,
97.8
93.5
95.6
CoCoOp+NNemeais (PAN, == 1)
95.3
97.7
96.5
CoCoOp+ (PAN, 10)
97.8
93.9
95.8
CaCnOp++Nemesis (PAN, w 10)
94.8
97.8
96.3
CoCoOp+Nemeett (PAN, 20)
93.7
CoCoOp+ Nemesis (PAN, w 20)
96.2
(e) StanfordCars.
(f) Flowers 102.
Base
New
I
Base
New
H
CoCoOp
70.8
73.1
71.9
CoCoOp
95.2
68.9
79.9
CoCoOp+Neemesis (PUN,
71.0
73.3
72.1
CoCoOp+Nemmcis (PUN, =
95.0
70.8
81.1
CoCoOp=Nemesis (PUN, = 10)
70.2
73.6
71.9
CoCnOp+Nemeedi (PUN, = 10)
93.7
71.3
81.0
CoCoop+Nemesis (PUN, μ 20)
70.6
73.4
72.0
CnCnOpp+ Nemessis (PUN, = 20)
94.0
71.3
81.1
CoCoOp+Nemesis (PAN,
72.7
CoCoOp+ Nemesis (PAN,
81.8
CoCoOp NNemesss (PAN, 10)
70.2
73.5
72.1
CoCoOp+ Nemesis (PAN, 10)
95.2
71.8
81.8
CoCoOp+Nemesis (PAN. 20)
70.6
73.2
72.3
CoCoOp+Nemesst (PAN, 20)
93.5
71.5
81.0
gg) Food101.
(h) FGVCAircraft.
Base
New
H
Base
New
H
CoCoOp
90.5
91.3
90.9
CoCoOp
35.6
32.1
33.8
CoCoOp+Nemesia (PUN, +
90.6
91.5
91.0
CnCnOp+ Nemesis (PUN, =
35.7
32.7
34.1
CoCoOp (PUN, 10)
91.0
CoCoOp+ Nemesis (PUN,
34.7
CoCoOp+ (PUN, 20)
90.6
91.5
91.0
CoCoOp+ Nemesis (PUN,
35.8
34.8
35.3
CoCoOp+Nemesii (PAN, 1))
90.5
91.5
91.0
CoCoOp+Nenman (PAN, 1)
35.3
33.4
34.3
CoCoOp+ (PAN, = 10)
90.5
91.3
90.9
CuCaOp+ Nemesis (PAN, w 10)
35.5
32.3
33.8
CoCoOp Nemesis [PAN, 20)
91.5
91.0
CoCoOp+NNemesis Nemesis (PAN, w 20)
35.4
(i) SUN397.
(j) DTD.
New
H
New
H
CoCoOp
76.6
CuCoOp
64.8
CoCoOp+Nemesia (PUN,
79.2
75.9
77.5
CoCoOp+ Nemesis (PUN,
77.5
55.4
64.6
CoCoop+Nemesis (PUN, = 10)
79.2
76.7
77.9
CoCnOp+ Nemesis (PUN, w 10)
77.4
54.9
64.2
CoCoOp+Nemeee (PUN, w = 20)
79.4
77.0
78.2
CuCoOp+ Nemesis (PUN, w 20)
76.4
58.7
66.4
CoCoOp NNemssis (PAN,
79.3
76.7
78.0
CoCoOp+ Nemesis (PAN, w
C.CooOp+Nemmsis
77.3
65.3
CoCoOp+Nemesis (PAN, 10)
79.1
77.0
78.0
CoCoOp+Nemessi (PAN,
76.0
57.3
65.3
CoCoOp+Nemesis (PAN. =0)
79.2
76.7
77.9
CoCoOp+NNmmciis (PAN. 20)
76.6
56.7
65.2
(k) EAT.SAT.
(I) UCF101.
Basc
New
H
Base
New
H
CoCoOp
88.1
65.2
74.9
CoCnOp
82.5
73.8
77.9
CoCoOp+Nemesii (PUN, w 1)
87.4
64.1
74.0
CuCuOp+ Nemesis (PUN, w
81.9
72.6
77.0
CoCoOp+Nern #Nemesis (PUN, 10)
86.6
66.9
75.5
CoCoOp+ Nemesis (PUN, w
78.0
CoCoOp+Nemesia (PUN, 20)
87.5
64.0
73.9
CoCoOp+ (PUN.
81.9
77.8
Neeooes-Nemsss (PAN,
85.8
61.8
71.9
NemosisNNmmsis (PAN, ==1l
82.1
72.5
77.0
CoCoOp=Nemesis (PAN, 10)
88.0
68.6
77.1
CoCnOp+ Nemesis (PAN, = 10)
81.7
72.6
76.9
CoCooO+Nemesis (PAN, NAN, =0)
86.2
71.3
78.1
CaCnOp+ Nemessi (PAN, = 20)
81.6
74.5
77.9
22

---

Published as a conference paper at ICLR 2024
Table A5: Comparison of CoOp and CoOptNemesis (ours) in few-shot image recognition tasks, including 11 different datasets. All methods are implemented with ResNet-50 backbone.
Dataset
Methods
shot
shots
shots
shols
16 shots
CoOp
56.62
56.96
59,40
61.38
62.70
Nemesis (PUN,
57.
ImageNet
Nemesis (PUN.
56.16
58.00
61.14
62.30
62.96
Nomesis (PUN.
60.44
61.78
62.54
Nemesis (PAN, 2=0.1)
57.00
57.44
59.64
61.76
63.14
Nemesis (PAN, 2211)
56.70
57.10
59.54
61.78
63.28
58.52
CoOp
86.98
87.32
89.28
89.82
91.70
Nemesis (PUns,s (PU.,,
87.64
87.36
89,46
90.32
91.86
Caltech101
Nemesis (PUN..=1)
86.90
86.98
91.78
Nemesis (PUN.
85.76
87.92
90.36
91.28
91.42
Nomesis (PAN,
90.36
Nemesis (PAN, 2=1)
87.58
87.44
89.50
90.12
91.90
Nemesis (PAN, 22
87.32
87.06
88.92
90.16
91.70
CoOp
Nomesis (m0N,
86.28
83.
86.38
85
86.18
OxfordPets
Nemesis c2U),
Nomesis o=5))
86.22
87.18
82.68
85,82
83.76
86.08
Nemesis (PAN, 2=0.1)
86.28
83.16
86.38
85
86.18
Nomesis (PAN,
68
86.06
Nemesis (PAN, 4=30)
85.90
83.48
86.90
87.68
88.90
CoOp
55.62
58.44
62.84
67.78
72.80
(PUN, (.0.1)
StanforCars
Nomesis (PUN, wm)
56.16
58.08
63.772
68. 18.10
71.50
Nemesis (PUNN
56.42
59.62
63.34
67.08
70.46
Nemesis (PAN, am0.1)
56.20
68.
73.56
Nemesis (PAN .=1)
56.14
58.54
63.46
68.52
73.60
Nemesis (PAN,
58.08
67.
CoOp
67.48
76.92
85,28
90.84
94.30
Nemesis (PUN, o=0.1)
67.70
77.04
85,46
18
94.76
(=1),
Flowers102
Nomesis (PUN, w=10)
69.96
80.16
86.56
90.46
93.70
Nemesis (PAN,
67.76
77.44
85,86
91.50
94.58
Nemesis (PAN, a=1)
67.94
77. 12
94,90
Nemesis (PAN, (PAN)
68.44
78.86
18.116
91.92
94.46
CoOp
72.62
Nemesis (PUN,
74.02
72.84
74.12
71.68
74.32
Food101
Nemesis (PUN,
73.70
71.18
71.34
71.44
76.24
Nemesis (PUN, 5m)0)
Nomesis (PAN, 2=0.1)
74.10
72.98
74.34
72.06
74.48
Nemesis (PAN,
74.08
72.84
74.06
71.52
74.28
Nomesis (PAN, ==50)
72.10
71.74
78.38
79.36
CoOp
9.56
18.30
21.04
26.66
31.64
Nemesis (PON.
RGVCAirema
Nemesis (PUN,
10.02
19:40
21.76
27.04
30.92
Nemesis (PUN, x=10))
12.48
19.88
22.72
26.86
29.64
Nemesis (PAN,
Nemesis (PAN, ==1)
9.88
18.52
20.94
27 (0)
32.32
Nemesis
10.28
19.36
22.16
27.45 77.48
30.50
CoOp
60.12
59.96
62,70
65.28
68.82
60.12
60. 14
62,78
65
69.36
SUN397
Nemesis
Nomesis (PUN.
39.16
61.38
65.08
67 30
69.74
Nemesis (PAN 1=0.1)
60.32
59.96
63.06
65.74
69.42
(PAN,
Nemesis (PAN, 10)
58.78
$9.80
64.92
67.84
69.92
CoOp
44.16
46.24
$3,16
58
62.70
Nemesis (PUN,
44.60
46.28
$3.64
58.85
63.06
DTD
Nemesis (=UN)
Nomesis (PUN.
43.98
46.64
53,48
58.74 78.74
63,16
Nomesis (PAN, ==0.1)
43.96
46.94
$3.10
58. 96
63.06
Nemesis (PAN ==1)
43.76
46.64
53.54
59.00
63,26
(PAN,
CoOp
48.78
$9.90
67.54
77.32
83.24
Nemesis (PmN,
EuroAAT
Nomesis (PUN.
Nemesis (PUIN,
49.06
51.10
61.22
60.62
69.52
69.26
77.10
77.22
82.86
83.50
Nonesis (PAN, w=1)
50.66
61.24
67.68
777
83.52
Nemesis (PAN, 2=10)
50.62
61.70
67.94
77.60
83.36
CoOp
62.66
64.40
68.50
73.46
76.58
Nemesis Nemesis(PUIN, (PUNN,.-..)
63.38
65.00
69.26
73.28
76.58
UCF10)
Nemesis (PUN.
64.40
73.36
77.28
Nemesis (PENi, a=UNN)
60.88
64.66
69.04
72.74
75.80
Nomesis (PAN, w=0.1)
63.14
64.74
68.64
73.62
76.94
73.52
Nonesis (PAN, 62.44
64.60
68.76
74.04
77.02
23

---

Published as a conference paper at ICLR 2024
Table A6: Hyper-parameee analysis on threshold
Threshold r Caltech101 StanfordCars EuroSAT
DTD
Flowers102
UCF101
Average
0.1
89.29
63.91
67.79
53.32
83.35
69.52
85.44
0.5*
89.25
63.59
67.13
53.30
83.44
69.58
85,23
0.9
89.31
63.81
68.28
53.55
83.51
69.40
85.57
Table A7: Hyper-parameter analysis different types of p-norm.
Norm type Caltech101
StanfordCars
AuroSAT
DTD
Flowers 102
UCFI01
Average
2-norm*
89.25
63.59
67.13
53.30
83.44
69.58
85.26
I-norm 89.16
63.48
67.43
53.22
84.21
68.76
85.25
Inf-norm 89.33
63.95
67.21
53.23
83.39
69.52
85.33
Table A8: Analysis of computation costs, All comparison results are obtained by the average of multiple training batches and based on the benchmark of the training time of CoOp. For fair com- parison, we do not report the computation costs of PLOT on the ImageNet dataset as needs to reduce batch size for single GPU execution. The PAN loss adopts the default setting N = 1.
Datasets
Classes
CoOp
CoOp+PUN
CoOp+PAN
CoOp+PUN+PAN
PLOT
ImageNet
1000
1x (0.22)
2.27x (0.50)
4.00x (0.88)
4.05x (0.89)
SUN397
397
1x (0.11)
1.91 (0.21)
3.36x (0.37)
3.45x (0.38)
3.00 (0.33)
StanfordCars
196
1x (0.07)
1.57x (0.11)
2.86x (0.20)
2,86x (0.20)
2.57x (0.18)
1lowers102
102
1x (0.06)
1.17x (0.07)
2.00x (0.12)
2,00x (0.12)
1.83x (0.11)
UCF101
101
1x (0.06)
1.17x (0.07)
2.00x (0.12)
2.17x (0.13)
1.83x (0.11)
Food101
101
1x (0.06)
1.17x (0.07)
2.00x (0.12)
2.00x (0.12)
188x (0.11)
Caltech101
100
1x (0.06)
1.17x (0.07)
2.00x (0.12)
2.00x (0.12)
1.83x (0.11)
FGVCAircraft
100
1x (0.06)
1.17x (0.07)
2.00x (0.12)
2,00x (0.12)
1.83x (0.11)
DTD
47
1x (0.05)
1.00X (0.05)
1.60x (0.08)
1.60x (0.08)
1.40x (0.07)
OxfordPets
37
1x (0.05)
1.00x (0.05)
1,40x (0.07)
1.60x (0.08)
1.40x (0.07)
EuroSAT
10
1x (0.05)
1.00X (0.05)
1.20x (0.06)
1.20x (0.06)
1.00x (0.05)
Table A9: Analysis of computation costs about N. All comparison results obtained by the average of multiple training batches and based on the benchmark of the training time of CoOp.
Datasets
CoOp
CoOp+PAN(NEI)
CoOp+PAN.N=2)
CoOp+PAN.N=d)
CoOP+PAN(N=B)
CoOpePAN(V=11)
ImageNet
1x (0.22)
4.00x (0.88)
4,86x (1.07)
6.64x (1.46)
10.09x (2.22)
17.05x (3.75)
SUN397
1x (0.11)
3.36x (0.37)
4,09% (0.45)
5,36x (0.59)
8.18x (0.90)
13.64x (1.50)
StanfordCars
Ix (0.07)
2,86x (0.20)
3.43x (0.24)
4.43x (0.31)
6.57x (0.46)
10.71x (0.75)
Flowers102
1x (0.06)
2.00x (0.12)
2,50x (0.15)
3,00x (0.18)
4,43x (0.26)
6883 (0.41)
UCH101
1x (0.06)
2,00x (0.12)
2,33x (0.14)
3,00x (0.18)
4,43% (0.26)
6.83x (0.41)
00od101
Ix (0.06)
2,00x (0.12)
2.33x (0.14)
3,00x (0.18)
4,43% (0.26)
6.67x (0.40)
Caltech101
(0.06)
2.00x (0.12)
2.33x (0.14)
3.00x (0.18)
(0.25)
6.67x (0.40)
PGVCAircraft
Ix (0.06)
2,00x (0.12)
2.33x (0.14)
3,00x (0.18)
4.17% (0.25)
6.67x00.00
(0.05)
1.60x (0.08)
1,80 (0.09)
2.20x (0.11)
2,80x (0.14)
4.20x (0:21)
OxfordPets
1x (0.05)
1.40x (0.07)
1,80x (0.09)
1,80x (0.09)
2.40 (0.12)
3.40x (0.17)
EuroSAr
(0.05)
(0.06)
1.20x (0.06)
(0.06)
(0.07)
(0.09)
Table A10: Results from incorporating the PUN loss into visual prompt tuning for image classifica- tion. The experiment setting follows the VTAB Ik (Zhai et al., 2019) benchmark. w = 0 denotes the case without the PUN loss.
Datasets
w=0
w=0.001
w=0.01
w=0.1
W=1
w= 10
Sun397 (Natural)
49.86
49.76
50.04
48.89
46.67
45.07
EuroS AT' (Specialized)
91.89
91.30
92.00
92.81
91.63
91.31
DMLab (Structured)
33.54
33.39
33.75
34.85
35.75
35.50
Table A11: Results from incorporating the PUN loss into prefix-tuning for paraphrase classification. The experiment is conducted on the MRPC dataset (Dolan & Brockett, 2005). w = 0 denotes the case without the PUN loss.
Datasets
Performance Metrics
w=#
0,00000
w=0.001
w=0.01
0=
W=l
FI Score Accuracy
89.37
91.06
90.90
88.70
80.62
80.81
MRPC
85.10
88.00
88.00
85.10
69.62
70.90
24


Reviews:
[{"review_id": "w5zyKrQ9Qf", "reviewer": "Reviewer_KnC5", "summary": "The paper discussed the influence of soft-prompt to VLM, introduced REPLACE and RESCALE corruption affecting VLM, and proposed two normalization loss improving the performance of soft-prompt. The authors conducted a lot of experiments to confirm the effectiveness of method.", "questions": "1、The paper proposes two normalization methods, while only testing the effects of PEN and PAN on the experimental results respectively. Why cannot both types of losses be used simultaneously? If there is a contradiction between the two losses, it is recommended that the authors discuss the differences. If the two losses are similar, can the two losses be unified? If the two losses gain from different perspective, should relevant experiments be provided?\n2、Can author discuss application circumstance of two normalization methods? In practical applications, what kind of normalization loss should we choose for what situation? Suggest the authors to discuss.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1、The paper is the first study to discuss the influence of soft-prompt toward VLM.\n2、The paper conducted REPLACE and RESCALE to discuss the normalization of soft-prompt, and proposed Nemesis including two normalization losses to improve the effectiveness of soft-prompt.\n3、The paper has conducted a lot of experiments to prove the effectiveness of the method.", "weaknesses": "1、The writing of some parts of the paper are not clear enough. It is recommended that the authors check. For example, there is a discrepancy between formula 4 and the symbol definition in the previous paragraph.\n2、The two types of losses proposed in the paper lack a correlation with practical significance, suggesting authors discuss why the two forms of normalization affect soft prompt.\n3、The paper lacks discussion on the applicable scenarios of two normalization losses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmBGXmUuc2", "reviewer": "Reviewer_hcSQ", "summary": "The paper, at its core, explores the significant yet uncharted territory around the impact of norms of soft-prompt vectors on the performance of vision-language models (VLMs), like CLIP. The authors have brought to light a unique phenomenon termed the \"Low-Norm Effect\", highlighting how reducing norms of specific learned prompts can sometimes boost the performance of VLMs. The effect seems to be more prevalent in certain datasets like Imagenet, OxfordPets, and Food101 as compared to others. Interestingly, the Low-Norm Effect appears to have a stronger presence when there's limited training data, hinting at potential issues with soft-prompt methods under data constraints.\n\nTo harness this Low-Norm Effect, the paper proposes a method named \"Nemesis\". This approach introduces two techniques – Position Equality Normalization (PEN) loss and the more refined Position Awareness Normalization (PAN) loss. While the PEN loss aims to normalize the norms of all prompt vectors, the PAN loss is more discerning, identifying positions that might induce the Low-Norm Effect before selectively normalizing them. The authors suggest that this method can notably enhance VLM performance without incurring significant computational costs.", "questions": "Given the significance of the parameter $\\beta$ in differentiating between the two variants of the Nemesis method, why was an ablation study not conducted to evaluate its impact? Additionally, have you considered exploring decimal values for $\\beta$ to potentially strike a balance between the effects of the PEN and PAN losses?\n\nHow does the proposed Nemesis method compare with other soft-prompt tuning methods in terms of computational efficiency and scalability, especially in larger datasets or more complex tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper pioneers a systematic investigation into the role of soft-prompt vector norms in VLMs, addressing a previously unexplored research question.\n\n2. The proposed Nemesis method, with its innovative PEN and PAN losses, offers a potential solution to the Low-Norm Effect, showing promise for improving VLM performance.\n\n3. Extensive corruption experiments shed light on the Low-Norm Effect's impact, providing valuable insights for future soft-prompt tuning endeavors.", "weaknesses": "1. $\\beta$ can be either 0 or 1, corresponding to two variants of the proposed Nemesis method. However, there is no ablation study on the selection of $\\beta$, nor is there an exploration of the potential impact of setting $\\beta$ with decimal values to assign weights to the two methods.\n\n2. The paper introduces a pre-inference step before each training batch to identify positions inducing the Low-Norm Effect. Such a step could introduce computational overhead, especially with larger datasets or when rapid training iterations are required. The study hasn’t provided a detailed analysis of the computational cost or time implications this might have in different scenarios.\n\n3. The Position Equality Normalization (PEN) loss applies equal weight to the norms of soft prompts at all positions. While the paper does acknowledge that normalizing prompt vectors at positions unaffected by the Low-Norm Effect may not yield performance improvement, the inherent assumption of the universality of the Low-Norm Effect across positions may not hold true for all datasets or real-world scenarios. The approach could benefit from a more dynamic, adaptive mechanism.\n\n4. The paper utilizes the RESCALE operation with a specific rescaling factor, τ, described as a positive real number less than 1. However, there’s no mention of how the value of τ is determined, if it's consistent across datasets, or its sensitivity. The choice of τ could have implications on the effectiveness of the Nemesis method, and without clear insight into its selection, there’s potential variability in results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D7II3bm0HG", "reviewer": "Reviewer_bzG1", "summary": "This paper answers the question \"do we need to normalize the soft prompts in VLMs?\" by (1) uncovering a phenomenon called the low-norm effect and (2) proposing a new method named normalizing the soft-prompt vectors of vision-language models (Nemesis) to normalize soft-prompt vectors in VLMs. The contributions include, (1) new soft-prompt vector normalization method for VLMs (normalizing soft prompts during soft-prompt tuning), (2) better results when evaluated by domain generalization settings for VLMs.", "questions": "1. could there be a combination of between soft-prompt tuning and hard-prompt tuning? (hard = explicitly use some predefined words/phrases as part of the prompts);\n2. any idea of further combining existing PEFT (prompt tuning, prefix tuning, LoRA...) with your Nemesis method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) new soft-prompt vector normalization method for VLMs, which can be incorporated into any soft-prompt based methods;\n(2) better results when evaluated by domain generalization settings for VLMs.", "weaknesses": "1. prefer to learn more details of how you decide the length of soft prompt vectors, e.g., why 4 and 16, will there be more ranges to be investigated basing on the specificl tasks for VLMs?\n2. prefer to learn more investigations of combining Nemesis with existing PEFT algorithms to see if the results can be further improved or not so that other researchers can better leverage your method to their existing frameworks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)