Semantic Alignment for Prompt-Tuning in Vision Language Models
Authors: Hari Chandana Kuchibhotla
Sai Srinivas Kancheti
Abbavaram Gowtham Reddy
Vineeth N. Balasubramanian
Venue: Accepted by TMLR ()

Abstract: Going beyond mere fine-tuning of vision-language models (VLMs), learnable prompt tuning has emerged as a promising, resource-efficient alternative. Despite their potential, effectively learning prompts faces the following challenges: (i) training in a low-shot scenario results in overfitting, limiting adaptability, and yielding weaker performance on newer classes or datasets; (ii) prompt-tuning's efficacy heavily relies on the label space, with decreased performance in large class spaces, signaling potential gaps in bridging image and class concepts. In this work, we investigate whether better text semantics can help address these concerns. In particular, we introduce a prompt-tuning method that leverages class descriptions obtained from Large Language Models (LLMs). These class descriptions are used to bridge image and text modalities. Our approach constructs part-level description-guided image and text features, which are subsequently aligned to learn more generalizable prompts. Our comprehensive experiments conducted across 11 benchmark datasets show that our method outperforms established methods, demonstrating substantial improvements.

Published in Transactions on Machine Learning Research (12/2024)
Semantic Alignment for Prompt-Tuning in Vision Language Models
Hari Chandana Kuchibhotla
ai20resch! 1006@ttt..c..oo
Indian Institute of Technology Hyderobad, India
Sai Srinivas Kancheti
c>21resch0100 @ith ac. in
Indian Institute of Technology Hyderabed, India
Abbavaram Gowtham Reddy"
wuutham.abbuwarrrmnvvoo de
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Vineeth N Balasubramanian
vineethnb @iith ac in
Indian Institute of Technology Hyderabad, Indio
Reviewed on OpenReview: https: // openreview. forun? id= avDr560jji
Abstract
Going beyond mere fine-tuning of vision-language models (VLMs), learnable prompt tuning has emerged a promising, resource-effcciett alternative. Despite their potential, effectively learning prompts faces the following challenges: (i) training in low-shot scenario results in overfitting, limiting adaptability, and yielding weaker performance on newer classes or datasets; (ii) prompt-tuning's efficacy heavily relics on the label space, with decreased per- formance large class spaces, signaling potential gaps in bridging image and class concepts. In this work, we investigate whether better text semantics can help address these concerns. In particular, we introduce a prompt-tuning method that leverages class descriptions ob- tained from Large Language Models (LLMs). These class descriptions used to bridge image and text modalities. Our approach constructs part-level description-guider image and text features, which are subsequently aligned to learn more generalizable prompts. Our comprehensive experiments conducted across 11 benchmark datasets show that our method outperforms established methods, demonstrating substantial improvements.
1 Introduction
Foundational Vision-Tanguage Models (VLMs) like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have displayed remarkable zero-shot and open-vocabulary capabilities recent years. This has led to VLMs being employed in various vision-only downstream tasks such open-vocabulary image classification (Liang et al., 2022), object detection (Feng et al., 2022), and image segmentation (Lüddecke & Ecker, 2022). Trained extensive web data, these models often use contrastive loss to align image-text pairs in shared embedding space, allowing them to represent diverse concepts.
Recently, learnable prompt-tuning (Zhou et al., 2021; Khattak et al., 2023; Fahes et al., 2023) has emerged as a promising parameter-cffiiiet alternative for finc-tuning foundation models. Prompt-tuning methods introduce additional learnable parameters called prompl veclors, which are luned on task-specific data. This approach adapts VL.Ms for a specific downstream task without affecting the pre-trained parameters of the VLM. While prompt-tuning methods have shown great promise, efficiently learning prompt vectors faces the following challenges: (i) training prompts in a low-shot setting leads to overfitting, hindering their general- izability, and exhibiting sub-optimal performance when applied to newer classes or datasets (Shi & Yang, 2023; Khattak et al., 2023; 2022), (ii) the performance of prompt-tuning methods can be highly dependent on the label space used for classification. During inference, if the label space is large, the performance tends to
Equal contribution, Work done IIT Hyderabadd

---

Published in Transactions on Machine Learning Research (12/2024)
Image
CoOp
PromptSRC
SAP (Ours)
Image
CoOp
PromptSRC
SAP (Ours)
A photo of aaplyynng Npstick has person applying lipstick to lips'
A photo of clean and jerk which has 'a person lifting a barbell
Proposed GZS
Benchmark
W//e Navel HM
Base-to-Novel Benchmark
Accuracy (%)
73
74.2
76.1
76.2
27
76.
75.1
77.5
70
66.9
69.8
68.1
67
71
70.7
70.300.
65.2
65-
03
$1.9 62.1
032
60
55
55
ProGrad CLP KgCoOp CoOp CoCoOp
VPT
PSRC MaPLe
SAP
ProGrad CLP KgCoOp CoCp CoCoOp
VPT
PSRC
MaPLe
SAP
Figure 1: Top: Comparison of GradCAM (Selvaraju et al., 2017) visualizations for our proposed method SAP against other baselines, on classes "Applying Lipstick" and "Cleon and Jerk" from an Action Recognition dataset (Soomro et al., 2012). The saliency maps indicate image regions that are most relevant to the descriptions "A photo of applying lipstick has a person applying lipstick to lips" and "A photo of clean and jerk which has a person lifting a barbell" respectively. SAP effectively localizes the text semantics in images compared to baselines. Bottom: SAP surpasses other baselines on Generalized Zero-Shot (G7S) and Base-to-Novel (B2N) benchmarks, showing improvements of +1.6% and +1.2 on Novel Accuracy and Harmonic Mean (HM) for GZS, and +1.4% and +0.9 for B2N compared to best performing baselines.
decrease due to bias towards the seen classes the model was fine-tuned on (see empirical evidence in Tab. 2 of 5). These issues indicate that there is a lack of understanding of images and classes based on their detailed semantic components. For example, an image of a cat should be understood through its specific features like 'whiskers' and 'tail', not just the class name 'cat.' To address this, we propose SAP (Semantic Aligument for Prompt-tuning), which uses class descriptions to learn generalizable prompts. Our semantic alignment module brings together class descriptions for class names with the text modality and the image modality using cross-attention mechanism and learnable bias. This alignment helps the model grasp the relationship between different parts of an image and their textual descriptions, leading to more detailed and accurate representation.
Our method uses class descriptions to guide the creation of such image and text features that correspond to specific parts or aspects of a class. However, we obscrve that merely using class descriptions alone does not address the challenges presented above, as shown Tab. of S 5.2. We demonstrate that careful semantic alignment between image and text features is crucial for effectively leveraging class descriptions. Given a set of class descriptions, we show how to construct description-guiie for image of cat, and class description 'has large tail', the corresponding description-guidee image feature encodes the part-level visual semantic information related to the description, using cross-attention mechanism and a learnable bias. The rescription-guided text features constitute the class name along with the semantics of each class (obtained from the descriptions), resulting in information-rich and robust text features. We then compute semantic alignment as the average cosine similarity between dnscription-guiiee image and text features, for relevant descriptions. Note that our approach includes the alignment between both improved text and improved image features, making our approach multimodal. We use pre-trained Large Language Models (LLMs) to generate class descriptions in an inexpensive manner. A recent set of works (Menon & Vondrick, 2023; Yang et al., 2022) has shown that class descriptions obtained from LLMs can be naively used to classify images on given datasct with fixed categorics. We go beyond and leverage these class descriptions to perform low-resource prompt-tuning, and show that such adapled VLMs show better generalization to unseen, novel classes. Fig. 1 illustrates the effectiveness of SAP over other baselines on two benchmarks, Generalized Zero-Shot Classification (G7S) and Baso-to-Novel Classification (B2N), defined in $ As our semantic alignment levt-level, SAP showcases superior localization of visual concepts relevant to class description, as seen through class activation maps, when compared to other baselines.
image and text features. For instance,
2

---

Published in Transactions on Machine Learning Research (12/2024)
Text Prompix
Use External Krowlldge
Purt-levd ing-taet oligument
Evaluation Benchmarrke
Additonal Trainable
Mathod
Pronptt
Farsmnceers
CuOp UCV 22, CuCobp (CVPR 122), KgChOp (CVPR '88
X
BIN, Xihtssert
Protired (CDV (5]), ProDA "OVPR 22]
MaPLo CVPR 22), PSRC JOCV 22),, LoGoProngt (DCC 22]
/
x
X
B2N, XDatsert, DG
30k- 9.5531
KAPT pocy 23). CoPrumpt [[CLR '20), CLIP-VDT JICVWW 24
/
/
/
X
B2N, Xihasset, DG
1.5 4.7dM
(Ours)
/
4
V
628. XDstvvr,
S6K
Cluciistinn
Table 1:
Comparison of the proposed method, SAP, with other related work on various key aspects involving fine- tuning VLMs for better generalization. B2N: Base-to-Novel, XDataset: Cross Dataset, DG: Domain Generalization, G7S: Generalized Zero-Shot.
Tab. 1 delineates the key differentiators of our approach compared to other baselines. Most existing prompt- tuning methods do not use additional text semantics; even among the recent few that use such information, our method utilizes class descriptions at part-level for both image and text. This strategy leads to non- trivial performance enhancements across benchmark datasets and improved localizations in novel classes or datasets. Additionally, we highlight a gap in the evaluation scheme used in existing comppttuunig efforts, which demonstrate the performance of Iearned prompts primarily on the tasks of Baso-to-Novel classification and cross-dataset evaluation. Inspired by the traditional Generalized Zero-Shot Learning (G-ZSL) (Xian et al., 2017; Liu et al., 2023) paradigm, we posit that generalization in the zero-shot setting is more realistic when considering both base and novel classes at inference. We call this protocol GZS evaluation the first such effort among prompt-tuning methods. We also propose another benchmark Classification without Class-names - where the method is exclusively evaluated using class descriptions to classify images when its label lies outside CLIP's vocabulary. Our contributions are:
We propose a prompt-tuning method to fine-tune VLMs that can leverage class descriptions obtained from an LLM. Our semantic alignment module allows for integration of class descriptions obtained from class names with both text modality and image modalities using a cross-attention mechanism and learnable bias, thus bridging the two modalities. This improved alignment allows us to learn prompts that can generalize well to unseen classes and datasets.
We carry out a comprehensive suite of experiments with comparisons against state-of-theeart and very re- cent methods on eleven standard benchmark datasets. We outperform existing baselines with significant margin all evaluation protocols.
We propose two new evaluation protocols: GZS evaluation and Classification without Class-namm to bet- ter study the generalizability of prompt-tuning methods for VLMs. Our method consistently outperforms earlier baselines on these protocols, too.
2 Related Work
Vision-Language Models. Vision-language models (VLM) exhibit significant promise in acquiring generic visual representations. VLMs aim to harness natural language guidance for image representation learning and concurrently align both the text and image features within a shared embedding space. We consider encoder-only VLMs which comprise of three components: a text encoder, an image encoder, and a learning methodology that effectively utilizes information from both text and image modalities. Recent research on learning transferable visual representations delves into establishing semantic connections between text and visual elements, capitalizing on a vast reservoir of internet-basod imago-text pairs. For instance, CLIP (Rad- ford el al., 2021) is the product of contrastive learning from 400 million image-text pairs, while ALIGN (Jia et al., 2021) utilizes 1.8 billion noisy image-text pairs extracted from raw alt-text data. Nonetheless, sub- stantial challenge persists in transferring these foundational models to downstream tasks while preserving their initial generalization capabilities. To address this, we use auxiliary information in the form of class descriptions to better align image and text. features, enhancing the model's performance and generalizability.
Prompt- Tonitt-Tuning. Prompt-tuning introduces task-specific text tokens designed to be learnable to customize the pre-trained VLM for downstream tasks. Context Optimization (CoOp) (Zhou al., 2021) marks the pioneering effort in replacing manually crafted prompts with adaptable soft prompts, fine-tuned on labeled
3

---

Published in Transactions on Machine Learning Research (12/2024)
few-shot samples. Conditional Context Optimization (CoCoOp) (Zhon et al., 2022) builds upon this by generating image-specific contexts for each image and merging them with text-specific contexts for prompt- tuning. In contrast, Visual Prompt Tuning (Jia et al., 2022) introduces learnable prompts exclusively at the vision branch, resulting in sub-optimal performance for transferable downstream tasks. ProDA (Lu et al., 2022) focuses on learning the distribution of diverse prompls. KgCoOp (Yao el al., 2023) introduces regularization to reduce the discrepancy between learnable and handcrafted prompts, enhancing the gen- eralizability of learned prompts to unseen classes. PSRC (Khattak et al., 2023) shares a similar concept with KgCoOp (Yao et al., 2023) but introduces Gaussian prompt aggregation. ProGrad (Zhu et al., 2023) selectively modifies prompts based on gradient alignment with hard-coded prompt. MaPLe (Khattak et al., 2022) introduces prompts at text and image encoder branches and link them with a coupling function. In a different approach, LoGoPrompl (Shi & Yang, 2023) capitalizes on synthelic lext images as effective visual prompts, reformulating the classification problem into a min-max formulation. Although these methods have shown promising results, they suffer from overfitting to the training classes when trained in a low-shot manner. This overfitting limits their generalizability and results in sub-optimal performance on newer classes datasets. We address this issue by leveraging external information in the form of class descriptions to semantically align image and text features, helping us learn generalizable prompts.
Use of External Knowledge. A set of recent works (Menon & Vondrick, 2023; Yang et al., 2022; Pratt et al., 2022) provide evidence that visual recognition can be improved using concepts, and not just class names. However, (Menon & Vondrick, 2023; Pratt et al., 2022) does not facilitate a way to perform fine- tuning on a downstream dataset. In contrast, (Yang et al., 2022) is a concept bottleneck model with fixed label space and thus cannot be used for zero-shot classification. In fine-tuning methods incorporating external knowlodge, KAPT (Kan et al., 2023) introduces complementary prompts to simultancously capture category and context but lacks semantic alignment of each class description at the part-level of both image and text. On the other hand, CLIP-VDT (Maniparambil et al., 2023) utilizes semantic-rich class descriptions only in the text modality, without semantic alignment with images. In CoPrompt & Etemad, 2024), class descriptions are utilized via a regularizer acting as a consistency constraint to train the text prompts. There is no consideration of explicit semantic aligument with the image modality. In contrast, our approach utilizes class descriptions to semantically construct both text and image features, enhancing part-level alignment between the two modalities. This improved alignment helps us learn prompts that can generalize well to unseen classes and datasels. A comparison of our method with existing works is shown in Tab. 1.
3 Preliminaries and Background
VLMs perform image classification on a downstream dataset by comparing an image representation with text representations of the class names in the dataset's label space. When a small amount of labeled data is available, it has been shown that fine-tuning VLMs substantially boosts downstream performance (Zhou et al., 2021; 2022). However, the fine-tuned model does not generalize to novel classes that were absent during fine-tuning (Zhou et al., 2022). Tn this work, we propose Semantie Alignment. for Prompt Learning (SAP), that leverages class descriptions to fine-tune VLMs for better generalization to novel classes. Before we describe our methodology, we briefly discuss the required preliminaries, beginning with CLIP (Radford et al., 2021), the VLM chosen as our backbone following earlier work (Zhou et al., 2021; 2022; Khattak et al., 2022; Lu et al., 2022; Yao et al., 2023; Khattak et al., 2023; Zhu et al., 2023). A summary of notations and terminology is presented in Appendix $ A.
CLIP Preliminaries. CLIP consists of an image encoder 0 and a text encoder o, which are trained contrastively on paired image-text data to learn common multi-modal representation space. @ takes an image x as input and returns the image feature 0(x) E Rd. & processes text string S into a d-dimensional feature vector Φ(S) E Rdd CLIP is trained with InfoNCE loss (van den Oord et al., 2018) to enhance cosine similarity for matching image-text, pairs and to reduce it for non-matching pairs.
CLIP performs zero-shot recognition of an image x by choosing the most similar class name from a set of candidate class names y, i.c., predicted class & arg max sim(8(x), ((y)), where the similarity sim is cosine-similarity. In practice, for class name y, Φ(y) is the text representation of manually crafted prompt
4

---

Published in Transactions on Machine Learning Research (12/2024)
af buturrlly
Ttei promppss
Froner Transbiie
Descriptiosguiie test foutures
" phetn Pigron
" phetn
carnera
Text eaceder
Class description
descripdon
scOre
L ce
Isp coburisl wingg well patients
Desirtptionguibee fratures
antednne head
Inige
feee
fear plmpp oody
Iastund
fint
has a souap
has menoray card slot
preeegas
Image features
Local Image
guided image
features
Fedle
Fuwdd Imagge
Image encoder
Global image
edkigg wog.g.tlggg p
Lites
Fasion module
Figure 2: Our proposed workflow, SAP, performs part-based semantic alignment between image and text features. SAP integrates class descriptions into the text template which are passed through the text encoder to construct description-guide text features. Global and local image features are obtained from the im- age encoder. Description-guided image features are obtained by performing parameter free cross-attention between class descriptions and local features. These image features are pooled into a mean description- guided image fcature, which is then fused with the global image feature obtaín the fused image feature. Description -ruiide text features and the fused image feature contain part-level semantic information, and are semantically aligned. We optimize cross-entropy loss Lee, and two steering losses L Laterrs and L Ltter
encapsulating y such as 'a photo of [y] Zero-shot classification performance significantly depends the label set y considered, and varies with the template of the text prompt (Radford et al., 2021).
Fine-Tuning CLIP with Learnable Prompts. To perform efficient adaptation under limited supervision, prompt-tuning methods add small number of learnable tokens to the input token sequence of either modality which are fine-tuned to generate task-specific representations. For instance, CoOp (Zhou et al., 2021) adds n learnable text-prompts Pr {pí..... .... P',} to the token embeddings {wf.... w.y) } of some text S. The final sequence {pí..., Phy, wf,., w passed through Φ to obtain the prompled text feature (S)),, We follow IVLP (Rasheed el. al., 2022), which adds learnable prompt tokens al transformer layers of both image and text encoders. That is, along with text prompts, IVLP appends learnable visual prompts Pe to patch tokens of image x, which are passed through A to yicld the prompted visual feature (x(x). Let p [pt,Po} Po denote the set of all trainable lext and visual prompts. These prompls are trained to maximize the similarity between prompted image feature and the corresponding prompted text feature of its class label. Given B image-text pairs {(x, w)) where y/ E V, the likelihood of x; predicting class yi is given exp(sim(0, (x:) Øp (v:))/T
by (26 x:) Σ exp(sim (xi) Op (v))/T) where T is the lemperature and sim is cosine similarity. MEY
The negative ligelikelihood loss be optimized I(p) = B (E(R]
Σ log(P x:)).
With the above background, we now present our methodology to use class descriptions to learn prompts that helps VLMs generalize better to unseen, novel classes.
4 Semantic Alignment for Prompt-tuning: Methodology
Given labeled data, most existing methods learn prompts that largely limit themselves to incorporating text information in the form of class labels only. We propose SAP, Semantic Alignment for Prompt- tuning, which utilizes auxiliary information in the form of class descriptions obtained from LLMs to learn more generalizable prompts. Our method constructs description-guided image and lext features that are semantically aligned with each other. Specifically, class description provides semantic context, and the corresponding description-guide image or text feature encodes part-level information related to this
We add subscript p indicate prompted features images
5

---

Published in Transactions on Machine Learning Research (12/2024)
description. Our semantic alignment module allows for integration of class descriptions obtained from class names with both text. modality and image modalities using a cross-attention mechanism and a learnable bias. This external semantic knowledge, derived from class descriptions, transfers to novel classes because the semantics represent common concepts shared across multiple classes, such as 'large tail' or 'whiskers'. An overview of our methooolggy is shown Figure We begin by describing how class descriptions are generated using LLMs.
4.1 Generating Class Descriptions
Large language models (LLMs) act vast knowledge corpora that can be queried for the semantics of real- world objects. We use the popular LLM GPT-3.5 (Hagendorff et al., 2022) to obtain text descriptions for each class in a given dataset. Class descriptions commonly contain visual cues such as shape, lexture, and color, as well as narratives of objects commonly correlated with the class. To keep our method cost-efficient, we use descriptions that are class-specific but not image-specific, thus making them reusable for a set of image samples (note that this is done only once per class label). We use the responses from the LLM they are, and do not manually curate filter them any further. This keeps our approach low-cost, while integrating finer semantic details into fine-tuning of VLMs. Some examples of our class descriptions are provided in Appendix & E.
Class Description Features. For each class E V, where y is the label space under consideration, we denote by Ay the set of generated class descriptions. Let A = U Ay, N= denote the set of descriptions
yEy
of all classes and the size of the set, respectively. Class description features o(A) E R are obtained by passing the class descriptions through text-encoder ø. In the following sections, we describe how SAP leverages class descriptions to construct description-guider image and text features, enabling us to learn prompts that generalize well.
4.2 Leveraging Class Descriptions for Text Features
The text feature p(y) for class E y is generally obtained by encapsulating the class name in a text template, for eg. 'a photo of a [y] , and passing it through ø. When class descriptions A, are given, we append them to the text template to generate |Ayl distinct templates. For example for class = cat and Ag ("has whiskers', hhs a large lail'), we generate description-guided templates 'a photo of cat which has whiskers' and 'a photo of cat which has large tail'.
The description-guided templates features Φby; Ay) E RRl,, for class y. For image x, the semantic alignment between the image feature 0(x) and description-guider text features for class y is given by:
passed through text-encoder & to generate description-guided text
{(0(x), [Ayl
1
Σ aEA,
sim(0(x),
This simple way of incorporating class descriptions into the text modality works well in practice. We validate our design choices Tab. 6 by comparing against. alternative ways to incorporate class descriptions.
4.3 Leveraging Class Descriptions for Image Features
As shown in the fusion module of Fig. 2, we also leverage the class descriptions in the visual modality by first generating description guided image features, and then fusing them with the global image feature. We describe this process below.
Constructing Description-Guide Guided Image Features.
Final (n transformer black
proppt cle verloes talen
(x)
An image x is passed through the image encoder 0 (which is vision transformer in this section) and the output of the final Iransformer block of shape (1 + 196 + n) x d' is collected. Here, corresponds to the clsz token, n is the mumber of learnable prompt tokens, and d' is the dimension of the transformer layer. In all earlier works, including CLIP, the clsg output token is passed through the final projection layer proj IRd of 0 to
(x)
Figure 3: Addition of bias vector to the last transformer block in

---

Published in Transactions on Machine Learning Research (12/2024)
obtain the global image feature 0(x) E R These features capture the global context of the image but may not capture local object-level semantics (Rao et al., 2021). We aim to utilize the rich part-level local information hidden in the 196 patch tokens and establish their association with class descriptions. To obtain the local image features (x(x) E I 196xd we pass the patch tokens through proj and add a learnable d- dimensional bias offset as shown in Fig. 3. This bias added to fine-tune proj with local information, which otherwise is used only to obtain global image features from the last transformer block. The added learnable parameter, though small, effectively fine-tunes local features based on the descriptions, enhancing the quality of local image features.
We obtain description-guider image features by performing a parameter free cross-attention with class de- scription features as queries, and local image features as both keys and values.
(desc(x) Cross - ((A), K (x), 0x())
(2)
Here, ((4) E N are the class description features for all class descriptions A, ('(x) E 1996xd are the local features of an image The description-guided image features (dese(x) E RN xd encode part-level local information relevant to the N descriptions. For any description, the cross-altention module computes a weighted combination of the 196 local features, where the weights are determined by the similarity between the image patch and the description. Note that we obtain N description features, one per description, for a single image. Since descriptions are common across classes and even dalasets, these features contain information that can transfer to novel classes. Note that, similar to other works (Khattak et al., 2023; 2022), we only require a single forward pass to compute 0 and O Usually, only the cls token is used for image features. We leverage the remaining tokens (instead of discarding them) to compute local image features Oj, and consequently Odese-
Fusing Description-Guides Features with Global Image Feature. The description-guided image features described above use class descriptions from all classes, and not just the ground-truth class of the image. Since the class descriptions generated by LLMs may be noisy, not all descriptions are relevant to a specific image. To address this, we introduce a relevance score r E [0, 1],, which quantifies each description's 's similarity to the image. This is computed as:
(oftmax(((A) ((x))
(3)
We perform a weighted average of (dexe(x) with r, and obtain the mean description-guider feature (dese((x) E Rd, which captures finer contexts in an image and computed as:
(xe = gde))(x))
(4)
For an image, the global image feature ((x) Rd encodes class information pertaining to the image and the mean description-guided feature gdess(x) Rd encodes part-level visual context. We perform fusion of both these features to yield the final fused image feature (xx).
0xx) (1 1 a) 0(x) + a -ddec(
(5)
We give a higher weight a E [0, 1] to the part-level features (dese(x) of an image the descriptions attend strongly to specific patches of the image. For cach description, we consider the maximum attention weight over image patches as the specificity of the description. We then define a the average specificity for all descriptions. We note that a is not hyperparameter. This indicates the specificity of certain descriptions to some parts of the image. To see this, consider the case of background image. Clearly such image is uninformative w.r.t any class description, and its part-level features can be discounted. For each description, the maximum attention weight over image patches is a proxy for the specificity of the description. We then define a as the average specificity for all descriptions. The fused image feature 0(x) E IRd contains global visual semantics as well as part-level semantics.
4.4 Description-Guided Semantic Alignment
Given an image x, we obtain the fused image feature 0(x) as described in $ 4.3. For every class y E y, we obtain the description-guided text features ø(y; Ay) as described in S 4.2. We denote the learnable prompt
7

---

Published in Transactions on Machine Learning Research (12/2024)
vectors by p. and we represent prompted features with subscript p. For instance, the prompted fused image feature is (x(x), and so on. Prompts are trained by minimixing the negative log-likelihood of the training
data {(x,, 31) H/)
(p) B
1 Σ
log exp(E(0,(xx)))
A, Am)))))))))))))))))))))))))))))) Σ (xp(E(0,(xx)) (yl Ag))/T)
where ((0p(x))
1 Ay
sim (x) aEAs
Σ
yEy
(6)
where is the temperature parameter, and sim is cosine similarity. To compute semantic aligmment &. we aggregate similarity between the fused image feature and the descriplion-guided text feature over all pertinent class descriptions and normalize by their count. A relevant description in the image enhances its similarity to the class; however, the absence of a description in the image does not penalize its similarity to the class. Following (Yao et al., 2023; Khattak et al., 2023), we add regularization terms designed to penalize prompted features that deviate significantly from their unprompted counterparts. We use the L1 penalty to regularize global image features and description guided text features.
1 B
Lyter (p) = Σ 1=1
- ((x))112
L'teer (p) = |
Σ (1) - o(u; Ag)Ila DEY
(7)
B
The final objective is L(p) = Loe (p) + x1 A1 + A2 Atter (p), where A1 & A2 are hyperparameters.
Inference: Let y" be the inference time label space, and A, be the class descriptions of class & y" Using the learned prompt p. we compute the prompted fused image feature and the description-guider text features for all classes in )" The class with the highest semantic alignment E(0p(x'), Øp(;; (pp(2;AA)) Az)) is then predicted the final label. The overall algorithm of SAP is presented in Appendix $ B
5 Experiments and Results
We comprehensively evaluate the generalization performance of SAP on two newly proposed benchmarks - (i) Generalized Zero-Shot Classification (GZS) and (ii) Classification without Class-names (CwC) and existing benchmarks (iii) Base-to-Novel Generalization (B2N) and (iv) Dooss-D Generalization.
Proposed Evaluation Benchmarks:
(i) Generalized Zero-Shot Classification (GZS). In GZS, the label space of a dataset is equally split into disjoint base and novel classes. Only a small number (e.g., 16-shot) of labeled samples from the base classes are available as training data. However, during evaluation, the classification label space is the union of base and novel classes. As explained in $ 3, zero-shot classification performance depends on the label space considered, and introducing the union of base and novel classes into the label space tests the bias of the fine-tuned model Lowards base classes. Hence, we believe this benchmark is a more realistic measure of the generalization performance of VLM fine-tuning methods. Though this setting has existed in traditional zero-shot learning (Xian et al., 2017), introduce it back into the realm of VLM evaluation.
(ii) Classification without Class-names (CwC). VLMS require explicit class names to perform classifi- cation (Radford et al., 2021). This is a limitation for images whose label lies outside the VLM's vocabulary. CwC tests the ability of a VLM to classify truly novel images without explicitly using class names. During inference, all class names are replaced with the word 'object', and the model is tested on it's ability to classify an image based on descriptions alone. For example, to classify a 'Pikachu' image, we just use the descrip- tions ('has yellow body', "has round red cheeks') and not the class name 'Pikachu', hence the text template looks like 'a photo of an object, which has yellow body' etc. The model is fine-tuned on base classes, and evaluated base and novel classes separately by removing all class names.
Existing Evaluation Benchmarks:
(iii) Base-to-Novel Generalization (B2N). In this setting, following prior work (Zhou et al., 2021; 2022; Khattak et al., 2022; Yao et al., 2023; Khattak et al., 2023; Shi & Yang, 2023), the datasct is split into equal disjoint base and novel classes, and the model is fine-tuned on few-shot (16-shot) training split of the base classes. During evaluation, unlike G7S, the label space is constrained to either just the base classes or just
8

---

Published in Transactions on Machine Learning Research (12/2024)
Dataset
CLIP
CoOp
VPT
CoCoOp [CVPR (22)
MaPLe (CVPR (23)
KgCoOp (CVPR '25)
ProCred
PSRC (IOOV '23)
CLIP-VDT
SAP
[CMM) '21)
(IJCV 22)
(EOCV '22;
(1OCY '2)
(ICCVW (23
(Oums)
Avurage
BBaus
60.81
75.19
73.48
73.13
75.47
76.86
70.15 55.07
78.81
63.75
79.47
(+0.66)
on 11
eNowl
63.21
60.39
66.62
65.23
67.09
62.12
08.13
63.89
69.75
(1.62)
datuwots
AHM
61.99
66.99
69.89
68.96
71.04
68.71
61.70
73.08
63.82
74.29
(+1.21)
Table 2: Results on the GZS benchmark. gNovel & gBase indicate the accuracy of the novel classes and base classes respectively under the joint classification label space. gHM is the harmonic mean of gBase and gNovel. The best numbers are in bold, and the second best are underlined. SAP outperforms the best performing baseline on average gBase (by +0.66%), gNovel (by +1.62%), and gHl (by +1.21) computed across all datasets. Detailed dataset-wise results are presented in Appendix $ D.
the novel classes. The testing phase for B2N is thus separate for base and novel classes, whereas the GZS benchmark has a unifiod testing phasc.
(iv) Cross-Dataset Generalization. In this setting, the model is fine-tuned on ImageNet (Deng et al., 2009) and tested on the remaining datasets. This measures the ability of a VLM fine-tuning method to generalize to novel datasets.
Baselines. We compare SAP, against state-of-the-art baselines, including very recent prompt-tuning meth- ods (summarized in Tab. 1), such as CLIP (Radford et al., 2021), CoOp (Zhou et al., 2021), VPT (Jia et al., 2022), CoCoOp (Zhon et al., 2022), ProDA (Lu et al., 2022). MaPLe (Khattak et al., 2022), KgCoOp (Yao et al., 2023), ProGrad (Zhu et al., 2023), PSRC (Khattak et al., 2023) and LoGoPrompt (Shi & Yang, 2023). We also compare against contemporary works that use external knowledge, such as KAPT (Kan et al., 2023), CLIP-VDT (Maniparambil et al., 2023) and CoPrompt (Roy & Etemad, 2024).
Datasets. We follow (Zhou et al., 2021; 2022; Khattak et al,, 2022; 2023) to evaluate method on 11 image classification datascts of varying complexity. These datasets encompass diverse domains, including generic object datasets like ImageNet (Deng et al., 2009) and Caltech101 (Fei-Fei et al., 2004); fine-grained datasets like Stanford Cars (Krause et al., 2013), OxfordPets (Parkhi et al., 2012), Flowers102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAireraft (Maji et al., 2013); scene recognition dataset SUN397 (Xiao et al., 2010); action recognition dataset UCF101 (Soomro et al., 2012); texture datasct DTD (Cimpoi et al., 2013), and satellite image dataset FuroSAT (Helber et al., 2017).
Overview of Results. We present average base class accuracy, novel class accuracy, and their harmonic mean across 11 datasets for the GZS, CwC, B2N, and Cross-Dataset benchmarks in $ 5.1 - -Tab. 2, Fig. 4, Tab. 3, and Tab. 4 respectively. Dataset-wise expanded tables for all benchmarks, along with Domain Gen- eralization and ResNet-50 backbone results are present in Appendix $ D. In $ 5.2, we show class activation maps to visualize image regions most relevant to a class description, where SAP demonstrates better local- ization capabilities. We study the goodness of our design choices in $ 5.2 and show that part-level semantic alignment between image and text features helps learning better prompts.
5.1 Main Results
(i) Generalized Zero-Shot Classification. This newly proposed benchmark tests the ability of method towards it bias to base classes and also it's generalization to novel classes within a dataset. We compare SAP against baselines and report the results in Tab. 2. The metric gBase is the average accuracy of test images belonging to base classes when the label space is the set of all classes (union of base and novel classes). The metric govel is the average accuracy of test images belonging to novel classes when the label space the set of all classes. gHM is the harmonic mcan of the gBase and gNovel. SAP's ability to leverage descriptions helps in mitigating the bias towards base classes, resulting in good generalized novel class accuracy. We outperform recent state-of-the-art method PSRC, achieving better results in 8 out of 11 datasets (see Appendix & D), with a +1.21% margin in gHM averaged over all 11 datasets. Compared to the second-best method MaPLe, we have significant margin of +3.25% in average gHM, outperforming it on all 11 datasets. We don't report the results of PRoDA, LoGoPrompt, and KAPT in this setting due to code unavailability.
(ii) Classification without Class-names In this newly proposed benchmark, we study the ability of a pretrained VLM to classify images whose class names lie outside CT.IP's vocabulary. Since the list of datasets
9

---

Published in Transactions on Machine Learning Research (12/2024)
30.0
47.5
ProGrad
KgCoOp CoDp
CoCoOp
VPT
PSRC
MaPLe
SAP (Oums
CLIP
45.6 (+1.7)
Accuracy (96
45.0
43.3 (+1.7)
43.9
43.743..743.3
44.4
42.5
41.5
41.9 42
42.4
40.1 40.2 40.4
40.1 40.4
37.5
40.0
37.9
16.9
38.5
37.5
25.0
35
34.4
33.2
32.5
30.0
Base class performance
Novel class performance
Harmonic Mean (HM)
Figure 4: Comparison in the CwC setting. We show average Base, Novel, and HM accuracies over all datasets. During evaluation, descriptions of each class are provided instead of the class name, and visual recognition is conducted based on these descriptions. SAP outperforms baselines by average Base (by +1.75%), Novel (by +1.76%) and HM (by +2.04%) computed over all datasets. Detailed dataset-wise results are presented in Appendix & D.
Datawet
CLIP
CoOp
VPT
CaCoopp
ProDA
MaPLe
KgCnOp
Profirdd
PSRC
I.Prompt
CLI-VVDT
KAPT
SAP (Ours)
Average
Bove
69.34
82.69
80.81
81.56
62.28
80.73
82.48
84,26
84.47
82.48
(+0.21
on 11
Novel
74.22
63.
70.36
71.69
72.90
75.14
73.60
70.75
76. 10
74.24
74.50
72.24
77.51
(+1.4]
datosets
HM
71.70
71.66
70.36
75.83
76.65
78.55
77.00
76.16
79.97
79.03
78.28
76.41
80.94
(+0.97)
Table 3: Comparison on Base-to-Novel Generalization benchmark. The best numbers are in bold, and the second best are underlined. SAP outperforms the best performing baseline on average Base (by +0.21%), Novel (by +1.41%) and HM (by +0.97%) computed over all datasets. Expanded tables are in Appendix $ D.
CLIP was trained on is not public knowledge, to empirically evaluate this setting use the standard 11 datasets itself, but remove access to class-names during evaluation. Similar to the B2N setting, all models are trained on base-class images. For all baselines (including ours), we find the similarity of an image x with a class (not given to the model) as the average similarity between the image and the corresponding class-descriptions of y, which are known. We report average accuracies on 11 datasets in Fig. 4, where we outperform MaPLe (Khattak et al., 2022) by +2.04% in HM.
(iii) Base-to-Novel Generalization We compare our method with twelve baselines and report the average accuracies in Tab. 3, where we outperform all baselines. We report per dataset accuracies in the Appendix $ D, and show that SAP outperforms the stato-of-tho-art mothod PSRC in 7 out of 11 datasets whilc retaining performance in the others. We show significant gains in challenging datasets such as EuroSAT and DTD, where we perferrrm PSRC by a margin of +5.66% and +2.92% in HM respectively. We also show improved performance on the UCF-101 dataset, which contains a wide variety of human actions captured in diverse settings, where we show an improvement of 22.49% in HM over PSRC. These results indicate that SAP can integrate semantic knowledge provided by class descriptions to learn generalizable prompts.
(iv) Cross-Dataset Generalization. We compare our method with nine baselines and outperform all of them as shown in Tab. 4. SAP outperforms PSRC (Khattak et al., 2023) by +1% and MaPLe (Khattak et al., 2022) by +0.5% on average test accuracy over all datasets, which indicates that our method learns prompts that generalize across datasets.
Dataset
CoOp
CoCoOp
VPT
MaPLe
KgCoOp
ProGrad
PSRC CLIP-VDT
KAPT
SAP (Ours)
Avg. on 10 Datasets 63.88
65.74
63.42
66.30
65.49
57.36
65.81
53.98
61.50
66.85 (+0.55)
Table 4: Cross- Generalization. Models are trained on Imagenet and tested on the entire label space of new datasets without fine-tuning, SAP outperforms all baselines average (see Appendix D).
Comparison against recent method that uses external knowledge. In Tab. 5 we compare SAP against CoPrompt (Roy & Etemad, 2024) on the B2N benchmark. Co- Prompt is a recent work that uses class de- scriptions to tune prompts and adapters, with total of 4.74M additional parameters over
CoPrompt promppaeendggtter
CoPrompt prompte
SAP (Ours)
prompte
Average 11
Base
84.00
83, 40
84.68
1 1.28)
Novel
77.23
76.90
77.51
0.61
datusots
HM
80.48
80.02
80.94
0.92)
Table 5: B2N results comparison against CoPrompt. SAP outnum- bers the prompt-only version by margin on Base (by +1.28%), Novel (by +0.1%),, and HM (by 0.92%).
10

---

Published in Transactions on Machine Learning Research (12/2024)
Image
CLIP
CoOp
PromptSRC
SAP (Ours)
Image
CLIP
CoOp
PromptSRC
SAP (Ours)
A photo of body weight squats which has Jeessnn bending knees and hips
photo of 770-320' which has farge commercial alroraft size
photo of "biking' which has person riding a bloyste
A photo of 727-2000 which has two engines the wings
Figure 5: Images are highlighted regions of highest activation relevant to specific text phrases, as identified by their prompted image and lext encoders. Qualitatively, SAP localizes better than existing baselines.
CLIP. SAP outperforms CoPrompt by +0.46% average HM, despite only having 36K additional learnable parameters over CLIP. SAP outperforms a prompt-only version of CoPrompt, indicated by CoPrompt in Tab. 5, by +0.92% in average HM.
5.2 Additional Results and Ablation Studies
Class Activation Maps. We present Class Activation Maps (CAMs) for the ViT-backbone CLIP image encoder to show image regions that most correlate to a given text description. We visualize activations of the pre-final self-attention layer of the transformer that maximize the cosine similarity between an image and given text description. We present qualitative results in Fig. 5, where prompts learned by our method lead to better localizations. In & D, Tab. 17, we show quantitative results using an occlusion metric to mcasure the localization capabilities of our learned prompts.
Study on Design Choices. In this section we justify our design choice of computing semantic alignment as the average similarity between the fused image feature and various description-guidee teat features. Our key contribution not just integrating descriptions into prompt learning for VLMs, but how descriptions are integrated into both visual and text modalities. We consider three alternative ways to incorporate class descriptions and show that our methodology leads to the best results. For our first alternative, we show that taking the unnormalized mean of description-guiden text features to compute similarity leads to a drop in performance (SAP w/ mean text feature in Tab. 6). That is, computing semantic alignment as ((0,(x), (p Ay)) ())( TnyT Σ ))(y)a)), leads to drop in performance. This is in contrast to aEA, our design choice of taking the mean similarity, as shown in Eq. 1. Intuitively, descriptions of class that are not well represented in pre-trained CLIP result in description-guided features with a low norm because CLIP has not encountered such associations during training. Information related to such descriptions is lost when the duscriptiionguided features are simply averaged out, without normalization.
We also observe that simply appending all class descrip- tions at once to generate a single description-guider text feature also leads a drop performance (SAP w/ agg descriptions in Tab. 6). Finally we show that replacing our text modality construction with that used by CLIP- VDT (CLIP-VITT text SAP's Visual in Tab. 6) leads to significant drop in average HM. These experiments show that how we add class descriptions is important, and that our approach is different from recent approaches that uses external information. We show average HM results across all 11 datasets of other design choices in Tab.
Method
Avg HM
SAP
80.94
SAP w/ mean text feature
80.31
SAP w/ agg descriptions
79.17
CLIP-VDT Text SAP's Visual
78.63
Table 6:
Comparison with alternative design choices for incorporating class descriptions into the text modality.
Effect of Removing Class Descriptions Our method SAP incorporates class descriptions in both image and text modalities, as described in $ 4.2 & & 4.3. Here study the effect of removing description guidance from both modalities. To remove description guidance from the text modality, we just use the default class name template i.e. 'a photo of a [y]', without using any class description. We denote this baseline as SAP-TG. The results shown in Tab. 7 indicate that adding class descriptions to the text modality, as SAP does, results a gain of +1.53% in average HM and a gain of +%2.72 in avg Novel.
11

---

Published in Transactions on Machine Learning Research (12/2024)
To study the effect of removing class descriptions from the image modality, we construct baselines by removing the cross attention module. We first consider a bascline that uses just the global image feature ((x) instead of the fused feature and call this SAP w/ global. Then, we consider a base- line that naively combines global and local fea- tures (without incorporating class descriptions via. cross-attention) by averaging them and denote it by SAP w/ global & local. We first compute the mean of the 196 local features to obtain a single mean local feature, and then average this mean local feature with the global feature. Note that both baselines construct description guided text features. The results presented in Tab. 7 justify our design choice of incor- porating class descriptions into the image modality. Furthermore our method incorporates class descriptions into images through a fully non-parametrie cross-attention, and adds no computational overhead.
Method
Avg. Base Avg. Novel
Avg. HM
Elfect of Removing Class descriptions from the Text Modality
SAP TGG SAP
84.62 84.68
74.79
79.41
77.51
80.94
Effect of Removing Class Deseriptions from the Image Modality
SAP w/ global
84.56
77.04
80.63
SAP w/ global & local SAP
84.66
76.81
80,35
84.68
77.51
80.94
Table 7:
All results are on the B2N generalization bench- mark, and are average results over datasets.
Role of the Learnable Bias Term. In this section we demonstrate the role of Icarnable bias in SAP. We add the bias term to enhance the learming of local image features. Poor local features arise because CLIP was originally trained using only the CLS token. Initial attempts to use the patch tokens directly to compute the local features resulted in poor performance due to pretrained layer norms (as in Fig 3). Therefore, we opt to adjust the local features in a parameter-officient manner by adding learnable bias. The bias term is necessary for obtaining high-quality local features. However, the bias alone is not sufficient; it must operate alongside cross-attention. The bias refines local features, while cross-attention integrates descriptive information into the visual domain, using local features. We study the effect of the bias along with our cross attention module in Tab. 8. The Class Descriptions column in Tab. 8. indicates their usage in the text modality. The table highlights that directly augmenting global image features with bias fine-tuned local features without cross-attention (row 2) performs worse than using just the global image features (row 4). This indicates that the bias alone is insulficient, and best results are obtained when the bias fine-tuned local features are integrated with class descriptions using our aross-attention module (row 1). Additionally, the result in row 3 suggest that without lcarnable bias, the local features become uninformative, even when cross-attention is used. This indicates that the learnable bias plays a critical role in making the local features useful for semantic alignment. To further support the importance of the cross-attention module and bias together, we studied the performance drop when both the cross-attention and bias are removed across different architectures and settings. The results are shown in Tab. 9. The bias and cross-attention modules are complementary, and together they aid in incorporating class descriptions into the visual modality.
Class Descriptions
Bias for Local features
Cross Attention
Avg. Novel
Avg. HM
Srchitecturo-Setting Avg. Novel Avg. HM
ViT-B2N
-0.47
-0.31
/
77.51
80.94
RN50-B2N
-0.49
-0.35
/
x
76.81
80.55
ViT-G7S
-0.49
-0.53
X
V
75.72
79.9
RN50-G7S
-0.40
-0.31
X
X
77.04
80.63
Table 9: Drop in the performance when both cross-attention Table 8: Effect of learnable bias along with our and bias are removed from SAP across different architecture cross-attention module the B2N benchmark.
and settings.
6 Conclusions
Prompt learning has emerged as a valuable technique for fine-tuning VLMs for downstream tasks. However, existing methods encounter challenges such overfitting due to limited training data and difficulties handling larger label spaces during evaluation, resulting in bias towards seen classes. Additionally, these methods struggle when class labels are not present in the vocabulary. We study if better text semantics can improve prompt learning, and propose an approach, named SAP, that learns prompts which better generalize to novel classes. Our proposed approach highlights that careful part-level semantic alignment between image and text features is crucial to leverage additional semantic information. We showcase the efficacy of our approach across four benchmarks, demonstrating significant improvements. We hope this work inspires further exploration into leveraging class descriptions in VLMs.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Acknowledgements. Hari Chandana Kuchibhotla would like to thank MoE for the PMRF fellowship support. Sai Srinivas Kancheti would like to thank MoF. for the PMRF fellowship support, as well as Microsoft Research for the PhD Fellowship support. We thank the anonymous reviewers for their valuable feedback that improved the presentation of this paper.
References
Anthropic. Claude haiku. https://sww.r http://wwww..tr.i....cccc. coaps////////aaa/aaa///// 2024. Accessed: 2024-11-300
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101 mining discriminative components with random forests. In European Conference Computer Vision, 2014.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. 2014 TEEE Conference on Computer Vision and Pattern Recognition, pp. 3606 1600--613,, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEFE Conference on Computer Vision and Pattorn Rocognition, pp. 248-255, 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Mohammad Fahes, Tuan-Hung Vu, Andrei Bursuc, Patrick Pérez, and Raoul de Charette. Poda: Prompt- driven zero-shot domain adaptation. In ICCV, 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. 2004 Conference on Computer Vision and Pattern Recognition Workshop, pp. 178-178, 178, 2004.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open-vocabulary detection using uncurated images. In Proceedings of the European Conference on Computer Vision, 2022.
Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Machine intuition: Uncovering human-like intuitive decision-making in gpt-3.5, 12 2022.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
Patrick Helber, Benjamin Bischke, Andreas R. Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. TEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, 12:2217 2226, 2017.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp.
PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser Nam Lim. Visual prompt tuning. ArXiv, 12119, 2022.
Baoshuo Kan, Teng Wang, Wenpeng Lu, Xiantong Zhen, Weili Guan, and Feng Zheng. Knowledge-aware prompt tuning for generalizable vision-language models. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 15624 15634, 2023.
Muhammad Uzair Khattak, Hanoona Rasheod, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19113- 2022.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15190- 15200, October 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained cate- gorization. 2013 IEEE International Conference on Computer Vision Workshops, pp. 554 561, 2013.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, Learning to detect unseen object classes by between-class attribute transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 951-958. IEEE, 2009.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kumpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Péter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. 2023 TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7061 7070, 2022.
Man Liu, Feng Li, Chunjic Zhang, Yunchao Wei, Huihui Bai, and Yao Zhao. Progressive semantic-visual mutual adaption for generalized zero-shot learning. 2023 IEEE/ /CF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15337-15346, 2023.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5196 5205, 2022.
Timo Lüddecke and Alexander Ecker, Image segmentation using text and image prompts. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7086-7099 June 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- tine-grained visual classification of aircraft. ArXiv, abs/ 110//3355511, 2013.
Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel E. O'Connor. Enhancing clip with gpt-4: Harnessing visual descriptions prompts. 2023 IEEE/CVF International Conference Computer Vision Workshops (ICCVW), pp. 262-271, 2023.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. ICLR. 2023.
Maria-Elown Perii-EEenaaaa Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729, 2008.
AT Gpt-4o mini: Advaneing cost-efficient intelligence. https://openai. com/index/ 2024. Accessed: 2024-11-30.
gpt-4o- wtw-aadrwaacnngooos-s---aieeeeee
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. 2012 TEEE Conference on Computer Vision and Pattern Recognition, pp. 3498 3505, 2012.
Sarah Pratt, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. 2023 IEEE/ CVF International Conference on Computer Vision (ICCV), pp. 15645 15655, 2022. URL https://api www.nticscholese.e.... hrt.:/arr.rrg/ooppussd:2222222 2521111028
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Languago-guider dense prediction with context-aware prompting. 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 18061-18070 2021.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. 2023 TEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6545 6554, 2022.
Shuvendu Roy and Ali Etemad. Consistency-guideel prompt learning for vision-language models. In The Twolfth International Conference Learning Representations, 2024.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-ban localization. In 2017 TEEE International Conference Computer Vision (ICCV), pp. 618 626, 2017.
Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can be good visual prompts for vision- language models. In Proceedings of the LEEE/CVF International Conference on Computer Vision (ICCV), pp. 2932-2941, October 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. ArXiv, abs/1212.04pB 2012.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1800 Abs/1807.03777 2018.
C. Wah, Steve Branson, P. Welinder, P. Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. In California Institute of Technology, 2011.
Yonggin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning the good, the bad and the ugly. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5077-3086, 2017.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to 20o. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485 3492, 2010.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Danicl Jin, Chris Callison-Burch, and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19187-19197, 2022.
Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-lanuaage prompt tuning with knowledge-guided context optimization. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6757 6767, 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for viniun-language models. International Journal of Computer Vision, 130:2337 2348, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16795-16804, 2022.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15659-15669, October 2023.
15


Reviews:
[{"review_id": "y3IIrSdh5a", "reviewer": "Reviewer_uP8W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is generally well-written and well-motivated.\n- Improving the fine-tuning methods of VLMs is of practical importance.\n- The proposed SAP approach is conceptually sound.\n- The empirical results of the proposed SAP are very promising.\n\nWeaknesses:\n- The proposed SAP approach leverages an LLM to generate label descriptions, which seems to heavily rely on the LLM's capabilities. It is unclear how powerful an LLM is required. This appears to necessitate the LLM having very strong knowledge of basic physical world concepts. Additionally, the extra costs for label description generation are not clearly discussed.\n- Using auxiliary knowledge to enhance VLMs (or foundation models in general) seems to be a common practice, which diminishes the novelty of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Vision-Language Models (VLMs) perform image classification on a downstream dataset by comparing an image representation with text representations of the class names in the dataset’s label space. It has been shown that, when a small amount of labeled data is available, \n fine-tuning VLMs substantially boosts downstream performance. However, the fine-tuned model does not generalize to novel classes that were absent during fine-tuning. This work proposes Semantic Alignment for Prompt Learning (SAP). The key technique of SAP is that it leverages class descriptions to fine-tune VLMs for better generalization to novel classes. Empirical evaluations across various benchmarks and tasks demonstrate that SAP can effectively improve VLMs' performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Based on the \"Weaknesses\" section, it would be great if the authors could include:\n- SAP's performance under various LLMs generated label descriptions.\n- The costs (computation or API querying prices) for generating the label descriptions.", "broader_impact_concerns": "There is no concern on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9LblrCZjw1", "reviewer": "Reviewer_xQ13", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n* The proposed technique achieves strong performance on a number of benchmarks as compared to the baselines.\n* The paper combines a number of different techniques in finetuning VLM’s and achieves strong performance.\n\n### Weaknesses:\n\n* The major weakness of the paper is the role of \"semantical alignment”. IMO, the ablations do not convincingly back up the importance of this component.\n* More details of the proposed model can be added.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a strategy to perform soft prompt-tuning while finetuning a contrastive VLM. It has two major contributions:\n\n* Replace standard CLIP style prompts “a photo of a {}” with class descriptors as obtained with GPT-3.\n* Cross-attention module between the class descriptor and local image representations. \n\nThe paper evaluates the proposed technique on Cross-Dataset Generalization and Base-to-Novel Generalization. They propose two further tasks: Generalized Zero-Shot Classification and Out-of-Vocabulary Classification.\n\n* The difference between Generalized Zero-Shot classification and Base-to-Novel generalization is that the label space is shared between base and novel classes at inference time.\n* The Out-of-Vocabulary classification benchmark replaces the class names with descriptors at test time.\n\nQualitative analysis is given on GRAD-CAM based visualization. Ablations are provided decoupling the importance of various components in the model.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Crucial\n\n* The major claim of the paper is that “semantic alignment” between the class descriptors and the local image features is crucial to achieve good performance. However, Table 8 displays that the impact of removing the cross-attention module is minimal to the final performance (+0.3-0.4). The improvement that comes from just incorporating the learnable bias parameter (+1.0) is much higher than the cross-attention module. Thus, the paper may adjust this strong claim including the introduction, abstract and title and also emphasize the importance of this learnable bias.\n* Further details of how hyperparameters $\\lambda_1$, $\\lambda_2$ and the number of soft prompts are tuned may be reported. This could be contrasted with the different baseline methods.\n* How is $\\alpha$ chosen?\n* While finetuning the VLM, the class-descriptions of the ground-truth class are readily available. So, to compute $\\theta_{desc}$, one can just use the class-decriptions of the ground-truth class instead of using all class descriptions $A \\in R^{N \\times D}$. Using all class descriptions is quite noisy and also does not scale well with the number of classes. The paper may perform an experiment to ablate this design decision.\n* It seems the proposed technique requires two forward passes to compute $\\theta$ and $\\theta_{desc}$. The same applies to the text embeddings. This can be made clearer in Section 4.3\n* The Out-of-Vocabulary benchmark replaces class names with class descriptors and is tested on the same 11 datasets as the other benchmarks. Then the term Out-of-Vocabulary may be misleading, since the classes are actually “in-vocabulary”. It may make sense to just name this benchmark as a “Class Descriptor classification” or something similar.\n\n### Minor comments:\n* What does naively combine “global” and “local features” mean?\n* The learnable bias adds a fixed vector across all positions. So IMO this is global rather than local.\n* The paper may add the clip baseline to fig 5. It can be interesting to see if the clip model has this localization property but loses it during finetuning.", "broader_impact_concerns": "The paper performs VLM finetuning on popular benchmarks. Hence, there are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "L2RQMoAysz", "reviewer": "Reviewer_3NEX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The SAP method's focus on part-level semantic alignment between image and text features is interesting, demonstrating improved performance and localization capabilities.\n- The introduction of the Generalized Zero-Shot Classification and Out-of-Vocabulary Classification benchmarks are also useful.\n- The authors conduct extensive experiments across 11 benchmark datasets, proving the method's improvements to existing approaches.\n\nWeaknesses:\n- The method heavily relies on the quality of class descriptions generated by LLMs. Any noise or inaccuracies in these descriptions could impact performance. It is better to discuss the impacts of these noises or inaccuracies for responses from the LLMs.\n\n- The method involves multiple sophisticated steps such as cross-attention mechanisms and fusion of global and local features, which might be complex to implement and replicate. How about the efficiency of using the proposed SAP when compared with other previous methods?\n\n- The differences in performance across different settings in the Effect of Removing Class Descriptions are not significant. We observe only minor changes in the metrics for different settings. The paper may also need to provide some hints about tuning the temperature parameter, which may be quite important for the final performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduced a method for prompt-tuning in VLMs called Semantic Alignment for Prompt-tuning (SAP). The primary contribution is leveraging class descriptions generated by LLMs to enhance the semantic alignment between image and text modalities at a part level. This approach addresses overfitting in low-shot training scenarios and improves generalization to unseen classes and datasets. The authors propose new evaluation protocols, Generalized Zero-Shot Classification, and Out-of-Vocabulary Classification, demonstrating that SAP significantly outperforms existing state-of-the-art methods across multiple benchmarks. The method's efficiency and effectiveness are validated through comprehensive experiments on 11 benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please try to address the comments on weakness.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most reviewers are positive about the paper.

One Reviewer noted that Table 8 showed only a minimal impact of cross-attention (+0.3%), whereas the learnable bias had a larger effect (+1.0%) which weakens a bit the claim of the paper of the importance of semantic alignment. I suggest the authors to elaborate more on this point in the final version: I think one way to improve the paper is to pinpoint really what is the most important component in the new architecture. I think the authors might move the clarifications added in Sec D of the Appendix under "Role of the Learnable Bias Term" earlier in the main paper. Another source of confusion was the fact that the lowest performance removing both bias and cross-attention module in the Table provided in the rebuttal is 79.9 (thus a drop of 1.0%), while the overall approach outperforms the baselines by 1.4%. There have been some questions as to why the discrepancy (what justifies the +0.4% wrt the baselines?) which I hope the authors will elaborate on in the paper. Overall, provided the authors clarify these aspects, the paper extensively demonstrates the efficacy of the overall architecture.
Decision: Accept (poster)