Attribute-based Visual Reprogramming for Vision-Language Models
Authors: Chengyi Cai
Zesheng Ye
Lei Feng
Jianzhong Qi
Feng Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Visual Reprogramming
Model Reprogramming
Vision-Language Models
Image Classification
Primary area: foundation or frontier models, including LLMs

Abstract: *Visual reprogramming* (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose ***Attr**ibute-based **V**isual **R**eprogramming* (AttrVR) for CLIP, utilizing ***des**criptive **attr**ibutes* (DesAttrs) and ***dist**inctive **attr**ibutes* (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the $k$-nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models. Our code is available at https://github.com/tmlr-group/AttrVR.

Published as a conference paper at ICLR 2025
ATTRIBUTE-BASED VISUAL REPROGRAMMING FOR VISION-LANGUAGE MODELS
Chengyi Cai' Zesheng Ye
Lei Feng Jianzhong Qi' Feng Liu'
'The University of Melbourne
Sottheast University $Idealism Technology (Beijing) (chengyi. cail, zesheng. ye, jianzhong qi]@unimelb, www.edu..... au fengl m ml@gmaili com
1 fenggage gma 1........... com
ABSTRACT
Visual reprogramming (VR) reuses pre-trained vision models for downstream im- age classification tasks by adding trainable noise patterns to inputs. When applied to usioo-language models (c.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class la- bels are inserted into fixed text templates to guide the optimization of VR pat- terns. This label- approach, however, overlooks the rich information and diverse attribute- guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose Attribute- Visual Reprogramming (AttrVR) for CLIP, utilizing descriptive attributes (De- sAttrs) and distinctive atfributes (DistAttrs), which respectively represent com- mon and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the k-nearest DesAttrs and DistAttrs for each image sample, en- abling more dynamic and sample-specific optimization. Theoretically, Att is shown to reduce intra-class variance and increase inter-class separation. Empiri- cally, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet- based CLIP. The success of VR facilitates more effective integra- tion of VR from unimodal vision models into vision-language models. Our code is available at https: / h/gttttb.bb///////gg/g thub. com/ tml r- rrrgrouo/aaee /tttoptupttrrr. AVR.
INTRODUCTION
Recent studies (Xu et al.. 2024; Chen al., 2024; Wang et al., 2024) have demonstrated that down- stream tasks can be efficiently addressed by repurposing pre-trained models from data-rich domains. For repurposing pre-trained image classifiers with a fixed label space (e.g.. pre-trained ResNet (He et al., 2016), VIT (Dosovitskiy, 2020)), visual reprogramming (VR) (Cai et al., 2024b; Chen et al., 2023: Chen, 2024), also known as adversarial reprogramming (Tsai et al., 2020; Elsayed et al., 2019), is model-agnostic technique that adjusts the input space while preserving the original mod- els. VR (full problem setup detailed Appendix 1) trains additive noise patterns on images using downstream samples and their corresponding labels. Recently, VR has been extended to vision- language models (VLMs), such as CLIP (Radford et al., 2021), for downstream image classifica- tion. Existing implementations of VR for CLIP (Oh et al., 2023; Bahng al., 2022) also follow the pipeline of vision models (i.e, image classifiers), relying on cemplate-proppted ground-tren -rrth labels (e.g., 'This a photo of [label]') ) to train the noise patterns.
However, VLMs are intrinsically different from unimodal vision classifiers in their capability to align attribute descriptions with image embeddings. Using label-based VR methods might fail to fully leverage such capability. Besides, similar syntactic structures in template-promptte ground- truth labels imply approximate text embeddings, leading to misclassifications of samples. Fig- ure 1(a) shows the t-SNE (Van der Maaten & Hinton, 2008) embedding visualization results (the up- labels. Classes
per plot) of images with VR patterns (the lower plot) learned by "British Shorthair' and 'Russia Blue* from the OxfordPets (Parkhi et al., 2012) dataset are used as examples. Many samples are observed to have similar distances to the cluster centers of both classes,
"Correspondence Feng Liu (fengliu.ml @ gmail.com)

---

Published as a conference paper at ICLR 2025
T-SNE Visualization of Image Features After Visual Reprogranming By Prompted Labels
(b) T-SNE Visualization of Descriptive Attribute (DesAtrr) and Distinetive Attribute (DistAtt)) Emibeddings
Average Cosine Dissance (1 Cosine Similarity) Between
Labels Altribute Embeddinggs
Deakitr 0.05
DecAtr
0.0k
0.10
0
n
Britah Shorbbir
Russion Blue
Firrin Ancs (Beiiish Sherthair)
Samples (Russian Blue)
Fitting Area (Briiish Sherthair) Fitting Area (Russian Bluc)
DesAttr (Russian Blue)
Fitting (Russiat Bluc)
(Bitish Shoethais)
DarAmi (Briish Shorthair)
Samplos (Uritih Sborthas?
Center (Russian Bluc)
DesAltr (Briish Shorlaair)
DistAttr (Russian Bluc)
Examples
Examples
An Example of
An Example of Russian Blue Visual Reprugramming Patrems
British Shorthirr
The Briid) Shortair a medun-itzd cat with a and musoal buil
The Britch Sheether hor a braad and strdyy hulli, with round and cholls
Visual Reprogramming Pattems
The Ruoiie likee b-s malim vocal eat with loa mastiir bady
The Riasón Sher has. dhimeeiiv cont short, denec, and plaak bie-ggray for
Figure I: T-SNE visualization results of (a) embeddings of images with label-based (i.c., 'This photo of [labelJ') VR and (b) embeddings of text DesAttrs and DistAttrs for classes 'British Shorthair' and 'Russia Blue'. Examples of images with VR attributes are shown below. Misclas- sifications occur in images with label-based VR, whereas attributes are easily distinguishable.
making them prone to misclassification. contrast, Figure 1(b) shows the text embeddings visu- alization results of attributes generated by large language model (LLM) GPT-3.5 (Brown, 2020) given the class name 'British Shorthair' and 'Russia Blue', respectively. The text embeddings of attributes exhibit greater distinguishability compared the embeddings of images with label-based approaches. Further, we use Descriptive attributes (DesAttrs) marked with !.' to denote the com- mon characteristics of certain classes (with examples shown in the figure), and distinctive attributes (DistAttrs) marked with '+' to describe features that differentiate the class from others or exhibit in- dividual differences. centers, while DistAttrs tend to be located away from the non-intenn inteeneedddd classes, e.g., DistAttrs of the red cluster are away from the blue cluster, especially the bottom left ones. The same observed at the top-right corner of the blue cluster. This is also confirmed by the cosine distance: Eembeddings of DistAttrs are farther away from the prompted labels compared with DesAttrs (Figure 1(b)).
important observation is that DesAttrs primarily concentrate near the cluster
Such observations suggest that guiding VR training with DesAttrs and DistAttrs could improve classification accuracy compared with cemplate-promphr ground-truie truth labels. In Section 4, we for- malize DesAttrs and DistAttrs (Definitions 2 and 3) and propose Attribute-baeed Visual Reprogram- ming (Att which harnesses the qurbbute-queriing ability of LLMs to describe DesAttrs and DistAttrs, capturing multiple common and unique features for each downstream class. Moreover, as images of the same class may reflect different attributes with evolving VR patterns, AttrVR queries the k-nearest DesAttrs and DistAttrs for individual image samples at each training epoch. By itera- tively updating the VR patterns with sample-specife attributes, AttrVR fosters more context-aware image-attribute alignment and mitigates the ambiguity caused by fixed template-prompter labels.
In Section 5, we further establish that guiding the representation learning with DesAttrs and DistAt- trs reduces intra-class variation and increases inter-class separation of image representations. This yields more discriminative embedding space, thereby facilitating classification performance.
Experiments conducted on 12 widely-used benchmarks demonstrate the effectiveness of AttrVR in Section 6. AttrVR consistently outperforms other VR methods when using different encoder backbones or fewer training samples. Visualizations of the embedding space and individual sam- ples with their top-matched attributes also substantiate the efficacy of Att. Additional ablation, hyper-parameter (see Section 6) and aggregation studies (see Appendix C.3) further examine the contributions of different components within Att.
Overall, both theoretical analysis and experimental results demonstrate that Att has a clear ad- vantage over label-based VR when applying CLIP to downstream classification tasks. The introduc- tion of AR represents a meaningful step towards adapting VR from repurposing single-modal pre-trained models with predefined label space multimodal models (i.e., VLMs) for classification.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Prompting in Classification. Prompt learning (Jia et al., 2022; Bahng et al., 2022; Oh et al., 2023) enables efficient adaptation of large pre-trained models to specific downstream tasks without fully finetuning the original models. Prompts can be trainable parameters integrated into different regions of pre-trained models. For vision models like ViT (Dosovitskiy, 2020), VPT (Jia et al., 2022) incor- porates prompts in conjunction with the embeddings of input patches in each layer. EEVPT (Han et al., 2023) and TransHP (Wang al., 2023) improve VPT by adding prompts within self-attention layers or learning prompt tokens for encoding coarse image categories.
For VLMS such as CLIP (Radford al., 2021), prompt learning methods are typically based on few-shot samples from downstream tasks. CoOP (Zhou et al., 2022b) optimizes the text prompts focusing on the text encoder part of CLIP, while CoCoOP (Zhou et al., 2022a) further improves by conditioning text prompts on input images. Beyond text prompts, MaPLe (Khattak et al., 2023) develops layer-specific mapping functions to connect visual and text prompts. PromptKD (Li al., 2024b) appends a projection to the image encoder and employs knowledge distillation to train the prompts.
Model Reprogramming and Input VR. In contrast to prompting methods that introduce param- eters within the model, model reprogramming (Chen, 2024) modifies the input and output spaces of downstream tasks. Therefore, it does not require meticulous design of parameter placement and is compatible with any model architecture. It has been applied in repurposing language (Ham- bardzumyan et al., 2021; Vinod et al., 2020), graph (Jing et al., 2023), vision (Tsai et al., 2020; Chen et al., 2023; Cai et al., 2024a) and acoustic models (Yang et al., 2021; 2023a; Hung et al., 2023).
Input VR refers to methods that add trainable noise patterns to images repurpose pre- pre-trained mod- els, being model-agnostic and preserving the original model parameters. The differences between input VR, visual prompting and finetuning are outlined in Appendix A. Recent work on unimodal vision classifiers adds trainable noise that overlays resized images (Cai et al., 2024b) or pads around (Elsayed et al., 2019; Tsai et al., 2020; Chen et al., 2023) images, and then optimizes noise patterns using ground-truth labels. When applying VR to VLMs (Chen et al., 2023; Bahng et al., 2022; Oh et al., 2023), prmpppee-rrompte ground-truth labels are used to train the noise patterns.
Visual Attribute Query. Visual Attribute Query (Pratt et al., 2023) refers to querying an LLM to obtain the corresponding visual features given downstream task labels. Current studies improve the zero-shot generalization performance (Pratt et al., 2023; Menon & Vondrick, 2023; Li et al., 2024a) and machine learning interpretability (Yang et al., 2023b; Yan et al., 2023) for VLMM.
Pratt et al. (2023) and Menon & Vondrick (2023) utilize GPT-3 (Brown, 2020) to generate descrip- tions of downstream task labels, thereby enhancing zero-shot classification accuracy. LaBo (Yang et al., 2023b) extends this approach by generating thousands of candidate concepts and constructing a class-concept weight matrix. To address the impact of redundancy in attribute descriptions, Yan et al. (2023) learn concise set of attributes, Tian et al. (2024) introduce attribute sampling and proposes class-agnowd -lass-agnostic negative prompts, while WCA (Li et al., 2024a) calculates the similarity between descriptions and local visual regions.
3 PRELIMINARIES
Clsed-based Classification. CLIP (Radford al., 2021) is a pre-trained VLM with an image en- coder fing S + Z and text encoder fixt : V + Z, where X C R is ds-dimensional image space, and V is the text space. These encoders map image X E 2 and a text description V E V. into shared embedding space Z C IRdd Then, the embedding similarity score between the image and the text description is calculated as
simcur(($ V) = cos (Zimg, Zext) /r, with Zimg = fing fXm)((x)) and Zixe = (V),
(1)
where cos(-.) -) denotes cosine similarity and 7 is temperature parameter. Upon pre-training, CLIP can align semantically similar image-text pairs by maximizing their embedding similarity scores.
When it comes to downstream classification task defined over XT, x y,,, where XT C Rdr is dr-dimensional image space and y is the label space of the downstream task, CLIP em- ploys label prompting. For example, given the downstream label variable yT E YT, TP(TT) 2
3

---

Published as a conference paper at ICLR 2025
"This is a photo of" |YT, where || denotes concatenation, is commonly used to map label y" y² into a text description. Following, CLIP leverages the pre-trained visual-text alignment ca- pability and assigns a label to X by selecting the most similar TP = y²) in the embed- ding space Z. Then, for a shape-compatible image x², i.e. dr = ds. label prediction follows arg max PCI.P | with a normalized conditional probability:
exp (simouip (2"", IP()"))
PC v x = =x=) ==²)= Eyeyt exp (simctap(
Input VR for CLIP-based Classification. Input VR (Cai et al., 2024b) extends the applicability of frozen pre-trained models (e.g., CLIP) to downstream tasks with mismatched input shape, i.e., dT * dg. It introduces a learnable input transform fia He + Id defined as fin fim(X11p (x*) & P P) + 5 @ M, where Pad(-) zero-pads around the input image and 5 E R are trainable parameters. M is a binary mask with 'O's in the area where X is located and 'I's in the padding area. The Hadamard product @ ensures that d only affects the padded regions. Thus, the transformed image is given by:
X fin fin(X=11) ER"
(3)
which allows CLIP to process inputs from X by embedding them in 2 with learned contextual information. With (Y that maps class label to a text description, the VR-adapted CLIP pre- diction Per Per(Y (YTITT x exp(simcir((X? (()) essentially follows Eq. (2) but is adapted for transformed input images. Given a downstream dataset DT = {(xT yi ) i.. ~ X x y", where N = n x | with n samples per class, the optimization of VR pattern is driven by the cross-entropy loss, such that
8" = -rg
1 En",, [oT = vi -= N
= =f)] =
(4)
Limitation of TP Eq. (4) implies that the optimization of 5 exclusively relies on TP for supervision. However, two labels yp. Ug y share similar syntactic structures in their fixed template-based TP(YTT siMCLIP (α"", (y,))) and SiMCLIP (T"", TP(Y))), may differ only slightly for the same input a². This marginal gap in similarity scores heightens the misclassification risks, particularly in few-shot settings, where the small sample size exacerbates the challenge of resolving ambiguities between closely related text prompts. Figure I(a) illustrates this concern, highlighting the potential classifi- cation errors due to the limited discriminative power of the text prompts.
yp) and TP(Y*T Yq), the image-class embedding similarity scores,
4 ATTRIBUTE-BASED VISUAL REPROGRAMMING
Describing Classes with Attributes. The aforementioned limitation calls for more informative and discriminative information of label beyond TP(Y"). Motivated by Figure 1(b), leverage visual attributes (Ferrari & Zisserman, 2007) to capture more fine- fine-grinedd visual features specific to each class than label-only representations. We thus propose uttribute-based VR (AttVRR that substitutes template-prompte ground-truth labels with descriptive and distinctive attributes, based on CL.IP's pre-trained visual-text alignment capability. AR aligns image representations with fine-grained attributes- based representations of classes that more effectively distinguish different classes than template-prompted labels. We begin by formalizing relevant concepts used in ARr Definition 1 (Attributes). Let X be the input space of images, y be the set of class labels, and A be the universal set of all possible attributes (e.g., tall plant, red color). Define a mapping fm from a class label y to a set of attributes A(y)- For each attribute E A, define an indicator function identified in input x based on a specified
fa : X > {0, 1}, such that fa(x) = 1 if attribute similarity criterion', and 0 otherwise. Then, for any class y E y, the set of attributes Aly) is connected to images by the features that characterize samples belonging to that class.
To further characterize the attributes most relevant for class description and distinction, we introduce two subsets from A(y), namely descriptive attributes (DesAuts) and distinctive attributes (DistAt- trs). DesAttrs refer to the most common visual features across multiple samples belonging to the same class, deseribing the class by capturing its general characteristics.
'The criterion can vary hased different contexts (Kumar et al., 2011; Pham al., 2021). In this study, we focus on CL.IP P similarity in the embedding space Z induced by VLM, which will he elaborated on in Section 4.
4

---

Published as a conference paper at ICLR 2025
(e) Laket hased Visusl Reprogramming
Epoch i
Epocht
attrbbteebusse Visual Reprogramming
Egoch
Epoch
Cosine Similarity
Cosine Similarity
Image
Brocolr
Inooge
Image
Image
Lugut
Input V.R.
Text Prompt
Destriptive Attribbues
Britah Shoetanirs
This is
brodd
hasa rocnd Saci
Test
Class
[Bettih: Shorthoiri
Distinctive
British Shorthorrs hove full ctooko
Labels
Attributes
CLIP
CLIP
Figure 2: The comparison of (a) previous label-based VR and (b) our surribute-based VR. Previous VR methods use fixed template-prompted ground-truth labels for all samples to optimize the VR pattern å (using Eq. (2) and Eq. (3)), whereas our method iteratively selects k nearest Attrutrs and DistAttrs for individual samples in each epoch to optimize the VR pattern ö (using Eq. (9)).
Definition 2 (Descriptive Attributes), For a class y E Y and set cardinality m E N4, DesAttrs are defined as the mn most frequently identified attributes from the samples within class y,
Adnaly) 2 {a: E A(y) U2((s)) 2 Uy(a), E (L....,m), m.. Va E A(y) (ai.....a:]]
(S)
where X, denotes the set of all images in class y, and E [a(x)/|X,) | the frequency of attribute E A in class y E V. Uy(a) can be used to rank the m
attributes.
In contrast, DistAttrs visual features that distinguish a class from other classes, appearing in the class while being the least common in the other classes.
Definition 3 (Distinctive Attributes). For class y and set cardinality E N+, DistAttrs are defined as the altributes that are most uniquely associated with class y.
Anaal (v) {uc E A(y) Vylli) 2 Vyla), Vi E (L....,m}, Va E A(y) [ai......i)..
(6)
where (4((a) = EV EXEX, fa(x)/((211xx,)) calculates the presence of an attribute E A in samples of class E against its presence in all other classes y' EY (v).
Intuitively, describing (d) U (y) leads to more information of y than relying on TP()).
Method Overview. For downstream image classification, ARt follows the general input VR pipeline by optimizing the padded VR noise pattern 8 over a dataset DT = {(x; 2/ )) drawn from XT, x yT, as introduced in Section 3. Yet, it diverges from label-based VR approaches in two key strategies (see Figure 2). First, for each label y" E y/T, AttrVR replaces previously used text prompts TP TP(Y'T = )²²), adopts DesAttrs (Definition 2) and DistAttrs (Definition 3) that describe common and unique attributes of y as the supervision signal. Second, ARt employs a k-nearest neighbor iterative updating strategy to ensure that attribute assignments are continuously refined, allowing the most relevant attributes for each sample adapt dynamically as the trainable noise 6 evolves across epochs. The detailed strategies are elaborated on below.
Generating Attributes with LLMs. The concept of attributes (Definition 1) is built upon f. that maps class labels to subsets of A. However, fm is intractable due to the exponential growth of the possible number of attributes, making direct computation and storage of all possible attribute combinations impractical. Moreover, manually defining attributes for each class is also infeasible in complex domains where attributes may not be easily enumerated or predefined. To this end, use powerful LLMs with visual attribute query capabilities (Pratt et al., 2023), denoted by fLLM(Y as tractable surrogate for implementing fm (Y LLMs can infer relevant attributes for any class with context-driven queries, bypassing the need to compute the entire power set of A - they generate Ades (YT y and Adisa (Y y²) according to different downstream tasks and class labels.
Concretely, we adopt GPT-3.55 (Brown, 2020) to generate (vT and Adis(( each containing m attributes, by prompting the LLM with task-specific and class-specific queries, formulated as
Aua (v = fiAM(y) (idespproppp) 1), Adia (y = fy [dist (iiistprroppl))
(7)
As a result, we collect 2m attributes for each class y E yT, which will be used for optimizing 6. The details of attribute generation (prompts, settings, etc.) are in Appendix A.2.1.
5

---

Published as a conference paper at ICLR 2025
k-nearest Iterative Updating Strategy. Recall that CLIP-based image classification is performed upon ranking the image-text embedding similarity scores. However, the most similar attribute de- scriptions even from the same attribute set may vary between: (1) different images of the same class, i.e., inconsistencies of visual features among different samples, and (2) the same image with evolving VR patterns, i.e., changes in 5 during training, leading to potential misalignment between image and relevant attributes. In response, we propose k-nearest neighbor attribute query to reduce the sensitivity individual attributes for addressing (1) and employ an iterative updating strategy to adapt to changing VR patterns as a workaround for (2).
Specifically, consider the training dataset DJ of the downstream task. For each downstream image a. we first obtain its transformation E.. (cf. Eq. (3)). Then, we identify sample-specifie k-nearest DesAttrs for x!. by computing the CLIP embedding similarity between aT and all attributes from the L.L--generate DesAttrs Aye((y"). ranked in descending order of similarity, such that ( v (1 {ay H simeLaP ay1d(s) simoLiP (2) al
(8)
Va Ades(y") (a1,....
Here, 5(e) refers to the VR pattern in the training epoch e. Similarly, the sample-specific k-nearest DistAttrs (x, y ( can be obtained in the same manner.
Then, the attribute- based embedding similarity score between a.f and VyT VTT which incor- porates its both sample-specifer A & Ah "eT.uTTpp T(T(ee) and A & A Aliso (2., 1186))), is
computed by a weighted aggregation:
sim (2) . 1 (8()))
(i 1 lo
Σ dist
simcriP (2.,a'(0'0)) )
where À E [0, 1] balances the contribution of DesAttrs and DistAttrs. Then, the predictive proba- bility Per(Yt
x = is determined for each sample s!! with softmax()) resembling Eq. (2), but now with new attribute-basse embedding similarity score sim [2] ( (8(0)) eval-
uated at each epoch We iteratively update the VR pattern, optimizing parameters g(e+1)
g(e) - v with respect to the cross-entropy loss (cf. Eq. (4)) under learning rate a, over the training dataset DT.
Comparison with Label-based VR. Besides using easily distinguish- able attributes that replace previous template- template-prompted labels with sim- ilar syntactic structures classification, Attr also aligns with the evolving nature of 8. In contrast to label-based VR that aligns images of the same class with fixed TP (t ) Att re-queries Ak (xf . y ((e)) and A (xf T for each image at every epoch. This ables AurVR to iteratively refine image-attribute alignment, yielding refined sim y ((())) over epochs. In other words, while both label-based VR and AutVR target the cross-entropy objective, AurVR benefits from contextually relevant optimization with sample-specific k-nearest attributes as supervision signals.
Algorithm Training Pipeline of Att
1: Input: Few-shot training data DT {(x]] "2,") >)>
hyper-parameters k,, A, learning rate a, epoch number E. and pre-trained CLIP model
facilitate
2: Output: Trained VR pattern g(E) applying AttrVR
3: Step 1: Calculate and Store Autribute Embeddings
4: for y YT do
5: Obtain (u) and Adist (v) by Eq. (7)
6: Get (a) for Va Ayes(y) Adise (v)
7: end for
8: # Step 2: Begin Training the VR Pattern
9: Initialize ((0) (0)ds
10: for e = to E 1 do
11: for 1 to do
12:
Compute Ai (x. )() A vlō(e)) by
Eq. (8) with Stored Embeddings for Vy E yI
13: Compute Per (v. (x?) by Eq. (9)
14: end for
g(e+1) <- gle) av # Iterative Update
15:
16: end for
Training Pipeline and Efficiency. Algorithm outlines the pipeline of Att.VR. We note that the text embeddings of DesAttrs and DistAttrs are pre-computed before training, introducing negligible computational overhead compared to label-based VR. See Appendix C.5 for details.

---

Published as a conference paper at ICLR 2025
5 UNDERSTANDING THE EFFECTS OF ATTRIBUTES
This section will justity why DesAttrs and DistAttrs would facilitate classification. The ease of classification decision boundary depends on class separability (Lorena et al., 2019), which quantifies how well different classes can be distinguished in the embedding space. This measure is jointly determined by intra-class variance (i.e., the spread of embeddings within a class) and inter-class distance (i.e., the separation of embeddings from different classes).
Definition (Class Separability). Let X and y be the input and class spaces as in Definition 1. Let Z be the image embedding space induced by the image encoder Zima : X 7 Z. For each class y E V, let X, be the set of all images with label y, and let Py = =mnex, Zimg 2img(0)/ (2)/ be the mean image embedding of class y. Then, class separability (CS) is defined as:
cs())
Σ xENY
[Z)m(=))
Pull? + DIIDI
E BAy
M1 1
D
NEY
d(y.v") inter-class distance
())) intro-class variation
The value of CS) 2) measures the difference of average intra-class variance, i.e., (r(o and inter-class distance, i.e., d(y. v'). across all classes. A higher value indicates the image embeddings better separated in the embedding space. Thus, the goal of maximizing class separability is equivalent to reducing intra-class variation while increasing inter-class distance.
Lemma 1.
Let (y) S A(y) be the set of descriptive attributes for class as with Definition 2. Let EA and Σt, be the covariance matrices of the embeddings optimized with respect Ades(y) and respectively. Then, for any class y E y, we have Tr (Ex (y)) S Tr (Ex. (y))-
Lemma 1 (details in Appendix B) shows that Ades (1) leads to reduced intra-class variances of image embeddings Vimg(1r), as the most frequently identified attributes in Xg imply that text embedding Zext (a) of attributes closely align with 7amg(x) for x E Xy. In addition, aggregating over Ades (3) pulls 7m (z) towards to class mean py. reducing the dispersion of per-class sample embeddings.
Lemma 2. Let Adisa C A(y) be the set of distinctive attributes for class
with Definition 3. Let d (y)3y')) and dv(v.8)) s/) be l distance between mean embeddings of two classes y F y'. opti- mized with respect to Adis (v) and y. Then, for any y, V E Y. we have d ty.v/)
) Z dr. (y. vf) if (1) which is easy to satisfy since [J]] is fixed while the size of Adisa (y) is unrestricted.
(Adisa >
Lemma2 2 (details in Appendix B) implies that Adist (3) promotes inter-class separation. Adist is uniquely associated with class and minimally present in classes y'. For samples x' E Xyy. the similarity between Zimg(x') and Zexe (a) is low for a E Adist Thus, the mean embeddings of different classes are pushed further apart due to the minimal overlap between Adisty) and Adisa (V')-
Corollary 1. Let ZA and Z be the embedding spaces obtained through attribute-bass -bate and label- based optimization. Denote the respective class separability by cs(): 2x) and CS()); Z1.), as with Definition 4. Then, under the conditions of Lemmas I and 2, it holds that cS(): ZA) > CS(); Zt.)
Merits of attribute-based optimization inspire practical VR solution. However, Lemmas and 2 ex- amine the effects of DesAttrs and DistAttrs in isolation, but attribute sets may overlap. Quantifying their combined effect is challenging due to the complex non-linearity of neural network optimiza- tion, making a careful balance between DesAttrs and DistAttrs essential for better performance.
6 EXPERIMENTS
Baselines and Benchmarks. To evaluate Attr we use CLIP as the pre-trained model and conduct experiments on 12 downstream classification tasks with 16 shots for each class following Oh et al. (2023). These datasets encompass diverse visual domains, involving scenes, actions, textures, and fine-grained details (see Appendix A.2.2). We include four baselines, including (1) ZS, which is the zero-shot performance of CLIP, (2) AurZS, which applies our DesAttrs and DistAttrs for zero-shot classification (see Appendix A.3), and -fate-of-the- the-thhertt art VR methods for VLMs: (3) VP (Bahng et al., 2022), which overlays VR patterns on resized images, and (4) AR (Tsai et al., 2020; Chen et al., 2023), which pads VR patterns around images. See the implementation details in Appendix A.2.3. Regarding hyper-parameters in VR,, set k = 3 and À = 0.5 and will discuss their impact.
7

---

Published as a conference paper at ICLR 2025
Table I: Accuracy comparison of different methods trained on 16-shot downstream classification tasks, using WTT-B16-based CLIP as the pre-trained model (Mean % Std %, ours are highlighted and the highest is in bold).
Method
Aircraft
Caltech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
IN
Resisc
Avg.
ZS
22.4
89.0
65.2
41.1
38.7
65.5
84.4
86.1
61.7
66.7
64.2
55.9
61.7
AttzS
28.5
94.1
65.1
54.3
50.8
81.6
86,5
91.6
65.6
69.3
69.3
62.2
68.2
VP
32.1
93.5
65.5
61.4
91.2
82.5
82.3
91.0
65.8
73.8
64.2
79.1
73.5
+0.6
+0.1
+0.3
40.5
40.3
+0.4
+0.1
+0.3
40.2
+0.5
+0.1
+0.3
AR
31.7
95.5
62.0
93.4
85.9
85.2
92.7
67.9
78.1
66.0
81.6
75.7
20.3
=0.2
+0.3
10.1
10.1
#0.1
#0.1
40.3
+0.2
+0.0
+0.3
AIIVR
36.6
95.7
68.3
65.6
93.8
92.9
85.9
93.3
69.6
79.0
69.4
82.6
77.7
+0.3
=0.1
+0.3
+0.8
+0.3
+0.4
+0.1
+0.0
+0.1
+0.6
=0.0
+0.4
Downstream Task: Flowers
Downstream Task: Texture
Downstream Task: Aivcraft
Amrv
AnrVg
VP
Accuracy (%6)
65
Acewacy (%)
Acouracy
60-
80
70-
30-
4
16 Shots
16
32
8
16
52
Shots
Shots
Figure 3: Accuracy comparison of different VR methods trained on different shots from [1, 4, 8, 16, 32). Pre-trained WTT-B16-base CLIP is used. The striped area indicates the error bars.
Overall Performance Comparison. Using CLIP with a ViT-B16 visual encoder as the pre-trained model, the comparison results are shown in Table 1. It can be observed that even only using De- sAttrs and DistAttrs for zero-shot classification already outperforms some baseline few-shot VR methods on the Caltech, Food, and SUN datasets. This demonstrates the effectiveness of DesAttrs and DistAttrs. However, VR methods remain necessary for datasets with significant domain differ- ences, such as EuroSAT and DTD. VR surpasses the baseline VR methods VP and AR across all datasets, achieving an average improvement of 2% over the trateoof-the-an methods across the 12 datasets. The advantages of AttrVR are particularly notable in fine-grained classification tasks with distinct visual feature differences, such as Flowers (+7.0%). DTD (+3.6%), and Aircraft (+4.99 (+4.9)) On the Food dataset, Att shows slightly lower accuracy than AttrZS, which may be because the images used by VR methods have smaller size than those used in the zero-shot settings.
Results of Sample Efficiency. We evaluate the performances of all VR methods across sparse (I-, 4-, 8-shots) and abundant (32-shots) training settings on Flowers, Texture, and Aircraft datasets. Fig- ure 3 shows that AutVR maintains stronger performance across all settings than baselines, demon- strating both resilience to data scarcity and effective use of additional samples when available.
Results on Different Backbones. We investi- gate how VR methods perform across multiple pre-trained backbones from the CLIP model fam- ily, ranging from ResNet50 ViT-L14. Table 2 presents the mean accuracy over 12 datasets, demonstrating that (1) all VR methods become more effective with more powerful backbones, and (2)AttrVR consistently outperforms baseline VR methods regardless of the backbone archi- tectural scales. Detailed results and analysis for each dataset are provided in the Appendix C.I.
Table 2: Average accuracy of different VR meth- ods on 12 datasets, using different backbones as CLIP visual encoders (Mean Accuracy %, ours are highlighted and the highest is bold, RN stands for ResNet).
RN50
RN101
VIT-B32
WTT-B16
VYT-L.14
ZS
53.4
56.1
58.2
61.7
68.7
AttZS
59,9
62.4
63.8
68.2
73.2
VP
53.2
57.1
67.5
73.5
61.1
AR
59.9
62.3
65.5
75.7
71.9
ARtV
64.2
66.8
69.1
77.7
75.5
Visualization Results of ARt. Figure illustrates the results of applying the trained VR pattern to images from the Flowers task with the label "glob thistle' and images from the Texture task with the label BBanded'. also shows the closest DesAttrs and DistAttrs corresponding to these results. For 'globe thistle", the closest DesAttr primarily describes its height and width, while the DistAttr
8

---

Published as a conference paper at ICLR 2025
Desciptive
plant that 3 feet tall 22: wide.
Descrppive
altenzttng
Moxt Noat Alttsllat
Atribuutes
Atributes
paralet eech other.
Deetinttve
uruque sobetical shaps, which wis Itapart from other Eowers....
Desinctive
diffeeeat colors, testures muneral composiition within moterial...
Altiibutes
Attribuess
Deseripttve Attribunes
... tall and sqiay peremnial plazt taat bypically grows to be 2-3 Shet haight
Desrigtive
iltimatinn bandk of different onlors, pallens, lize.
Labd
Attibuutes
Distinctive
tummeross globe staped each with spicy blus purpue petals.
heeds.
Distinxtive
differest colurs teclures rurning bottzuztallyy vertically, digpndlly
Attributes
Attrlbute
Figure 4: Visualization of images with Att patterns, and their nearest Aestttts and DistAttrs, using the ViT-B16-bases 6-baeed CLIP as the pre-trained model. Two images labeled 'Globe Thistle' from 'Flowers' and two labeled 'Banded' from 'Texture' are chosen as examples (more in Appendix C.2).
pink primrooe
bandHlrvad pocket archid
conterbury bells
swaee pes
mooe orchid
globe thistle
monkstood
z8
VP
AR
AmtVR
Figure 5: T-SNE visualization results of image embeddings from seven classes in the Flowers task, utilizing the ViT-B16-based CLIP as the pre-trained model. In the first plot, embeddings zero- shot images indicated with ZS. The following three plots display embeddings of images with VR patterns, categorized by different training methods and marked as VP, AR, and AurVR, respectively.
highlights features that differentiate it from other flowers, such as its spherical shape. Different samples of 'globe thistle' may have different closest DistAttrs; for instance, the image with blue- violet petals will be closest to a DistAttr with similar description. Similarly, for images with the 'Banded' label, the DesAttrs mainly describes the common feature of alternating textures, while the DistAttrs capture unique characteristics of the class or individuals, such as 'mineral composition or 'diagonal textures' shown in Figure 4.
Visualization Results of Embedding Space. Figure 5 plots the 2D t-SNE embeddings (Van der Maaten Hinton, 2008) of classifying samples from the Flowers task under input VR methods, with different colors representing different categories. It can be observed that in the zero-shot (ZS) scenario, some classes, such as 'moon orchid' (marked with pink dots), are scattered and difficult to classify. However, label-based VR methods, such as VP and AR, help to clarify the boundaries of these classes, making the samples casier to distinguish. Despite this improvement, some classes, like 'canterbury bells' (marked with green dots) and 'sweet pea' (marked with red dots) still remain relatively indistinguishable. After applying our VRR, the embeddings of various categories cluster more distinctly in the 2D visualization plane, resulting in clearer and more separable distributions.
Ablation Studies. Table 3 presents the ablation studies, and sequentially details: (1) w/o VR: the re- sults of AR without training the input VR, where only zero-padded images from the downstream task are classified by our DesAttrs and DistAttrs, along with k-nearest neighbor attribute selection for zero-shot results; (2) w/o DesAnrs:: the results of training Att utilizing only DistAttrs, ex- cluding DesAttrs; (3) w/o DistAttrs: the results of training AR utilizing only DesAttrs, excluding DistAttrs; (4) w/o both Attrs: the results without both attributes, which correspond the results of the label-based VR approach; and (5) Ours: the results using our proposed method, ARr.
In the absence of training VR patterns, performance on downstream tasks can be unsatisfactory when there is significant domain shift from the pre-trained CLIP model's domain. For instance, low accuracy is observed when the downstream tasks involve remote sensing datasets such as ESST or Resisc, or texture datasets like DTD. Thus, training VR patterns is crucial for effectively adapting the pre-trained model to unfamiliar domains.
In the absence of DesAttrs, the method relies only on unique attributes during training, emphasizing class differences in the downstream task. This approach works well when the downstream domain closely matches the pre-trained model, in broad classification tasks like Caltech. However, for tasks with significant domain shifts and few classes, such as the 10-class remote sensing dataset AS, it may miss some overall attributes relevant certain classes, resulting in lower performance.

---

Published as a conference paper at ICLR 2025
Table Ablation studies of AttrRR using ViT-B CLIP as the pre-trained model (Mean % Std %, ours are highlighted and the highest is bold).
Method
Aireraft
Caltech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
IN
Resisc
Avg.
VR
25.4
94.1
62.3
48.5
91.6
64.3
68.0
61.2
10.4
10.1
10.1
10.1
10.3
10.3
10.1
10.1
10.0
10.3
10.1
10.1
w/o DesAttrs
36.1
95.9
68.2
64.8
93.1
92.6
85.8
93.3
69.4
78.0
69.3
81.9
77.4
+0.4
40.1
+0.1
+0,4
#0,2
+0.6
#0.0
#0.1
+0.0
=0.4
#0.1
#0.6
w/o DistAttrs
35.9
95.6
68.2
64.4
93.8
92.4
85.7
93.0
67.7
78.6
68.9
81.8
77.2
+0.3
+0.1
#0.2
+1.1
+0.3
+0.1
+0.0
#0.2
+0.2
=0.5
+0.0
#0.1
w/o both Attis
31.7
68.0
62.0
93,4
85,2
92.7
67.9
66.0
81.6
75.7
+0.3
+0.2
+0.3
#0.1
#0.1
#0.7
#0.1
#0.1
20.3
=0.2
+0.0
+0.3
Ours
36.6
95.7
68.3
65.6
93.8
92,9
85,9
93.3
69.6
79.0
69.4
82.6
77.7
10.1
10.1
20.0
Downstream Task: Flowers
Downstream Task: Texture
Downstream Task; Aireraft
94.0
Acturccy (%)
Accuracy (50)
66.0
92.0
64.0-
Accarncy
62.0
Mean
Mean
90.0
#1 Std
+1 Std
34.0-
+] Sid
0.6
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
94.0
Downstream Task: Flowers
68.0
Downstream Task: Texture
38.0
Downstream Aireratt
Accuncy (%6
Accuracy (%)
92.0
Accuncy
64,0
36.0
90.0-
Mean
Std
62.0
Sid
34.0
Mean
#1
$8.0
60.0
The first row shows the impact weight that balances DesAttrs and DistAurr. The second row shows the impact of k. being the number of nearest attributes selected for classification. Pre-trained ViT-B16-based CLIP is used.
Figure 6: Performance comparison applying different
In the absence of DistAttrs, the method may overlook some attributes crucial for differentiating between categories in the downstream task. For datasets with many differrrrfreet classes, such action classification datasets like UCF or texture classification datasets like DTT, not using DistAttrs can negatively impact classification performance. Besides, without both attributes, Att degenerates into label-based VR, forfeiting its advantages in fine-grained classification tasks.
Hyper-parameter Analyses. Figure 6 illustrates the impact of the hyper-parameters À and k. The weight used to balance the contributions of DesAttrs and DistAttrs. For different tasks, the optimal À varies, with accuracy generally rising and then dropping as À increases, indicating moderate is needed balance DesAttrs and DistAttrs. For convenience, we set X 0.5 for all. The parameter &t represents the number of nearest attributes selected for classification; a value that too small may result unstable classification, while a value that is too large may lead to attribute redundancy. We chose h = 3 for all datasets in this paper (see Appendix C.6 for its impact).
More Experiments. Appendix C.3 includes the aggregation studies of the h-nearest attributes se- lection. Appendix C.4 shows that label-based VR patterns are not compatible with Att patterns. Appendix C.7, C.9 includes results of generating attributes with other LLMs or VLMs, and Ap- pendix C.8 demonstrates how AttrVR handles cases when generated attributes are of low quality.
7 CONCLUSION
We introduce AttrVR, which extends unimodal Visual Reprogramming to VLMs (c.g., CLIP) for downstream classification tasks, specifically targeting the CLIP's inherent ability to align visual and textual information. Instead of using cemplate-prompter labels, AttVR optimizes through label- based attributes, making direct use of CLIP's cross-modal alignment properties. Both theoretical analysis and experimental results show that AurVR outperforms conventional label-based VR. The visualization results, along with ablation, aggregation, and hyper-parameter studies, validate the effectiveness of Vr in reprogramming CLIP. The introduction of Att marks an advancement in adapting VR from repurposing single-modal pre- pre-trained models with predefined label spaces to multimodal models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
CYC, ZSY, and FL are supported by the Australian Research Council (ARC) with grant number DE240101089, and FL is also supported by ARC with grant number LP240100101, DP230101540 and the NSF&CSIRO Responsible AI program with grant number 2303037. JZQ is supported by ARC with grant number DP240101006. This research is also supported by The University of Mel- bourne's s Research Computing Services and the Petascale Campus Initiative. We sincerely appreciate the time and dedication of the reviewers in carefully reviewing our manuscript.
ETHICS STATEMENT
Since the method proposed in this paper is used improve VR performance for downstream classi- fication tasks with CLIP, there is no potential negative impact.
REPRODUCIBILITY STATEMENT
Appendix offers clear explanations of the theoretical results in Section 5. For reproducing the experimental results presented in Section 6, we have included link anonymous downloadable source code in the abstract. Appendix A.2.2 provides details about the open-source datasets utilized in our experiments.
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv, 2022.
Lukas Bossard, Matthieu Guillaumin, and I luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In ECCV, 2014.
Tom B Brown. Language models few-shot learners. In NeurlPS, 2020.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Bayesian-guided label mapping for visual reprogramming. In NeurlPS, 2024a.
Chengyi Cai, Zesheng Ye, Lei Feng. Jianzhong Qi, and Feng Liu. Sample-specific masks for visual reprogramming-boiee prompting. ICML, 2024b.
Aochuan Chen, Yuguang Yao, Pin- Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and im- proving visual prompting: A label-mapping perspective. In CVPR, 2023.
Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Understanding and mitigating the label noise in pre-training on downstream tasks. In ICLR, 2024.
Pin-Yu Yu Chen. Model reprogramming: Resource-efficient cross-domain machine learning. In AAAI, 2024.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Bench- mark and state of the art. Proceedings of the IEEE, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv, 2020.
Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks. In ICLR, 2019.
11

---

Published as a conference paper at ICLR 2025
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR workshop, 2004.
Vittorio Ferrari and Andrew Zisserman. Learning visual attributes. NeurIPS, 2007.
Hambardzumyan, H Khachatrian, and May. Warp: Word-level adversarial reprogramming. In ACL-IJCNLP, 2021.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E" 2vpt: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
J Harold, G Kushner, and George Yin. Stochastic approximation and recursive algorithm and appli- cations. Application of Mathematics, 1997.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Yun-Ning Hung, Chao-Han Huck Yang, Pin-Yu Chen, and Alexander Lerch. Low-resource music genre classification with cross mroal neural model reprogramming. In ICASSP, 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Yongcheng Jing, Chongbin Yuan, Ju, Yiding Yang, Xinchao Wang, and Dacheng Tao. Deep graph reprogramming. In CVPR, 2023.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In CVPR, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013.
Neeraj Kumar, Alexander Berg, Peter N Belhumeur, and Shree Nayar. Describable visual attributes for face verification and image search. TPAMI, 2011.
Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bailey, and Feng Liu. Visual-text cross alignment: Refining the similarity score in vision-language models. In ICML., 2024a.
Zheng Li, Xiang Li, Xinyi Fu, Xin Zhang, Weiqiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In CVPR, 2024b.
Ana C Lorena, Luis PF Garcia, Jens Lehmann, Marcilio CP Souto, and Tin Kam Ho. How complex is your classification problem? a survey on measuring classification complexity. CSUR, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2016.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv, 2013.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023,
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian conference on computer vision, graphics & image processing, 2008
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In CVPR, 2023.
12

---

Published as a conference paper at ICLR 2025
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivas- tava. Learning predict visual attributes in the wild. In CVPR, 2021.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. In
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2012.
Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Argue: Attribute-guiee guiidegguddddddd prompt tuning for vision-language models. CVPR, 2024.
Yun- Yun Tsai, Pin-Yu Chen, and Tsung-Yi Yi Ho. Transfer learning without knowing: Reprogramming
black-box machine learning models with scarce data and limited resources. In ICML, 2020.
Hsi-Ai Tsao, Lei Hsiung, Pin-Yo Chen, Sijia Liu, and Tsung-Y Yi Ho. Autovp: An automated visual prompting framework and benchmark. In ICLR, 2024.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
Ria Vinod, Pin-Yu Chen, and Payel Das. Reprogramming language models for molecular represen- tation learning. NeurlPS, 2020.
Wenhao Wang, Yifan Sun, Wei Li, and Yi Yang. Transhp: Image classification with hierarchical prompting. In NeurlPS, 2023.
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Connecting the dots: Collaborative fine-tuning for black-box vision-language models. In ICML, 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. CVPR, 2010.
Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang. Towards few-shot adaptation of foundation models via multitask finetuning. In ICLR, 2024.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, 2023.
Chao-Han Huck Yang, Yun- Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In ICML, 2021.
Chao-Han Huck Yang. Bo Li, Yu Zhang, Nanxin Chen, Rohit Prabhavalkar, Tara N Sainath, and Trevor Strohman. From english to more languages: Parameter-efficient model reprogramming for cross-lingual speech recognition. In ICASSP. 2023a.
Yue Yang. Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- Burrhh, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. CVPR, 2023b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 2022b.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX 1: MORE TRAINING INFORMATION
A.1 THE PROBLEM SETTING OF VR FOR CLIP
Downstream Tacks
CLIP
Visual Finetuning
Downsttrom Tooks
CLIP Visual Prongt
Dowsstremm
Visual
Rocoder d VIT)
Image
Reprogramiin
Coeme Doemee Strran eo
Suntbelyy
Image
(ResNet VIT)
Oostae
This a photo [Russi Blue)
Text Encoder
This is photo of
This photo
Encoder
|Russia Blue)
Easoder
Figure 7: Different problem settings for repurposing CLIP for image classification tasks. The left shows finetuning the visual encoder, the middle illustrates generalized approach visual prompt- ing, and the right depicts visual reprogramming (VR). The trainable parameters are highlighted in "fires". VR merely modifies the input image space, making applicable to any encoder architecture.
The problem setting for VR and differences from other repurposing methods for CLIP in image classification tasks are illustrated in Figure 7. For finetuning methods, the weights in the pre-trained ResNet-based or ViT-based CLIP are optimized directly using samples from the downstream task, making the Image Encoder variable. Visual prompting methods (Jia et al., 2022; Khattak et al., 2023) add parallel trainable weights next to the embedding patches the first layer or each layer of the Image Encoder, but this approach is only applicable when VIT is used the visual encoder.
Unlike tine-tunngg or visual prompting, VR (Chen et al., 2023; Bahng et al., 2022) focuses on modi- fying the input space of the model rather than the pre-trained model itself. VR directly incorporates trainable parameters into the input images to achieve the repurposing of the pre-trained CLIP. Com- pared other methods, VR offers the following advantages:
Since VR only modifies the input space, it has fewer parameters (see Table 9 for details), and the number of parameters is independent of the size of the pre-trained model, only depending on the input image size. This results in lower training time overhead.
By solely altering the input space, VR ensures that the original parameters of the pre-traines trained model remain unchanged, effectively addressing practical issues such as catastrophic for- getting in large models and copyright concerns.
As the VR pattern is applied only to the images before input, is independent of the archi- tecture of the pre-traine pre-trained model's 's image encoder, making it applicable to all architectures.
The VR method is orthogonal to other fine-tuning methods VR modifies the input space, while other methods adjust the internal parameters of the model. Therefore, VR can be combined with various methods to further enhance performance.
A.2 IMPLEMENT DETAILS
A.2.1 GENERATING DESATTRS AND DISTATTRS
We used GPT-3.5 (Brown, 2020) to generate DesAttrs and DistAttrs. The specific hyper- prramtparameter settings for text generation were follows: temperature set to 0.99, maximum token size of 50, and generating 25 entries for each category. The termination signal was "", and only entries with considered valid and retained.
length greater than 20 characters
To generate DesAttrs, we queried each class in every downstream task with the following input instruction:
Describe the appearance of the [Task Info.] [Class Name],
where [Tas Info.] represents the description of downstream tasks, shown in Table 4, and [Class Name] represents the name of label y² E yr.
To generate DistAttrs, the following input instruction:
14

---

Published as a conference paper at ICLR 2025
Describe the unique appearance of a/an [Class other [Task Info.].
Name]
from the
Using this approach, we successfully generated DesAttrs and DistAttrs. In the experiments, we set 20 as the size for sets Ades (y*) and Adis (y*). When the number of valid entries generated for class y" is less than m, we randomly resampled to ensure that each class had exactly 20 attributes, facilitating subsequent experiments.
A.2.2 DATASET DETAILS
Table 4: Dataset Information
Aircratt
Caltech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
Resise
Info.
Tisk
sircraft
model
object
automobile
tise-grained
texture
remote sensing
land cover
pet object
sensing scone
remote
Class
100
196
47
101
1000
45
Batch
Size
64
64
64
64
64
64
64
This paper establishes benchmarks for downstream classification tasks following prior work (Oh et al., 2023), employing the same methodology to split the 16-shot training, validation, and test sets. The 12 datasets are listed as follows: FGVCAircraft (Aireraft) (Maji et al., 2013), Caltech101 (Caltech) (Fei-Fei et al., 2004). StanfordCars (Cars) (Krause et al., 2013), Texture (DTD) (Cimpoi ct al., 2014), Au (ESAT) (Helber et al., 2019), Flowers (Flowers) (Nilsback & Zisserman, 2008), Food101 (Food) (Bossard al., 2014), OxfordPets (Pets) (Parkhi et al., 2012), SUN397 (SUN) (Xiao et al., 2010), UCF101 (UCF) (Soomro et al., 2012), ImageNet (IN) (Deng et al., 2009), Resisc45 (Resisc) (Cheng al., 2017). All image datasets are publicly available. Detailed task information and the batch size used for training VR are provided in Table 4.
A.2.3 TRAINING VR PATTERNS
For all VR baseline methods compared in the paper, we adopted the following uniform training settings: an initial learning rate of 40, momentum of 0.9 using the SGD optimizer (Harold al., 1997), and cosine annealing learning rate scheduler (Loshchilov & Hutter, 2016). The total number of learning epochs was set to 200. The experimental results represent the average across three seeds.
Regarding method method-specific hyper-paramette parameters, for VP (Bahng et al., 2022), we maintained consis- tency with the original work, using VR noise pattern with a frame size of 30, as detailed in Table 9. For AR (Chen et al., 2023; Tsai et al., 2020), noted by Tsao et al. (2024), the size of different VR patterns can impact the results. In this study, we conducted experiments with frame sizes of [8, 16. 32, 48] and selected 16 as the final frame size due to its optimal performance with fewer parameters. To ensure fair comparison, our VR adopted the same parameter settings as AR.
A.3 DETAILS ABOUT ZERO-SHOT ATTRZS
For the i-th downstream image x!. and certain label TT AtrZS is the zero-shot version of Attr VR where we do train the VR noise pattern. AttZS first resizes the sample to the required input size for the model, then calculates the similarity between the resized sample and the attribute descriptions of each class in single pass, applying the similar equation of Attr
simAnzsl(a) .. xー
Σ K
simcLir Ta (T.
1-A k
Σ d'EAt
simeLar (
(10)
where k. À are hyper-parameters that also used in Aut G.. is the resized image and a, a' attributes chosen from the attribute set Ak 7k dist Then the label with largest simAurzs (xi)'
will be the prediction result for sample x!!"
15

---

Published as a conference paper at ICLR 2025
B APPENDIX 2: MORE THEORETICAL JUSTIHICATION
Lemma 3 (cf. Lemma 1). Let Ades(y) S Aly) be the set of descriptive attributes for class y with Definition Let La and Σt, the covariance matrices of the embeddings optimized with respect (y) and y, respectively. Then, for any class y E y. have Tr (EA (y)) = Tr (EL (v))-
Proof. Let X, be the set of all images belonging class y. We begin by defining the following: let Zr, X' + R2 be the label-based embedding function, let ZA X - R be the attribute-based embedding function, and let Z, X' + R be the embedding function for single attribute a E A.
Denote the mean embeddings, covariance matrices, and traces resulting from Zr. by
Fr. E, [z,( = ENEX, [Zn(x)) - 11.) (Z)(Z) - -vr)],
(T = ErEX, [1Z)(z) -u113
Similarly, for A. we have
μΛ E [Za(x)],
Σд ExEx, [(a((z) - MA) (ZA(x) - Ha)]
Ar = E,E, [I<A(T)) Mall?
By Definition 2, we further express Axxx) in terms of (x)):
Za(x) Adea aEAdrrly)
Σ
Zz(
and accordingly, the attribute-ment meab is
PA [Z^(x)] [Ades (3)l
Σ
EZEX, [Za(
aEAdme(v)
Then, the difference between embedding and mean is
(Za(x) ExEX, (z)). Za(x) - -MA [Ades()) Ax)
Σ
Jensen's inequality states that for any convex function f and probability measure p. we have: [X], < [X)]] Applying this to the squared norm (which is convex), we obtain:
(z))x=) 2all==
Σ
(Za - EZEx, [Z, (x)))
(y)
rEAAcc (v)
<
Σ
[[(Za(z) - Exex, 2Za (x))))
Ados(y)
Taking expectations on both LHS and RHS leads to
Exex, (1Z -
Σ
Excx, [[](Z(()) ELEx, 12. (x))
(I)
[ aEAAtes())
We also know that for any attribute E Ades
(a(a) 2 Uy(d'),
Va' E A(v) \ Adcs(y),
16

---

Published as a conference paper at ICLR 2025
where Uy(a) Xx, ExEx, fa( is the frequency of attribute in class y. Define Za Exex, [x)] as the mean embedding for attribute a class y, we can then express the variance of attribute-bis based embedding as:
|
R Rx,, [1a((t) mall]] BxEx,
(d) aEAdes())
Σ
(Za(zc) - 2ra))
Σ
EXEX, [(Zn(z) Za) (Zn Zo.
Ades(1) (y)l
By the Souccy-Schwarz inequality, we have:
Bxex, [[2](x) (Z. 2ar1] s ( 2al1²] [| -0000)2-)1))
Zor 2o.119]. (12)
Since
we have already established the relationship between BXEX, |22(()) - Hall" and
ExE [|Za(z)) - Za | we then proceed to prove that for any E (d)
ExeX,
[|(a(x) Zall²] S L,la) [[]z(() mzll"].
(13)
By definition of fz))), it takes value if attribute is in and O otherwise. We express Za (z(x) in terms of (x) and fa()):
Za(z) fa()) + (1- fa(z))
since (a) is the frequency of attribute in class y. Expanding the LHS, have:
ExEx, [| Za | ExEX,
(1 fa(z)) -
-1-
RREE [fa(x)² [a(x)>||Z((<)) 2. [|]
-U)(a)EDEX, ||Z)(x) - 2.1l 1| [fa(z) =
[| {a
- 121l
where the second equality holds since [x)) (x) and Exex, [x)] Uy(a), the last inequal- ity is justified based on mild assumption that Z, is closer to the class-specific mean embeddings than μr for a descriptive attribute.
Applying Eq. (12) and Eq. (13) to Eq. (11):
Exex, [IZa(x) [all3]] = Adoe(y))
Σ
EXEX, [(z(z)) - (Zw Zo.)]
(a)mm)))))))
Σ
(()(a)U, [1Z((z) mEl1
Ados (v)
0, (.
Adess (1)
Ades (v) BREX, [1 -mr113)
Exex, [1 - -ur.11?
The second inequality follows that (a) < 1 for all attributes. By definition of Tr(-), have derived Tr(Ea) < Tr(E..).
Lemma 4 (ef. Lemma 2). Let Adisty) C A(y) be the set of distinctive attributes for class as with Definition 3. Let da(v, sf) and dilu.y') be la distance between mean embeddings of two classes y + y. optimized with respect to (yis(y) and y. Then, for any y.y E y, we have dA (u. v') 2 dr. (v.1/) if Adis (v)) DD%
17

---

Published as a conference paper at ICLR 2025
Proof. Let Xy be the set of all images belonging to class y. We begin by defining the following: let Zz X' + Rd be the label-based embedding function, let ZA X - F be the attribute-basee embedding function. Let Tr and T' be the text embedding functions for prompted class labels and attributes, respectively.
For any class E V, define: (v) = EZEx, [z)( and ma(v) = Ezey, (Za))) For any two classes y. s E V, define distances by
dL(y. 3') (1 #2
dA(u,3') [IMA()) pA(V')ll
By Definition 3, for any (y) and y y V. E y we have:
Exxx [f(())] E _xx] [x)]]
Then, derive inequality regarding the similarity:
BeEx, [sim{ (x)),, Ta(a)) > REEx, Tx(a)),
(14)
for any a E Adiaa (v) and y * y/ where (r((z) = arg max, sim(z, (y)))) and (x = arg max; Asss(v) Aat sim(2, (a))), sim(:-) denotes the cosine similarity.
We then define a transformation o(.) that maps the embeddings to a space where the Euclidean dis- tance corresponds to the dissimilarity in the original space, which is isometric with respect to the cosine similarity in the embedding space. In this space, each dimension corresponds to the dissim- ilarity with specific attribute, preserving the similarity between and each attribute embedding. such that
(6z) = [V1 sim(z, (a(a)a,)))......... / sim(z, (((,,TTAaaaa (a())))))
where A = UyEy (y). Then, for any two classes and y/, the distance between their mean embeddings:
(()())())) ((AAW!))
LAI
(VI (w) v1 sim(u (i"( Ta()))
(Vi (Vi- (Aa) 1 Timss((((());
a:Evvw)ly)
Σ (V1- (V1- (in-sim(((()))TTTC - 1 (im(uam))))) 2
a3g.Auis())
Referring to Eq. (14), we know that the inequality relationship accumulates because of the summa- tion over all [A] terms.
We then define similar transformation for the label-based method:
(1(2) [VI sim(z, (z))) ................................ v sim(z, Tz (
Accordingly, the distance between mean embeddings under label-based methods:
D Σ
(lbl(1(()) (LL
sim (IL()) V sim(ue(0))
1=1
Denote SA - Ea.eAmm(s) (vI sim( - (0n
and
St. A [( dz (r(uz( (v (v )],,, and the average contribution of each component by SA Trtive (V) SA and S₁ S it is easy to check that SA > S,, because ExeN, > Exe (x)).
18

---

Published as a conference paper at ICLR 2025
Then, assume [Adist (3) |>] (this assumption is mild in common practical classification
tasks, where {}] often ranges from 10 to 100, whereas we can easily identify Adist (v) > 100 dis- tinctive attributes for class of objects because of the dimensionality of natural language vocabulary and sentences.), we have concluded that SA > SL, implying that
Σ
(V1- sim(ua (V), Ta(a:) V1
a.E.diia (1)
2 (r)) Tr.(f)))
(V): - / (ii(mmm((rr))))))
(imlaaimmaa)))aaaaaaa 2 non-negative, we
As En, (M) arrive at ((((a(v)) ((aa()))) > 11o - or. (mn(()))
Recall that transformations <b.)) and (.) isometric with respect the cosine similarity, have
()(u)())) -((x(())))((((()))1==)))) c
(z)=z())) - a(u (v MII = c. (v.
for some constant c 0. Dividing both sides by c, we can conclude du 1/) dr
C APPENDIX 3: MORE EXPERIMENTAL RESULTS
C.1 MORE RESULTS OF DIFFERENT BACKBONES
Table 5: Accuracy comparison using RN50-based CLIP as the pre-trained model (Mean % + Std %, highlighted and the highest is in bold).
ours are
RNS0
Aireraft
Callech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
IN
Resisc
Avg.
15.5
82.1
56.2
37.5
29.2
58.0
75.8
79.7
57.6
55.5
37.7
53.4
AtteSS
19.4
87.1
56.5
51.9
34.8
75.1
78.0
88.3
60.4
60.9
61,2
45.8
59.9
VP
16.2
80.1
44.0
43.4
59.7
53.6
65.3
77.2
48.8
52.0
49.7
47.7
53.2
AR
18.6
86.5
53.9
46.4
66.6
60.9
74.2
82.5
56.8
59.7
54.4
58.4
59.9
AREVR
20.7
89.1
$3.9
54.4
72.0
74.8
75.3
88,9
59.9
63.6
59.2
58.2
64.2
Table Accuracy comparison using RN101-based CLIP as the pre-trained model (Mean % + Std %, ours are highlighted and the highest in bold).
RN101
Aircraft
Caltech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
IN
Resisc
Avg.
ZS
17.1
86.0
63,9
39.0
28.1
59.7
79.6
81.9
56.5
58.4
58.7
44.4
56.1
AmtZS
20.8
90.9
63.0
51.4
35.1
75.8
81.2
89.5
62.1
63.7
63.8
51.2
62.4
VP
19.3
83.0
53.7
43.4
62.8
57.2
71.2
80.2
53,5
54.2
53.1
54.0
57.1
AR
19.5
89.7
62.0
46.3
70.4
60.4
78.0
84.4
58.4
60.6
$7.9
60.2
62.3
AirVR
23.3
92.0
62.2
55.6
70.3
76.2
79.5
89.3
62.1
64.5
62.2
64.5
66.8
Table Accuracy comparison using ViT-B32-based CLIP as the pre-trained model (Mean % Std %, ours are highlighted and the highest bold).
ViT-B32
Aircraft
Caltech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
IN
Resisc
Avg.
ZS
18.3
89.4
60.1
40.0
37.0
60.8
79.2
82.5
59.4
61.4
60.1
49.8
58.2
AttiZS
22.2
91.5
59.2
50.7
44.3
76.0
81.0
89.5
63.6
66.8
64.3
63.8
VP
24.3
92.3
58.6
54.9
85.9
71.2
75.0
86.8
61.0
67.3
59.0
73.9
67.5
AR
92.7
56.9
85.6
66.7
59.9
63.5
57.5
VR
24.5
92.0
56.6
56.8
88.6
77.8
77.2
89.8
62.8
67.9
61.0
73.9
69.1
Tables 5-8 present the results of different methods using various architectures of the CLIP image encoder. For all models, we employed the same VR parameter numbers and hyper-parameter settings those used in VIT-B16. 16. In this configuration, the VR method requires downscaling the images
19

---

Published as a conference paper at ICLR 2025
Table Accuracy comparison using ViT-LL4-based CLIP as the pre-trained model (Mean % Std %, ours are highlighted and the highest bold).
ViT-L141
Aireral
Caltech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
IN
Resise
Avg.
ZS
29.7
90.4
77.1
51.1
55.3
73.7
88.9
89.0
65.0
72.9
71.3
60.4
68.7
AttiZS
94.4
77.0
54.4
85.6
91.4
94.2
75.2
76.2
63.9
73.2
25.7
88.3
59.8
45.5
30.9
69.1
74.9
84.5
63.4
58.8
74,4
61.1
31.7
93.0
75.6
55.9
70.4
74.7
89.0
91.5
65.4
69.7
72.2
71.9
A
38.2
96.1
74.8
61.1
74.9
85,8
90.0
94.3
68.6
77.8
73.8
70.1
75.5
and adding trainable noise at the edges or directly on the images, which can sometimes adversely affect image quality and, consequently, classification results. As result, certain benchmarks that demand high detail and resolution, such as Cars and Food, the accuracy may be lower than that of shot learning. In summary, the following observations can be drawn from the tables:
When comparing zero-shot methods, our proposed approach, AttZZS, which uses DesAttrs and DistAttrs in conjunction with k-nearest neighbor attribute selection, achieves an aver- age accuracy improvement of 4.5% to 6.5% across 12 benchmarks compared to label-based zero-shot classification. This clearly demonstrates the effectiveness of using attributes in- stead of labels.
When comparing VR for CLIP methods, even in cases where the baseline VR methods, such as VP and AR, exhibit underfitting with RN50 or overfitting with ViT-L14, replacing label-based VR with Att consistently improves average accuracy by 3.6% to 4.5%.
Furthermore, VR outperforms AttrZS, especially in downstream tasks where there is sigmificant domain shift between the task domain and the CLIP pretraining domain (c.g., ESAT and Resisc). This advantage is even more pronounced when using small image encoder (e.g., RN50), in the pre-trained CLIP model.
C.2 MORE RESULTS OF TOP MATCHED ATTRIBUTES
Figure 8 and Figure 9 show the visualization of image samples with Vtt patterns, and their near- est k DesAttrs and DistAttrs before and after training VR patterns. To illustrate the differences between individual samples within the same class, we selected two samples from the same task and class for demonstration. Figure 8 shows two samples labeled as 'banded' from the Texture task, while Figure 9 displays two samples labeled 'Globe Thistle' from the Flowers task. From the visualization results, we can draw the following conclusions:
The necessity of the Iterative Updating Strategy: It is evident that the DesAttrs and DistAttrs closest to the same training sample differ before and after VR pattern training. Prior to training, the attribute descriptions that are closer to the sample often share similar keywords; for instance, the DesAttrs describing 'banded' tend to emphasize 'different bands' (Figure 8), while those describing 'Globe Thistle' focus the height of the flowers (Figure 9). However, after VR pattern training, the DesAttrs and DistAttrs that are close to the same sample or different samples vary significantly.
The necessity of both DesAttrs and DistAtrs:: DesAttrs and DistAttrs have different focal points. DesAttrs primarily describe the overall characteristics; for example, shown in Figure 9, in the de- scription of Globe Thistle, keywords like 'tall' and '3 feet' would appear. In contrast, DistAttrs emphasize features that distinguish Globe Thistle from other flower categories, such as 'spherical shape' and 'globe-shaped', During the training of the VR pattern, both the overall information of individual classes and the distinguishing features between different classes are needed. This aligns with the experimental results Table 3.
Combining DistAttrs with k-nearest neighbor aftribute selection enhances the ability to capture unique features of individual samples: Some distinguishing characteristics of specific class may not be universal (i.e., some samples possess these features while others do not). In such cases, Att employs k-nearest neighbor attribute selection to filter out attributes that are unique to individual samples. For instance, in the 'Banded' examples in Figure 8, the first image depicts land sediment, leading to the presence of the keyword 'rock or sediment', while the second image features diagonal stripes, resulting in the closest DistAttr including the keyword 'diagonal'.
20

---

Published as a conference paper at ICLR 2025
Before Training VR [Banded]
Descriptiee Attributes
Top
appears as series of distinct horizontal or vertical bands...
Top
distinct stripes or bands of different colors, materials, or thicknesses.
Top
alternating bands or stripes different colors, patterns, or materials.
Distinctive Attribatts
Top
has distinct stripes layers that are visible on the surfice.
Top 2
...bands or layers of different colors, textures, or mineral composition within material, such as rock or sediment.
Top
distinctive layers or bands of different materials colors.
A ffter Training VR [Banded]
Descriptive Attributrs
Top
altemaating bands different textures run parallel to each other.
Top
parallel lines or bands that run acmss the surface.
Top
altemating bands different colors, patterns, or lines.
...bands or layers of different colors, textures, or mineral composition within material, such as rock or sediment
Distinztive Attributta
Top
Top 2
distinctive layers or bands of different materials or colors.
Top
alternating bands of different colors or pattems..
Before Training VR [Banded]
Descrittiee Attributes
Top ... alternating bands of different colors, patterns, lines.
Top
... appears series of distinct horizontal vertical bands...
Top
there are distinct, parallel bands or stripes visible.
Distinctive Attributtas
Top
paralicl stripes or bands that are often of different colors or textures.
Top 2
alternating bands of contrasting colors, shades, or textures.
Top
alternating stripes bands of diffcrent colors textures.
A ffter Training VR [Banded]
Descriptive
Attributes
Top 1
... alternating bands of different colors, pattims,, lines.
Top 2
appcars series of distinct horizontal vertical bands...
Top
distinct and repetitive lines or bands that run parallel each other.
Distinctive
Attrilatts
Top 1
alternating bands of different or textures running horizontally, vertically, diagonally across surface.
Top 2
stripes of different colors or textures running horizontally or vertically
Top
parallel stripes or bands that are often of different colors or textures.
Figure 8: Visualization of images with AttrVR patterns, and their nearest A = 3 DesAttrs and DistAttrs before and after training VR patterns, using the ViT-B16-based CLIP as the pre-trained model. Two images labeled 'Banded" from the Texture task are chosen examples. The closest DesAltr and DistAttr after training convergence of VR patterns are highlighted with red borders.
C.3 MORE RESULTS OF AGGREGATION STUDIES
Aside from the i nearest neighbor (knn) attribute selection method applied in AttrR,, we also conduct some aggregation studies replacing funn with the following modules to test its impact:
The maximum similarity (max), Unlike Aur in this experiment, we retained only the nearest attribute with the maximum similarity in the knn attribute selection phase for subsequent calcula- tions. The specific logits output sim max g(c) can be expressed as:
( Attr (x y (j(()) given sample x; label y E y with pattern
sim max SeT.vT1160) AsimeLIP (den) (1 A)simcuir Odist (
21

---

Published as a conference paper at ICLR 2025
Before Training VR
Descriptive
Attributtes
Top
tall, sturdy stem reaching up to 3 in height.
|Globe Thistle]
Top
striking plant that can up to 4 feet tall.
Top
perennial plant that grows up feet tall.
Distinctive Attribuues
Top
has a unique appeurunce that sets apart from other flowers.
Top
has distinct appearance that it apart from other flowers.
Top
has a distinctly unique appearance compared to other flowers.
After Training VR "GGlobe Thistle]
Descriptive Attribuutss
Top
plant that grows up to 3 feet tall and 12-24 inches wide.
Top
is striking flowering plant belonging to the A steraceee family.
Top
a perennial plant grows up to feet tall.
Distirative Attribultss
Top
its unique spherical shape, which sets apart from other flowers
Top
peremnial herbaceous plant that is easily distinguishable from other
Top
a distinctive spherical flower head that measures about 2 inches diameter.
Before Training VR |Globe Thistle]
Desciplive Attributes
Top
is striking flowering plant belonging to the Asteraccac family. striking plant that can grow to 4 feet tall.
Top
Top 3
has a tall, sturdy stem reaching 3 feet in height.
Distinctive Attributes
Top
has a distinct appearance that scts it apart from other flowers.
Top
has unique appeurance that sets apart from other flowers.
Top
unique spherical head made of tightly packed, spiky, purple-blue flowers.
A After Training VR |Globe Thistle]
Descriptive Attiibuts
Top
and spiky perennial plant that typically grows to be 2-3 feet height.
Top
a striking flowering plant belonging the steraceae family.
Top
eye- eye-catching plant that has a spherical shape with spiky, round flowers.
Distinctive
Attiibuttss
Top
numerous globe-shaped flower heads, each with spiky blue or purple petals.
Top 2
its unique spherical shape, which sets it apart from other flowers
Top
distinctive spherical flower head that measures about 2 Inches in diameter.
Figure 9: Visualization of images with AttVR patterns, and their nearest 3 DesAttrs and DistAttrs before and after training VR patterns, using the ViT-B16-basee CLIP as the pre-trained model. Two images labeled "Globe Thistle' from the Flowers task chosen examples. The closest DesAttr and DistAttr after training convergence of VR patterns are highlighted with red borders.
where Ah=1 (a (17188)) (8)()) setting k 1.
{ades} and Āk=1 (zf (o:."""" (( { can be obtained with Eq. (8)
The average similarity (arg). In this experiment, do not compare the similarity between vari-
ous attributes and individual samples; instead, we simply calculate the average similarity between all DesAttrs, DistAttrs and a single sample. The specific logits output simavg Astr (o: y (((e)) given
sample x.! label y² E y/T with pattern g(e) can be expressed as:
sim wrg (xT , F (((())
Σ T
a(g(e)) L (2 +
1
Σ
simCLIP ( aal,.a'88 d'(8(o)),
m
a' d'AAaaa (VT)
22

---

Mean attribute similarity (mean). In this experiment, remove the knn attribute selection mod- uel. Instead, we first compute the text embedding centers Ty and (y") for the DesAttr and DistAttr sets of label T E yT. We then these centers to calculate the similarity with Published as a conference paper at ICLR 2025
Flowers
Texture
Aircraft
95
68
38
Accuraey (%)
Accuracy (%)
Accurucy
90
64
34
88
62
60
max
uVg.
meca
rnd
knn
max
avg
mean
mnd
knn
max
avg
mean
md
knn
Figure 10: Results of aggregation studies, using the ViT-B16- CLIP as the pre-trained model.
'Max' calculates the maximum attribute similarity, 'avg" calculates the average attribute similarity, 'mean' calculates the similarity of the mean attribute, 'rnd" calculates the average similarity of randomly selected attributes, and 'knn" represents the k-nearest iterative updating strategy applied in AttrVR.
where (y)) and A are attributes set with a size of m, generated by Eq. (7).
The random similarity (rnd). In this experiment, we simply calculate the average similarity be- tween & randomly selected DesAttrs, DistAttrs and a single sample, where k is the same hyper- Atte
parameter applied in Att.VR. Similarly, the logits output sim (x. .y.y"" ((()()) given sample s!. label y" E EVT with pattern 8(e) can be expressed as:
sim
and Attr (x. y (p(e) =
h Σ Ageand( (D))
simCLIP(T, (1)
A
Σ
(15)
k
d'Ezand(AAtie (.)).
where Ades (V") and Ayis()") attributes set with size of m, generated by Eq. (7) and rand(A, k)) is the random sampling function that chooses k items from set A.
each sample in the downstream task and update the VR pattern 5(r) accordingly. The logits output simm (x.,v²" (((e)) of sample z!f and label y² can be formulated as:
simmean (z. (2 (()))) = >simeLIr(E Bies ())(())) + (1 A)simelar ( Zaist (v (v))))( ())))),, where Zaes ()=) 1
Σ
(ax, Zdiat (y) m
Σ faxila)..
m
(1))
d'EArtee(y))
Conclusion. The results for each module shown in Figure 10. It can be observed that the kun attribute selection module in the current AurVR achieves the highest accuracy. This is attributed the fact that the knn module filters out redundant irrelevant feature descriptions, while also determining the sample class based on the nearest attributes, thereby enhancing the robustness of classification.
C.4 CROSS TEST BETWEEN LABEL-BASED VR AND ATTRVR
To demonstrate that the improvement in downstream task accuracy with AttrV is due to the VR learning guided by attributes, rather than solely the attributes themselves enhancing the zero-shot accuracy during test time, designed in this section.
test between the label-based VR method and Att
We conduct the following experiments: (1) Label: Adding the VR pattern learned from labels the images and classifying the images using the labels; (2) Label2Attr: Adding the VR pattern learned from labels to the images and classifying the images using the DesAttrs and DistAttrs presented in
23

---

Published as a conference paper at ICLR 2025
Flowers- Cross Tests
Texture Cross Tests
Aireraft Cross Tests
70
Accuracy (%)
Accuracy (%)
Accuracy (%)
60-
80
30-
50
70-
40
20
Label
Label2Attr
Aitr
Label
Attr2Label
Label2Attr
Attr
Label
Attr2Label
LabelZAttr
Attr
Attr2L
Cross test results between label-based VR and out AttrVR, using the ViT-B16-based CLIP as the pre-trained model. 'Label' and 'Attr' respectively show the results of label-based VR (i.e., AR) and AttRR. 'Aur2Label" shows the results of classifying downstream images with the Aut pattern using a template-prompted label, whereas 'Label2Attr' shows the results of classify- ing downstream images with the VR pattern trained by the label-based method using DesAttrs and DistAttrs.
Figure 11:
this paper; (3) Attr2Label: Adding the VR pattern learned from Att to the images and classifying the images using the labels; (4) Attr: Adding the VR pattern learned from AttrVR to the images and classifying the images using our DesAttrs and DistAttrs.
The experimental results are shown in Figure 11. By analyzing the results, we can draw the following two conclusions:
The performance improvement of Att arises from the VR learning process based on attributes, rather than from the zero-shot classification performance gains obtained during testing by using attributes. It is evident that the results from label2Attr are significantly worse than those from Aut,R, and even slightly inferior to those from the label-based VR method. Therefore, it can be concluded that the major contribution comes from the VR learning process in Attr rather than the attributes used during testing.
The VR patterns learned through lahel-based VR and Atr are not interchangeable. The accuracy observed in the cross test (refer to Label2Attr and Aur2Labell in Figure 11) is significantly lower than the accuracies obtained from both label-based VR and AltrVR. This demonstrates that the two VR patterns are not generalizable repurposing VLMSs using attributes and repurposing VLMs using template-prompted labels are fundamentally different approaches. This further highlights the innovation of Att.VR.
one another. Thus,
C.5 ANALYSIS OF TRAINING COST
C.5.1 TIME COST COMPARISON BETWEEN BASELINES AND ATTRVR
Table 9:
Training cost of different VR methods, using the ViT-B16-based CLIP as the pre-trained model and the Flowers task as an example.
VP
AR
AttrVR
Parameter Number
69.8k
39.9k
39.9k
Training Time for each Epoch (s)
2.9740.02
2.8540.03
2.83+0.03
Training Time in Total (min)
9.78+0.07
9.44+0.05
9.54+0.04
This section provides summary of the parameter counts and runtime for different VR methods. Experiments are conducted on a single A100 GPU.
The VP (Bahng et al., 2022) method employs noise pattern with frame size of 30. For input image size of 224 x 224, the parameter count is calculated as 224 x 224 x 3 (224 - 60) x (224- 60) 3 69840. In contrast, both AR (Chen al., 2023; Tsai et al., 2020) and ARt (ours) use a noise pattern with a frame size of 16, resulting parameter count of 224 x 224 x 3 (224 32) (224- 32) 3 39936.
24

---

Published as a conference paper at ICLR 2025
Detailed results presented in Table 9. We can draw the following conclusions:
Compared to VP and AR, the additional time incurred by our AttrVR involves calculating the text embeddings for all DesAttrs and DistAttrs once, as well as the knn attributes selection once per epoch (see Algorithm 1). However, Table 9 shows that both the time for a single epoch and the total training time are similar between AttrV and AR. Therefore, the extra time overhead introduced by AttrVR can be considered negligible.
In comparison to VP, AR and Att have fewer parameters, resulting in lower overall training time.
C.5.2 TIME COST OF ATTRVR USING SELECTING MODULES
Table 10: Training cost different selecting modules of AttrVR using the ViT-B16-based CLIP as the pre-trained model and the Flowers task example.
Atte knn)
AttrR (w mean)
Att.VR (w avg)
AttrVR (w max)
AttrVR (w md)
(ours)
Batch Forward Time (ms)
7.67#0.56 2.88+0.08
7.57=0.43
7. 7.30=0.04
16.63=0. 10 2.8640.03
7,39=0.00 0.00
Training Training Time in Total (min)
Epoch (s)
2.8640.04
9.5540.12
9.51=0.07
9.53+0.05
9.7540.12
9.5420.04
Table 10 shows the time for using different selection modules in AttVR (i.e., mean, avg. max, rnd, knn, see Appendix C.3 for module details). It is observed that the additional computational overhead introduced by the knn attribute selection is negligible, though it requires sorting and averaging the nearest attributes for each sample. The reason is shown below:
Assuming there
samples and m attributes for each class, the time complexity for computing the mean or maximum is ((n)), while the complexity for sorting and averaging the top A: attributes is O(n(m (m log m k)). Since feature selection does require training, the difference between these complexities is insignificant. In our case, m 20, k = 3, making the computational difference almost negligible. Moreover, when compared to the computational cost of the CLIP forward pass, these overheads tend to be trivial.
C.6 IMPACT OF SHARED AND DATASET-SPECIRS HYPER-PARAMETER
Table 11: Differences between shared and dataset-optimizeed hyper-parameterrs (using ViT-16-based CLIP the example).
Aircruft
Caltech
Cars
DTD
ESAT
Flowers
Food
Pets
SUN
UCF
Resise
AR (Baseline)
31.7
95.5
62.0
93.4
85.8
85.2
92.7
67.9
78.1
81.6
AtrVR (I=3, A=0.5)
36.6
95.7
68.3
65.6
93.8
92.9
85.9
93.3
69.6
79.0
69.4
82.6
AtteR (optimized k))
36.6
95.8
68.6
65.6
93.8
92.9
85.9
93.3
69.7
79.0
69.5
82.8
Att (optimized X)
37.0
95.9
68.5
66.0
93.8
92.9
85,9
93.3
70.0
79.0
82.6
optiaset-ptimized
3
3
3
3
Difference Between
Shared and Specific
0.0
-0.1
-0.3
0.0
0.0
0.0
0.0
0.0
-0.1
0.0
-0.1
-0.2
optasee-optimized
0.75
0.25
0.75
0.25
0.5
0.5
0.5
0.5
0.25
0.5
0.25
0.5
Difference Between
Shared and Specific
-0.4
-0.2
-0.4
0.0
0.0
0.0 0.0
0.0
-0.1
0.0
Further experiments are conducted where we select the optimal & and À for each dataset and compare them with shared papee--prraettrss & and A. The optimized hyyer-parameen values, performance, and accuracy differences are presented in Table 11. As shown, the differences between the optimal k and À for each dataset and shared value 3, À 0.5 minimal. Therefore, believe that our choice for shared and can be widely used across datasets.
C.7 MORE RESULTS OF USING DIFFERENT LLMS
also feasible to replace the GPT-3.5 LLM model used by our AttVR with smaller or open- source L.Mss. Even in scenarios where L.MMs unavailable, the VR training and knn selection
25

---

Published as a conference paper at ICLR 2025
Table 12: example).
Results of using other LLMs generate attributes (using ViT-16-based CLIP as the
Baseline: AR
Ours: AurVR
Handeraft Prompts
Phi 3.1
Llama 3.1
GPT-4o
GPT-3.5
LLMs
Mini 128k
Mini
Mini
Turbo (ours)
Open-source
Yes
Yes
No
No
Parameters
3,8B
NA
NA
Aireraft (accuracy,
31.7=0.3
33.6 6.0.8
35.5+0.5
35.7=0.3
36.8+0.9
36.6:0.3
DID faccuracy, %)
62.0e0.1
63.0+0.7
65.1+0.6
64.8+0.7
65.9+0.9
65.6++.88
Flowers (uccuracy, %
85.9=0.7
87.5+0.6
89.9=0.5
90.1=0.1
92.7+0.1
92.9=0.4
modules in our Att can still be utilized to obtain optimized handcrafted prompts (Radford et al., 2021) with labels for individual samples, thereby improving baseline performance. Moreover, the attribute generation process by LLMs is independent of training, which means it does not introduce aspect.
additional training time overhead, and whether the LLM is open-source or not does affect this
Table 12 shows the impact of different attribute generation methods: (1) baseline method AR (with- out attributes), (2) handcrafted prompts with labels (without LLMs), (3) LLM Phi 3.1 Mini 128k, (4) LLM Llama 3.1 Mini, (5) LLM GPT-4o Mini, (6) GPT-3.5 Turbo (used in AR).
The following conclusions can be drawn from Table 12:
Even when using only handerafted prompts with labels without LLM-ggnnrated attributes, AttrVR still outperforms the baseline.
Using smaller or open-source models can also achieve significant improvements compared to the baseline. So choosing which LLM use might not be that important.
Higher-quality LLMs (such as the GPT series) tend to yield slightly better results, making them more suitable for generating attributes.
C.8 THE CASE OF GENERATED ATTRIBUTES WITH LOW QUALITY
Table 13:
Results of randomly-chosen attributes and knn-chosen (ours) attributes facing attributes
with different quality (using ViT-16-based CLIP the example)
Normal-quality Attributes
Low-quality Attributes
High-quality Attributes
Aircraft
DTD
Flowers
Food
SUN
Cars
ESAT
UCF
Resisc
Caltech
Pets
AR (Baseline)
31.7
62.0
85.8
85.2
67.9
66.0
68.0
93.4
78.1
81.6
95.5
92.7
AurVR rnd)
36.4
64.3
90.5
85.6
68.3
67.8
92.4
77.6
80.9
96.2
93.6
AutVR (w knn)
36.6
65.6
92.9
85.9
69.6
69.4
68.3
93.8
79.0
82.6
95.7
93.3
Some of the generated attributes might have low qualities, and may negatively impact model perfor- mance. However, since the knn attribute selecting module in AttrR designed to select the most relevant attributes for each sample, it effectively filters out low-quality ones, ensuring the quality and relevance of the final selected attributes.
In Table 13, we present a comparison between attributes randomly selected from the generated set (marked with rnd') and those selected using our knn module (marked with 'w knn') ).n). When the generated attributes are of low quality, it is likely that randomly selecting attributes might yield worse results than not using attributes (e.g. Cars, AS UCF, Resisc dataset). However, the knn module in AttrVR effectively selects high-quality and relevant attributes, leading to results that outperform the baseline method. On the contrary, when the attributes are of sufficiently high quality (e.g, Caltech, Pets), randomly selecting attributes may already achieve a high accuracy. Then the advantages of our knn module might diminish. In most cases, km module successfully helps to ensure that only attributes with higher quality and relevance will be considered.
26

---

Published as a conference paper at ICLR 2025
C.9 MORE RESULTS OF USING MULTIMODAL LARGE LANGUAGE MODEL (MLLM) INSTEAD OF LLMS
Table 14: Results of using MLLM instead of LLMs generate attributes (using ViT-16-basee CLIP as the example).
Aircraft (accuracy. %)
DTD (accuracy. %
Flowers (accuracy. %)
AR (Baseline Method) ARt (GPT-4o-mini-LLM) AttVR (GPT-to-mini-VLMM)
31.7+0.3
62.0+0.
85.9+0.7
36.8+0.9
65. 65.9+0.9
92.7=0.1
36.6+0.4
68.2+0.3
93.7=0..4
The Multimodal Large Language Model (MLLM) can generate attributes and integrate seamlessly into our AttrVR framework by replacing the LLMs. In this section, the experiment is conducted using the VLM module GPT-4o-mini to generate attributes per class based on five randomly sampled images with the size of 224 x 224 from the training set and the prompts below.
DesAttr: This is photo of [Class Name]. Describe the appearance of the [Task Info.] [Class Name].
DistAttr: This is a photo of [Class Name]. Describe the unique appearance of a an [Class Name] from the other [Task Info.].
where [Task Info.] represents the description of downstream tasks, shown in Table 4, and [Class Name] represents the name of each target class.
Comparative results with generating attributes using the LLM module in GPT-4o-mini presented Table 14, from which we draw the following conclusions:
VLM--generaed attributes yield better results in visually distinctive tasks (e.g., texture in DTD dataset). while their advantage over LLM-generated attributes diminishes in tasks with subtle visual differences (e.g., different models of airplanes in Aircraft dataset).
AttVR outperforms the baseline in all cases, using either L.M-ggnneraed or VLM- generated attributes, demonstrating its robustness regardless of the attribute generation model.
27


Reviews:
[{"review_id": "rJzdV1vmzH", "reviewer": "Reviewer_uGif", "summary": "This paper introduces a new method called Attribute-based Visual Reprogramming (AttrVR) for CLIP-based image classification tasks. Unlike traditional label-based visual reprogramming methods, AttrVR fully leverages CLIP's rich attributes and diverse textual representation capabilities. The method introduces Descriptive Attributes (DesAttrs) and Distinctive Attributes (DistAttrs), which represent the common and unique features of different categories, respectively. Additionally, AttrVR achieves more dynamic and sample-specific optimization by iteratively optimizing the $k$ nearest DesAttrs and DistAttrs for each image sample. Theoretical analysis shows that AttrVR can reduce intra-class variance and increase inter-class distance, thereby improving classification performance. Experimental results demonstrate that, whether using ViT-based or ResNet-based CLIP, AttrVR outperforms other VR methods across 12 downstream tasks. Visualization results further confirm the effectiveness of AttrVR in enhancing image-attribute alignment. In summary, AttrVR not only improves CLIP's performance in downstream image classification tasks but also provides a more effective integration scheme for transitioning visual reprogramming techniques from unimodal visual models to multimodal visual-language models.", "questions": "1. How significantly would limited or unavailable access to these large language models impact the method's effectiveness?\n\n2. Is it possible to use smaller or open-source language models to generate attribute descriptions to reduce dependence on large models?\n\n3. Does the paper provide mechanisms to assess and ensure the quality and relevance of attribute descriptions generated by large language models?\n\n4. If the generated top attributes do not match the target categories, how does it impact model performance? \n\n5. Has consideration been given to validating the method's effectiveness on larger or more challenging datasets to ensure its generalizability?\n\n6. Although the AttrVR method leverages learnable noise, how does its performance improve in more complex image scenarios?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper addresses the limitations of Visual Reprogramming (VR) in vision-language models like CLIP by introducing Attribute-based Visual Reprogramming (AttrVR). Unlike traditional VR methods that rely on fixed text templates and category labels, AttrVR leverages Descriptive Attributes (DesAttrs) and Distinctive Attributes (DistAttrs) generated by large language models (e.g., GPT-3.5) to capture both common and unique features of each category. During training, AttrVR dynamically selects relevant attribute descriptions for each image, optimizing input noise patterns to better capture detailed features and reduce category confusion. This attribute-based approach significantly improves classification performance across 12 downstream tasks with ViT and ResNet backbones, even with limited training samples, demonstrating robustness and effectiveness. Additionally, theoretical analysis and visualizations show that AttrVR reduces intra-class variance and enhances inter-class separability, thereby enriching the model's embedding space and offering new insights for advancing vision-language model applications. Overall, AttrVR effectively combines the strengths of visual and language models, addressing the shortcomings of label-based VR methods and fully utilizing CLIP's capability to understand rich textual descriptions, thereby advancing the application of vision-language models in downstream tasks.", "weaknesses": "1. **Dependence on Large Language Models:** The AttrVR method relies on large language models (e.g., GPT-3.5) to generate descriptive and distinctive attributes. This dependence may increase computational costs and limit the method's applicability in resource-constrained environments.\n\n2. **Reliability of Attribute Generation:** The attribute descriptions generated by large language models may have issues with accuracy and relevance. If the generated attributes do not match the target categories, they could negatively impact the model's performance.\n\n3. **Computational Complexity and Efficiency:** AttrVR queries 𝑘 nearest neighbor attributes for each image sample in every training cycle, increasing computational overhead. This may impose pressure on training time and resource requirements for large datasets.\n\n4. **Scope of Experimental Validation:** The paper primarily conducts experiments on 12 downstream tasks. It remains unclear whether the method's effectiveness can be validated on larger or more challenging datasets.\n\n5. **Impact of Hyperparameter Selection:** Hyperparameters in AttrVR (e.g., the number 𝑘 of nearest neighbor attributes) significantly affect model performance. While the paper selects the optimal 𝑘=3, it does not demonstrate whether this choice is equally suitable in more general scenarios across different datasets, failing to prove the method's reliability in various practical environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DkdFsJmav2", "reviewer": "Reviewer_UhVi", "summary": "This paper proposes an adaptation technique to improve pre-trained vision-language models for downstream classification tasks. The method uses visual reprogramming, a technique that alters the space of the input to the model – that is a learnt padding. This is done by leveraging the vast knowledge of LLMs to generate descriptive details, thus allowing to harness the potential of the joint embedding training of vision-language models, unlike prior works that only employ the class label used in the cross entropy objective. Distinctive attributes and descriptive attributes of classes together with a few clever training time tricks improves the downstream classification accuracy of VLMs.", "questions": "- According to the equation for Class Separability (CS), CS would be maximized when the intra-class variance is maximized, and the inter-class distance is minimized. Can the authors clarify if this is an error in the equation (requiring a negative sign) or if there's a reason for defining CS this way that isn't apparent to me?\n- AttrZS in Table 1 is unclear? How are the attributes used for zero-shot classification? The authors can add a brief explanation of this for clarity.\n- Tables 5 and 6 incorrectly mention the backbone as ViT-RN. I recommend the authors carefully review all table headers and labels for consistency and accuracy throughout the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a neat incorporation of the vast knowledge of LLMs to improve downstream classification of vision language models. The use of descriptive and distinctive attributes of classes to leverage the language capability in vision-language joint pre-training is well validated through experimental results and theoretical backing.\n- The proposed method achieves excellent improvements above existing visual reprogramming methods across a diverse set of visual classification datasets and different CLIP backbones.", "weaknesses": "- Is there a way to ascertain that the distinctive captions generated are actually distinct in the embedding space? While lines 78-79 claims this is the case in Figure 1, I disagree. The distinctive attributes seem to not necessarily be farther from the cluster center compared to descriptive attributes. The authors could compare of cosine distance of descriptive attributes and distinctive attributes with the class labels. Ideally, there should be a higher distance from other classes for distinctive attributes than descriptive attributes.\n- It would be interesting to see an ablation on the use of kNN for the attributes. How does it compare to randomly sampling k attributes.  \n- Line 167 wrong citation. I believe the intended citation is [1]. A kind suggestion for authors to double-check all their citations for accuracy\n\n[1] Cai, Chengyi, et al. \"Sample-specific Masks for Visual Reprogramming-based Prompting.\" arXiv preprint arXiv:2406.03150 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LrtHax3uAy", "reviewer": "Reviewer_ELTS", "summary": "The paper introduces Attribute-based Visual Reprogramming (AttrVR), a method for enhancing image classification by leveraging descriptive (DesAttrs) and distinctive attributes (DistAttrs) within vision-language models like CLIP. AttrVR replaces traditional label-based reprogramming with these attributes, refining classification through sample-specific, attribute-guided adjustments that improve intra-class consistency and inter-class separability. The approach demonstrates superior performance across multiple benchmark datasets, offering a more dynamic and attribute-focused alternative for repurposing pre-trained models in diverse downstream tasks.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written, which provides theoretical insights showing how AttrVR reduces intra-class variance and increases inter-class separation, enhancing the model’s discriminative power.\n2. The motivation for this submission is easy to understand, which leverages descriptive and distinctive attributes instead of traditional label prompts, maximizing CLIP’s multimodal capabilities to improve classification accuracy.\n3. AttrVR is tested across multiple datasets, with results indicating superior performance over existing methods in various visual classification tasks, especially in fine-grained categories.", "weaknesses": "In general, the motivation for this submission is easy to understand and insight is interesting. However, there are still several weaknesses, as follows:  \n1. While the paper mentions using GPT-3.5 for generating descriptive and distinctive attributes (DesAttrs and DistAttrs), it lacks detailed reasoning on why GPT-3.5 was specifically chosen over other potential models. Further, the method relies heavily on the accuracy and quality of DesAttrs and DistAttrs generated by a language model. However, there is limited discussion on how inaccuracies or noise in these attributes might affect the model's performance.  \n2. The effectiveness of the model appears sensitive to hyperparameters like λ (balance between DesAttrs and DistAttrs) and k (the number of nearest attributes). While the authors conduct a hyperparameter analysis, this could raise concerns regarding the robustness of the model in practical settings where optimal hyperparameters may vary widely across datasets.  \n3. While standard contrastive learning ablation is provided, the study lacks a comparison between hard negatives and simple negatives. Additional experiments could assess performance when hard negatives are removed, highlighting whether they significantly enhance alignment and fine-grained recognition capability.  \n4. Although the k-NN aggregation outperforms other methods like max, avg, and mean aggregation in attribute selection, the iterative querying process for the k-nearest attributes per sample per epoch raises concerns about computational efficiency and scalability, especially for large datasets. In addition, the sample-specific optimization approach using k-NN may cause the model to overfit to specific samples instead of capturing generalizable features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)