Attribute-based Visual Reprogramming for Vision-Language Models

Authors: Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, Feng Liu
Primary area: foundation or frontier models, including LLMs
Keywords: Visual Reprogramming, Model Reprogramming, Vision-Language Models, Image Classification

Abstract:
*Visual reprogramming* (VR) reuses pre-trained vision models for downstream image classification tasks by adding trainable noise patterns to inputs. When applied to vision-language models (e.g., CLIP), existing VR approaches follow the same pipeline used in vision models (e.g., ResNet, ViT), where ground-truth class labels are inserted into fixed text templates to guide the optimization of VR patterns. This label-based approach, however, overlooks the rich information and diverse attribute-guided textual representations that CLIP can exploit, which may lead to the misclassification of samples. In this paper, we propose ***Attr**ibute-based **V**isual **R**eprogramming* (AttrVR) for CLIP, utilizing ***des**criptive **attr**ibutes* (DesAttrs) and ***dist**inctive **attr**ibutes* (DistAttrs), which respectively represent common and unique feature descriptions for different classes. Besides, as images of the same class may reflect different attributes after VR, AttrVR iteratively refines patterns using the $k$-nearest DesAttrs and DistAttrs for each image sample, enabling more dynamic and sample-specific optimization. Theoretically, AttrVR is shown to reduce intra-class variance and increase inter-class separation. Empirically, it achieves superior performance in 12 downstream tasks for both ViT-based and ResNet-based CLIP. The success of AttrVR facilitates more effective integration of VR from unimodal vision models into vision-language models. Our code is available at https://github.com/tmlr-group/AttrVR.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730465447007

thread_type:
official_review

<thread_object>
creator: reviewer_elts

<content>
summary:
The paper introduces Attribute-based Visual Reprogramming (AttrVR), a method for enhancing image classification by leveraging descriptive (DesAttrs) and distinctive attributes (DistAttrs) within vision-language models like CLIP. AttrVR replaces traditional label-based reprogramming with these attributes, refining classification through sample-specific, attribute-guided adjustments that improve intra-class consistency and inter-class separability. The approach demonstrates superior performance across multiple benchmark datasets, offering a more dynamic and attribute-focused alternative for repurposing pre-trained models in diverse downstream tasks.

strengths:
1. The paper is well-written, which provides theoretical insights showing how AttrVR reduces intra-class variance and increases inter-class separation, enhancing the model’s discriminative power.
2. The motivation for this submission is easy to understand, which leverages descriptive and distinctive attributes instead of traditional label prompts, maximizing CLIP’s multimodal capabilities to improve classification accuracy.
3. AttrVR is tested across multiple datasets, with results indicating superior performance over existing methods in various visual classification tasks, especially in fine-grained categories.

weaknesses:
In general, the motivation for this submission is easy to understand and insight is interesting. However, there are still several weaknesses, as follows:  
1. While the paper mentions using GPT-3.5 for generating descriptive and distinctive attributes (DesAttrs and DistAttrs), it lacks detailed reasoning on why GPT-3.5 was specifically chosen over other potential models. Further, the method relies heavily on the accuracy and quality of DesAttrs and DistAttrs generated by a language model. However, there is limited discussion on how inaccuracies or noise in these attributes might affect the model's performance.  
2. The effectiveness of the model appears sensitive to hyperparameters like λ (balance between DesAttrs and DistAttrs) and k (the number of nearest attributes). While the authors conduct a hyperparameter analysis, this could raise concerns regarding the robustness of the model in practical settings where optimal hyperparameters may vary widely across datasets.  
3. While standard contrastive learning ablation is provided, the study lacks a comparison between hard negatives and simple negatives. Additional experiments could assess performance when hard negatives are removed, highlighting whether they significantly enhance alignment and fine-grained recognition capability.  
4. Although the k-NN aggregation outperforms other methods like max, avg, and mean aggregation in attribute selection, the iterative querying process for the k-nearest attributes per sample per epoch raises concerns about computational efficiency and scalability, especially for large datasets. In addition, the sample-specific optimization approach using k-NN may cause the model to overfit to specific samples instead of capturing generalizable features.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730501187023

thread_type:
official_review

<thread_object>
creator: reviewer_uhvi

<content>
summary:
This paper proposes an adaptation technique to improve pre-trained vision-language models for downstream classification tasks. The method uses visual reprogramming, a technique that alters the space of the input to the model – that is a learnt padding. This is done by leveraging the vast knowledge of LLMs to generate descriptive details, thus allowing to harness the potential of the joint embedding training of vision-language models, unlike prior works that only employ the class label used in the cross entropy objective. Distinctive attributes and descriptive attributes of classes together with a few clever training time tricks improves the downstream classification accuracy of VLMs.

strengths:
- The paper presents a neat incorporation of the vast knowledge of LLMs to improve downstream classification of vision language models. The use of descriptive and distinctive attributes of classes to leverage the language capability in vision-language joint pre-training is well validated through experimental results and theoretical backing.
- The proposed method achieves excellent improvements above existing visual reprogramming methods across a diverse set of visual classification datasets and different CLIP backbones.

weaknesses:
- Is there a way to ascertain that the distinctive captions generated are actually distinct in the embedding space? While lines 78-79 claims this is the case in Figure 1, I disagree. The distinctive attributes seem to not necessarily be farther from the cluster center compared to descriptive attributes. The authors could compare of cosine distance of descriptive attributes and distinctive attributes with the class labels. Ideally, there should be a higher distance from other classes for distinctive attributes than descriptive attributes.
- It would be interesting to see an ablation on the use of kNN for the attributes. How does it compare to randomly sampling k attributes.  
- Line 167 wrong citation. I believe the intended citation is [1]. A kind suggestion for authors to double-check all their citations for accuracy

[1] Cai, Chengyi, et al. "Sample-specific Masks for Visual Reprogramming-based Prompting." arXiv preprint arXiv:2406.03150 (2024).

questions:
- According to the equation for Class Separability (CS), CS would be maximized when the intra-class variance is maximized, and the inter-class distance is minimized. Can the authors clarify if this is an error in the equation (requiring a negative sign) or if there's a reason for defining CS this way that isn't apparent to me?
- AttrZS in Table 1 is unclear? How are the attributes used for zero-shot classification? The authors can add a brief explanation of this for clarity.
- Tables 5 and 6 incorrectly mention the backbone as ViT-RN. I recommend the authors carefully review all table headers and labels for consistency and accuracy throughout the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731070604085

thread_type:
official_review

<thread_object>
creator: reviewer_ugif

<content>
summary:
This paper introduces a new method called Attribute-based Visual Reprogramming (AttrVR) for CLIP-based image classification tasks. Unlike traditional label-based visual reprogramming methods, AttrVR fully leverages CLIP's rich attributes and diverse textual representation capabilities. The method introduces Descriptive Attributes (DesAttrs) and Distinctive Attributes (DistAttrs), which represent the common and unique features of different categories, respectively. Additionally, AttrVR achieves more dynamic and sample-specific optimization by iteratively optimizing the $k$ nearest DesAttrs and DistAttrs for each image sample. Theoretical analysis shows that AttrVR can reduce intra-class variance and increase inter-class distance, thereby improving classification performance. Experimental results demonstrate that, whether using ViT-based or ResNet-based CLIP, AttrVR outperforms other VR methods across 12 downstream tasks. Visualization results further confirm the effectiveness of AttrVR in enhancing image-attribute alignment. In summary, AttrVR not only improves CLIP's performance in downstream image classification tasks but also provides a more effective integration scheme for transitioning visual reprogramming techniques from unimodal visual models to multimodal visual-language models.

strengths:
This paper addresses the limitations of Visual Reprogramming (VR) in vision-language models like CLIP by introducing Attribute-based Visual Reprogramming (AttrVR). Unlike traditional VR methods that rely on fixed text templates and category labels, AttrVR leverages Descriptive Attributes (DesAttrs) and Distinctive Attributes (DistAttrs) generated by large language models (e.g., GPT-3.5) to capture both common and unique features of each category. During training, AttrVR dynamically selects relevant attribute descriptions for each image, optimizing input noise patterns to better capture detailed features and reduce category confusion. This attribute-based approach significantly improves classification performance across 12 downstream tasks with ViT and ResNet backbones, even with limited training samples, demonstrating robustness and effectiveness. Additionally, theoretical analysis and visualizations show that AttrVR reduces intra-class variance and enhances inter-class separability, thereby enriching the model's embedding space and offering new insights for advancing vision-language model applications. Overall, AttrVR effectively combines the strengths of visual and language models, addressing the shortcomings of label-based VR methods and fully utilizing CLIP's capability to understand rich textual descriptions, thereby advancing the application of vision-language models in downstream tasks.

weaknesses:
1. **Dependence on Large Language Models:** The AttrVR method relies on large language models (e.g., GPT-3.5) to generate descriptive and distinctive attributes. This dependence may increase computational costs and limit the method's applicability in resource-constrained environments.

2. **Reliability of Attribute Generation:** The attribute descriptions generated by large language models may have issues with accuracy and relevance. If the generated attributes do not match the target categories, they could negatively impact the model's performance.

3. **Computational Complexity and Efficiency:** AttrVR queries 𝑘 nearest neighbor attributes for each image sample in every training cycle, increasing computational overhead. This may impose pressure on training time and resource requirements for large datasets.

4. **Scope of Experimental Validation:** The paper primarily conducts experiments on 12 downstream tasks. It remains unclear whether the method's effectiveness can be validated on larger or more challenging datasets.

5. **Impact of Hyperparameter Selection:** Hyperparameters in AttrVR (e.g., the number 𝑘 of nearest neighbor attributes) significantly affect model performance. While the paper selects the optimal 𝑘=3, it does not demonstrate whether this choice is equally suitable in more general scenarios across different datasets, failing to prove the method's reliability in various practical environments.

questions:
1. How significantly would limited or unavailable access to these large language models impact the method's effectiveness?

2. Is it possible to use smaller or open-source language models to generate attribute descriptions to reduce dependence on large models?

3. Does the paper provide mechanisms to assess and ensure the quality and relevance of attribute descriptions generated by large language models?

4. If the generated top attributes do not match the target categories, how does it impact model performance? 

5. Has consideration been given to validating the method's effectiveness on larger or more challenging datasets to ensure its generalizability?

6. Although the AttrVR method leverages learnable noise, how does its performance improve in more complex image scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733231565111

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary after the rebuttal: Responses and Revisions (Part I)

comment:
We appreciate the thorough feedback from all reviewers and have addressed each concern through additional experiments and clarifications. We outline our key responses and improvements to the manuscript in this summary.

**Responses to Reviewer uGif**

1. **Dependence on Large Language Models (LLMs)**
- Concerns: Can smaller or open-source LLMs be used? What happens if LLMs are unavailable?
- Response: We demonstrate that AttrVR is compatible with *smaller* (GPT-4o mini) or *open-source* (Phi 3.1 Mini 128k, and Llama 3.1 Mini) LLMs, and remains effective even *without LLMs* by utilizing handcrafted prompts with labels. We **add a section (Appendix C.7)** to include the results from these additional experiments.

2. **Reliability of Attribute Generation**
- Concerns: How is attribute quality assessed? What is the impact of low-quality attributes?
- Response: We demonstrate that low-quality attributes would negatively affect VR training but **not** AttrVR performance, due to the $k$-nn attributes selection module proposed to ensure attribute quality (on Page 6). We **add a section (Appendix C.8)** to include a comparative analysis between $k$-nn and random attribute selection, confirming its efficacy.

3. **Computational Complexity of $k$-nn**
- Concerns: would $k$-nn introduce significant computation overhead?
- Response: We have measured time per batch, epoch, and total training time, confirming that $k$-nn selection results in negligible overhead **(detailed in Appendix C.5)**.

4. **Scope of Experiments**
- Concerns: Does AttrVR generalize across datasets and handle complex image data?
- Response: We explain that our empirical studies follow commonly used benchmarks in CLIP-related literature, span 12 diverse datasets covering scenes, actions, textures, and fine-grained details, as well as ImageNet, which may have already constituted a sufficiently large and challenging benchmark dataset.

5. **Hyperparameter Sensitivity**
- Concerns: How sensitive is AttrVR to hyperparameter $k$?
- Response: We conduct additional experiments to show hyperparameter-induced performance variations are under 0.3%. We **append these results in Appendix C.6**.

6. **Inference Process**
- Concerns: Does the inference of AttrVR require generating descriptions for each image through LLM?
- Response: We **clarify** that AttrVR precomputes attribute descriptions *before* training, with inference involving a standard forwawrd pass. We **include timing details in Table 10**.

7. **Model Replacement**
- Concerns: Can Large Vision-Language Models be used to generate attributes?
- Response: We **report additional results in Appendix C.9** to show that AttrVR can seamlessly integrate multi-modal LLMs (e.g., GPT-4o-mini-VLM) and demonstrates robustness **regardless of the attribute generation model**.

8. **Follow-up Clarifications**

- We explain the definitions of DesAttr and DistAttr, showcase specific examples and conclude their differences.
- We clarify that AttrZS (zero-shot version of AttrVR) requires no retraining and uses a single CLIP forward pass, which **ensures scalability**.
- We clarify that our method ensures full reproducibility, and the code is already provided via an anonymous link.
- We clarify that AttrVR addresses the limitations in VLMs arising from domain shifts and distribution discrepancies, such as EuroSAT. We highlight that AttrVR advances VLM alignment and distribution adaptation, contributing to foundational research in improving model robustness across diverse tasks.



**Responses to Reviewer UhVi**

1. **Distinctiveness of attributes in Embedding Space**
- Concerns: Are DistAttr embeddings distinct from DesAttr?
- Response: We **calculate average cosine distances** to confirm DistAttr is farther from label embeddings than DesAttr. We **include the visualization** of these results in Figure 1.

2. **Performance of $k$-nn Attribute Selection**
- Concerns: How does $k$-nn compare to random attributes?
- Response: We **conduct experiments** (Appendix C.8) and **add comparative analysis** (Appendix C.3) to demonstrate that $k$-nn selection ensures higher quality attributes are used in optimization and prediction.

3. **Typos and Clarifications**
- We have corrected typos and explained that AttrZS (Table 1) is the zero-shot version of AttrVR where no training is performed, **in Appendix A.3**.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:
This paper introduces a new method called Attribute-based Visual Reprogramming (AttrVR) for CLIP-based image classification tasks. The experimental results on 12 downstream tasks are given.

The main strengths include 1) strong experimental performance, and 2) well-motivated idea.  
The main weaknesses are 1) lack of ablation study on the setting of KNN, and 2) lack of evaluation on the Complexity of the k-nn module in AttrVR.  

All reviewers recognized the strengths of leveraging descriptive and distinctive attributes to maximize CLIP’s multimodal capabilities, and the issues raised by the reviewers were well addressed by the authors in the discussion stage.  
The AC agrees with the reviewers and recommends accepting this paper, but the authors should consider the reviewers' suggestions to include necessary experimental comparisons/analyses.