ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models

Authors: Seonghwan Park, Jaehyeon Jeong, Yongjun Kim, Jaeho Lee, Namhoon Lee
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: vision-language models, prompt-tuning, black-box optimization, zeroth-order optimization

Abstract:
Recent studies have introduced various approaches for prompt-tuning black-box vision-language models, referred to as black-box prompt-tuning (BBPT). While BBPT has demonstrated considerable potential, it is often found that many existing methods require an excessive number of queries (i.e., function evaluations), which poses a significant challenge in real-world scenarios where the number of allowed queries is limited. To tackle this issue, we propose Zeroth-order Intrinsic-dimensional Prompt-tuning (ZIP), a novel approach that enables efficient and robust prompt optimization in a purely black-box setting. The key idea of ZIP is to reduce the problem dimensionality and the variance of zeroth-order gradient estimates, such that the training is done fast with far less queries. We achieve this by re-parameterizing prompts in low-rank representations and designing intrinsic-dimensional clipping of estimated gradients. We evaluate ZIP on 13+ vision-language tasks in standard benchmarks and show that it achieves an average improvement of approximately 6% in few-shot accuracy and 48% in query efficiency compared to the best-performing alternative BBPT methods, establishing a new state of the art. Our ablation analysis further shows that the proposed clipping mechanism is robust and nearly optimal, without the need to manually select the clipping threshold, matching the result of expensive hyperparameter search.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730367711368

thread_type:
official_review

<thread_object>
creator: reviewer_nlrg

<content>
summary:
This paper introduces ZIP, a zeroth-order intrinsic-dimensional prompt-tuning method designed to efficiently optimize black-box vision-language models. By leveraging low-rank approximation, feature sharing, and intrinsic-dimensional gradient clipping, ZIP achieves faster training speeds and superior generalization performance while significantly reducing query requirements. Extensive experiments on diverse tasks demonstrate ZIP's robustness and query efficiency, outperforming existing BBPT methods and establishing it as a practical approach for resource-constrained scenarios.

strengths:
1.The paper presents a novel black-box prompt-tuning method, effectively addressing the issue in zeroth-order methods where an increase in trainable parameters adversely impacts accuracy. By reducing the number of parameters and query requirements, the proposed approach is well-suited for practical applications with limited query budgets.

2.The paper demonstrates strong performance across three extensive and diverse experimental settings, which effectively validate the method’s efficacy. The ablation studies further support the approach, particularly highlighting that the feature-sharing technique helps preserve the model’s expressive capacity.           

3.The intrinsic-dimensional clipping mechanism in ZIP requires no manual hyperparameter tuning, making it highly practical and user-friendly.    

4.The paper is well-written, with clear explanations and logical organization that make the proposed method and its contributions easy to understand.

weaknesses:
1.Although the paper performs ablation studies on individual modules such as  low-rank approximation with a diagonal matrix and feature sharing, it lacks ablation experiments on different combinations of these modules.   Without evaluating different combinations, it is challenging to fully understand the synergistic effects and the relative contributions of each module to the overall performance.      



2.The paper lacks an ablation study to isolate the effect of low-rank approximation alone, making it unclear if improvements are mainly due to the diagonal matrix. This analysis would clarify the diagonal matrix's contribution.

questions:
Suggestions: 

The caption for Figure 1 should include citations for the baseline methods (BAR, BlackVIP, BPT-VLM) to provide appropriate references and context for these comparisons. This would enhance clarity for readers unfamiliar with these specific methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730470058615

thread_type:
official_review

<thread_object>
creator: reviewer_fqbz

<content>
summary:
The paper introduces ZIP, a zeroth-order prompt tuning method designed for efficient prompt optimization in black-box vision-language models, particularly under limited query budgets. ZIP achieves high efficiency by using low-rank representations and intrinsic-dimensional gradient clipping, which reduces query usage while maintaining robust performance. Evaluations on multiple benchmarks show that ZIP not only outperforms state-of-the-art methods in accuracy but also greatly enhances query efficiency.

strengths:
(1) The paper is well-organized and accessible, with clear visuals and structured explanations that effectively communicate the method's strengths.

(2) ZIP innovatively enhances zeroth-order prompt tuning through intrinsic-dimensional gradient clipping and low-rank parameterization, making it highly efficient.

(3) Comprehensive evaluations demonstrate ZIP's superior accuracy and query efficiency across 13+ tasks, proving its practical value under query constraints.

weaknesses:
(1) While ZIP outperforms existing BBPT methods, comparisons with additional baseline methods in zeroth-order optimization could strengthen claims of superiority.

(2) While ZIP shows strong performance on various tasks, its results on ImageNet in Table 1 are comparatively modest, suggesting limitations in scalability to complex datasets. An in-depth analysis of ZIP's performance on larger, diverse datasets would clarify its robustness and potential for broader application.

questions:
(1) In Section 4.2, the paper introduces feature sharing to enhance expressiveness. Could the authors clarify whether this feature sharing technique affects the generalization ability on unseen datasets, and if so, how?

(2) ZIP has demonstrated strong results across vision-language tasks, but could the authors provide more insights into its potential for domain generalization? Specifically, how well does ZIP adapt to unseen domains or datasets outside the evaluated benchmarks, and would any adjustments be necessary to improve its robustness in such scenarios? Such as CoOp and CoCoOp.  

(3)  Could the authors elaborate on the sensitivity of ZIP to the choice of intrinsic dimensionality and low-rank approximation parameters? How do these choices impact both performance and query efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577551701

thread_type:
official_review

<thread_object>
creator: reviewer_tcth

<content>
summary:
The paper proposes a method to optimize black-box models without the need for computing gradients (zeroth-order). The key observation is that increasing the number of learnable parameters in soft prompts hurts the performance and training speed of zeroth-order optimization, while this trend is reversed for SGD-based prompt tuning (first-order). To overcome this, authors propose to reparameterize soft prompts in order to reduce the effective number of learnable parameters while maintaining the extrinsic embedding dimensionality. The proposed reparameterization involves projecting parameters into a diagonal matrix, feature sharing and gradient clipping. In addition, reducing the number of learnable parameters results in increased query efficiency (reduced number of forward passes through the model). The proposed method is applied to black-box prompt-tuning of a CLIP model, and evaluated on a suite of standard vision-language benchmarks, achieving improvements of 6% in few-shot accuracy and 48% in query efficiency compared to the best performing existing methods.

strengths:
* Good motivation to reduce the number of learnable parameters in ZO optimization (section 3) and clever idea to reduce the intrinsic dimensionality while maintaining the number of tokens (and the extrinsic dimensionality, which is a requirement from the model being optimized).
* Several techniques (diagonal matrix, parameter sharing) are applied to preserve performance while reducing the number of learnable parameters.
* The proposed method not only improves few-shot performance wrt existing ZO methods but also reduces considerably the number of function evaluations required to reach a certain level of performance (section 5.3).
* All the design choices for the soft prompt reparameterization are thoroughly ablated in section 6.
* The paper is clearly written and easy to follow.

weaknesses:
* Authors motivate fine-tuning black-box models with the use case of improving proprietary LLMs (e.g. GPT-4, Gemini) which are only accessible through API. However, this interface only accepts text and images as input, not soft prompts or embeddings, so the proposed method would not be directly applicable to API-based models.
* To verify the method's robustness and generality, it should be evaluated on other model families such as multimodal LLMs.
* Figures 2, 4, 6 and 7a should report validation accuracy since there could be overfitting.

questions:
* It is not until the background section that I understood what zeroth-order intrinsic-dimensional prompt-tuning means. I suggest to improve the introduction to make it clearer from early on.
* In figure 2, it would be good to add a baseline of accuracy when no soft prompts are optimized (i.e. m=0).
* Where are the learned soft prompts injected? Are they concatenated to text embeddings and fed to CLIP's text encoder?
* In table 3, the average accuracies for CDT between ZIP and the second-best method seem very close. Did authors run a significance test?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730772048035

thread_type:
official_review

<thread_object>
creator: reviewer_bgp2

<content>
summary:
The paper introduces ZIP for efficient zeroth-order prompt-tuning of black-box vision-language models. ZIP addresses the challenge of excessive query requirements in existing black-box prompt-tuning methods by reducing problem dimensionality and gradient estimate variance through feature sharing and intrinsic-dimensional gradient clipping. ZIP demonstrates significant improvements in few-shot accuracy and query efficiency over other existing methods. Various experiments on image classification show the effectiveness of ZIP.

strengths:
- ZIP is well-motivated.
- The paper is well-organized.
- Empirical analyses of the proposed method are sufficient.

weaknesses:
I'm not familiar with this research field, i.e. black box prompt tuning. Therefore, it's hard for me to accurately judge the novelty of the proposed method compared with existing works. 

From my perspective, one major weakness is that I find the competitors in the experiments are slightly old, e.g. BLACKVIP is published at CVPR'23 and BPTVLM is published at IJCAI'23. There are some more recent works like [a][b] in this field. I think the authors should better discuss the differences between ZIP and more recent works like [a][b], and provide fair experimental comparisons as well. 

[a] Language Models as Black-Box Optimizers for Vision-Language Models, CVPR 2024, https://openaccess.thecvf.com/content/CVPR2024/html/Liu_Language_Models_as_Black-Box_Optimizers_for_Vision-Language_Models_CVPR_2024_paper.html
[b] Connecting the Dots: Collaborative Fine-tuning for
Black-Box Vision-Language Models, ICML 2024, https://arxiv.org/abs/2402.04050

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733145193255

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Dear reviewers (`BGp2`, `Tcth`, `Fqbz`, `NLRG`) and area chair,

We sincerely appreciate the time and effort you dedicated to reviewing our paper. Your insightful feedback has been invaluable in refining our research.

We are pleased that most reviewers recognized the key strengths of our work. During the discussion period, we also provided clarifications requested by reviewers (**need validation accuracy** (`Tcth`), **adding baseline for Figure 2** (`Tcth`), **explanation of generalization** (`Fqbz`)). To further support our approach, we conducted additional ablation studies (**apply to other model families** (`Tcth`), **ablation for different combinations of modules** (`NLRG`), **feature sharing and robustness** (`Fqbz`)).

Below, we summarize the **Strengths highlighted by reviewers $\color{blue}{[S]}$**, and **Key contributions of our work $\color{cyan}{[C]}$**.

---

### **Strengths highlighted by reviewers $\color{blue}{[S]}$**
#### **Novelty** 
- “innovatively enhances zeroth-order prompt tuning through intrinsic-dimensional gradient clipping and low-rank parameterization” (`Fqbz`)
- “clever idea to reduce the intrinsic dimensionality while maintaining the number of tokens” (`Tcth`)
- “a novel black-box prompt-tuning method” (`NLRG`)

#### **Motivation**
- “good motivation to reduce the number of learnable parameters in ZO optimization” (`Tcth`)
- “ZIP is well-motivated” (`BGp2`)

#### **Evaluations**
- “comprehensive evaluations demonstrate ZIP's superior accuracy and query efficiency across 13+ tasks” (`Fqbz`)
- “improves few-shot performance wrt existing ZO methods” (`Tcth`)
- “the paper demonstrates strong performance across three extensive and diverse experimental settings” (`NLRG`)
- “all the design choices for the soft prompt reparameterization are thoroughly ablated in section 6.” (`Tcth`)
- “the ablation studies further support the approach, particularly highlighting that the feature-sharing technique helps preserve the model’s expressive capacity.” (`NLRG`)

#### **Practicality**
- “the proposed approach is well-suited for practical applications with limited query budgets” (`NLRG`)
- “proving its practical value under query constraints” (`Fqbz`)
- “the intrinsic-dimensional clipping mechanism in ZIP requires no manual hyperparameter tuning, making it highly practical and user-friendly.” (`NLRG`)

---

### **Key contributions of our work $\color{cyan}{[C]}$**
#### **Problem identification**
Black-box prompt tuning (BBPT), a method for training black-box models, is inherently restricted by limited API budgets, making query efficiency a crucial factor for its practicality. Despite this, prior works have not addressed this issue. For instance, existing works often require tens of thousands of queries, which severely limits their practicality. We are the first to explicitly identify this problem in the language-model-as-a-service scenario and propose a novel idea to significantly improve query efficiency.

&nbsp;

#### **Technical contributions**
We introduce two key technical contributions to tackle these challenges:
1. **Low-rank representation for dimensionality dependency**: To address the challenge of dimensionality dependency of zeroth-order methods (i.e., the number of queries scales with the dimensionality of the problem), we propose a novel low-rank representation. This approach reduces the dimensionality while effectively mitigating the loss of expressive power through feature sharing.

2. **Intrinsic dimensional clipping for variance reduction**: High variance in zeroth-order information can significantly degrade query efficiency. To tackle this, we propose a threshold-free gradient clipping method, termed “intrinsic dimensional clipping”. The clipping threshold is set to $\sqrt{d}$, which corresponds to the standard deviation of the zeroth-order gradient, where $d$ is the dimensionality of the problem. This approach not only reduces the variance of zeroth-order information but also achieves near-optimal performance without requiring manual tuning. 

&nbsp;

####  **Experimental results**
Our method achieves state-of-the-art performance in few-shot learning, base-to-new generalization, and out-of-distribution tasks under limited query budgets, while also delivering competitive results in cross-dataset transfer. Furthermore, it reduces the number of queries required to achieve a specific accuracy level by 48% compared to the best-performing alternative BBPT methods in few-shot accuracy. These results highlight the effectiveness of our approach in tackling key challenges in BBPT.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper tackles an important problem in black-box prompt tuning, i.e., existing methods rely on excessive queries for model update. The solutions proposed in this work include a low-rank reparameterization method for reducing learnable parameters and a gradient-clipping method for reducing the variance of zeroth-order gradients. The paper received four reviews with 3x borderline accept and 1x borderline reject. However, the "negative" reviewer indicated that he/she is not familiar with the topic and therefore not confident in his/her decision (the AC discounts this review). The rest of the reviewers are generally positive about this work: they found the paper well-motivated, the idea novel, and the performance strong.