GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models

Authors: Muhammad Jehanzeb Mirza, Mengjie Zhao, Zhuoyuan Mao, Sivan Doveh, Wei Lin, Paul Gavrikov, Michael Dorkenwald, Shiqi Yang, Saurav Jha, Hiromi Wakaki, Yuki Mitsufuji, Horst Possegger, Rogerio Feris, Leonid Karlinsky, James R. Glass
Primary area: foundation or frontier models, including LLMs
Keywords: llms, vlms, prompt optimization

Abstract:
In this work, we propose a novel method (GLOV) enabling Large Language Models (LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to enhance downstream vision tasks. 
Our GLOV meta-prompts an LLM with the downstream task description, querying it for suitable VLM prompts (e.g., for zero-shot classification with CLIP). 
These prompts are ranked according to a purity measure obtained through a fitness function. 
In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of text prompts preferred by the downstream VLM. 
Furthermore, we also explicitly steer the LLM generation process in each optimization step 
by specifically adding an offset difference vector of the embeddings from the \textit{positive} and \textit{negative} solutions found by the LLM, in previous optimization steps, to the intermediate layer of the network for the next generation step.
This offset vector steers the LLM generation toward the type of language preferred by the downstream VLM, resulting in enhanced performance on the downstream vision tasks. 
We comprehensively evaluate our GLOV on 16 diverse datasets using two families of VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models --
showing that the discovered solutions can enhance the recognition performance by up to $15.0$% and $57.5$% ($3.8$% and $21.6$% on average) for these models.

Decision: reject
Reviewer scores: 6, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1730215237224

thread_type:
official_review

<thread_object>
creator: reviewer_zknz

<content>
summary:
This paper aims to improve the VLMs’ performance in downstream tasks. One prompt optimization method, namely GLOV, is proposed by meta-prompting LLM to choose the type of language structure preferred by the downstream VLM. At each optimization step, one embedding space steering methodology is used to bound the outputs more strictly. Empirical experiments with different VLM architectures on multiple datasets show the GLOV’s effectiveness.

strengths:
[+] Improving the generalization of VLM in downstream tasks with low cost (parameters, fine-tuning data, training speed) is one practical topic that deserves further explorations.

[+] The paper is easy to follow and understand, having clear logic.

[+] Some experiments are conducted to demonstrate the idea’s performance, as well as the values of optimal prompt search.

weaknesses:
[-] Novelty. Meta-prompts have been introduced by [1], while this paper expands the idea to a few-shot training data, which is rather trivial and brings minor technological contributions to the community. For the designs of meta-prompts, how to verify that this solution is optimal?

[-] Impracticality. As we all know, due to the autoregressive paradigm, the LLM inference requires a significant amount of cost compared to encoder-based single-forward models. Thus, employing LLMs in an iterative workflow, to find optimal language prompts seems unrealistic for real-world applications. 

[-] Unknown efficacy. In the main manuscript, only the performance of downstream tasks is reported, without any computational/time complexity. The reviewer suggests to provide the inference time (in seconds) and the required GPU memory (in GB) for all methods in Table 1-2 to clarify its practical value.

[-] Incomplete comparisons. To improve the model performance on downstream tasks, one popular and effective idea is parameter efficient fine-tuning (PET), such as prompt-tuning, LoRA, and adapter, which has shown impressive few-shot capability. In Table 5 of the supplementary materials, CoOp performs even worse than CLIP, which is surprisingly and suspiciously. It is necessary to compare PET with the proposed method, in terms of performance, parameters, training time, and inference speed of under the same settings. 


[1] Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs.

questions:
For few-shot learning, the impact of uniquely labeled data on performance is significant. In this paper, how to select this sample to ensure that the reported results are statistically significant rather than random? What is the variance of performance if five times of experiments are conducted?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651649222

thread_type:
official_review

<thread_object>
creator: reviewer_qqdy

<content>
summary:
This paper introduces GLOV, a method that uses Large Language Models (LLMs) as implicit optimizers to improve Vision-Language Models (VLMs) for vision tasks. GLOV employs meta-prompts to help the LLM generate effective prompts tailored to specific tasks, which are ranked for their effectiveness. It incorporates feedback from previous optimization steps through an offset vector that guides the LLM in generating better prompts. Experiments on 16 datasets demonstrate the effectiveness of proposed methods.

strengths:
- This paper introduces an interesting framework, GLOV, to enhance VLMs for image classification.
- The use of meta-prompts and LLM steering provides fresh insights in this field.
- Experimental results demonstrate the effectiveness of the proposed methods compared to  baselines.

weaknesses:
- Lack of comparison. While GLOV approaches image classification from a novel perspective, previous works [1,2,3] in this area already achieved promising results with lower costs. Could authors provide a comparison of GLOV with these baselines?
- The generalization ability of GLOV is not clear. The authors demonstrated the effectiveness of the proposed methods on VLM image classification and visual question answers under the same topic. However, if the GLOV is not competitive compared with other works focused on VLM image classification[1,2,3]. Could authors prove the generalization ability of GLOV on visual tasks beyond image classification?
- Clarity of Figure 2: The method overview in Figure 2 is difficult to understand. If the authors could clearly show the flow of iterative optimization, the methods would be easier to follow.
- Lack of inference cost comparison: Could the authors show the curve of iteration steps and inference time to illustrate the trade-off between performance and cost in GLOV?


Reference:
[1]AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportation
[2]Sus-x: Training-free name-only transfer of vision-language models
[3]Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

questions:
Please kindly answer the question in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700624185

thread_type:
official_review

<thread_object>
creator: reviewer_cshe

<content>
summary:
This paper proposes a novel framework GLOV, which enables LLMs to act as implicit optimizers for VLMs to enhance downstream vision tasks. Experiments highlight the effectiveness of GLOV for both dual-encoder and encoder-decoder architectures.

strengths:
* The introduction of steering LLM response during the prompt optimization process presents a novel and effective methodology.
* The steering strategy designed by analogy with the gradient update process, while lacking a rigorous theoretical basis, conforms well to engineering intuition.
* The article is highly readable, featuring a well-defined and clear structure.

weaknesses:
* The applicability of GLOV optimization to a given task is constrained by the existence of an objective fitness function for the task.
* For encoder-decoder models such as LLaVA, it seems the VLM response has to be relatively concise in form. When dealing with complex responses (such as responses for image captioning tasks), the reliability of the sentence embeddings computed via Equation 3 remains unverified.

questions:
* In the context of encoder-decoder architectures, is there a potential for the emergence of lengthy and ambiguous symbolic representations during the optimization process? Furthermore, what measures can be implemented to ensure the efficacy of sentence transformers under such circumstances?

* The reviewer expresses concern that the adoption of top-k and bottom-k approaches for in-context examples may result in a significant disparity between positive and negative samples in the later stages of training, potentially hindering the model to learn subtle prompt refinements akin to the challenges posed by consistently employing a large learning rate in gradient-based optimization. Consequently, the reviewer prefers implementing a dynamic selection threshold as a more reasonable choice. Any insights regarding the current strategy would enhance the understanding of the paper.

* Similarly, in the steering of LLM, would it be more judicious to dynamically modify the rank interval between the positive (P+) and negative (P-)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731103968098

thread_type:
official_review

<thread_object>
creator: reviewer_h9hv

<content>
summary:
The paper proposes GLOV, an LLM-assisted framework for automatic optimization of VLM prompts for specific downstream tasks. Specifically, an LLM is meta-prompted to generate downstream VLM prompts based on task description and in-context examples from previously generated prompts as feedbacks. On top of the meta prompt, the paper also applies feature-level guidance, i.e., the difference of sentence embedding from bad prompts to good prompts, as a second measure to push the LLM output towards the more effective direction. The proposed method is evaluated mainly on 16 few-shot classification tasks and shows improvement over baselines, while preliminary results on VQA are also provided.

strengths:
* The motivation is sound and clear.

* The experimental results are transparent, with search outcomes in the appendix and code release promised.

* The prompts generated by search are shown to generalize within the same category (dual encoder) of VLMs.

weaknesses:
* **Feature-level guidance poses white-box LLM constraint**: Despite the feature guidance being novel for VLM prompt optimization, it requires access to LLM intermediate features, which could be hard to obtain given that many strong LLMs are either closed-source or too large to easily run locally. This could be a hedge against the advantages of black-box methods, as the LLM intermediate features could be even harder to get than parameters or gradients of VLMs in many cases.

* **Sensitivity to LLM choices is not clear**: While the proposed method shows clear improvements, it would make the argument stronger if more evidence could be given showing that the reasoning of the LLM indeed plays an important role, especially with the fluctuation (e.g., Fig 1, 3, 6) of the results and the general impression that LLMs at 7B-level are not very good at reasoning or agent-like applications. One way to show this is higher accuracy or less steps to convergence with a stronger LLM.

questions:
* **Clarity of Algorithm 1**: At lines 9, 12, 28, 29, it's unclear what is the meaning of the square brackets, given that $K$ is an integer according to the input. It's also not clear how the top-3 prompts used for ensemble are selected: Are they from the last step, a single best step, or all steps through out the search process?

* **Sensitivity to the hyper-parameters**: The LLM steering part introduces two hyper-parameters, layer $l$ and scaling factor $\alpha$. Are these hyper-parameters searched on one dataset and kept the same for the others, or searched independently on each dataset? How different could the optimal choices be over different datasets?

* **Additional placeholders in prompts**: Some searched prompts (e.g., at Ln 1187-1192) seem to contain additional placeholders (in angle brackets). Are they from additional metadata of the dataset? Is the search process altered in any way to account for these additional information?

* **Average accuracy in main tables** (e.g., Table 1, 2) would make it easier to cross-reference the results with the ablation studies (e.g., Table 4)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732065471985

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely thank the reviewers for the time and effort spent reviewing our paper and for their thoughtful feedback. The positive reception of our paper’s **novel methodology** (CSHE), particularly the use of meta-prompts and the **steering of LLM responses** (qqDY) during prompt optimization, is highly encouraging. Further, we are pleased that the **practical value** (ZKNZ) of improving VLM generalization with minimal resource costs was appreciated, along with the paper's **clarity and structure** (ZKNZ, CSHE). Additionally, the acknowledgment of **transparency of experiments** (h9hv) is also heartening.

Below, we respond to all the reviewers’ comments individually and hope to indulge in a constructive discussion during the author-reviewer discussion period. 

We thank you again.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733196223832

thread_type:
official_comment

<thread_object>
creator: area_chair_jpjw

<content>
title:
Discussion due soon

comment:
Dear all reviewers,

Our reviewer-author discussion will end soon. For each of you, please check all the files and see if anything you'd like to discuss with authors.

Best,
Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a GLOV method to enable LLM as an implicit optimizer for VLMs for downstream task improvement. It received mixed reviews initially. The reviewers raised technical unclear presentation, limited applications, lack of sufficient comparison, and limited novelty. During the rebuttal phase, the authors try to address these issues by providing more explanations and experimental results, which turn reviewers into positive. However, one reviewer [ZKNZ] points out the work is similar to an existing one [1] and the implacability remains.  Although authors try to further explain the novelty and real applications, the justification does not motivate [ZKNZ] to change the opinion. Overall, the AC has checked all the files, and find [ZKNZ] is reasonable for the novelty issue. There is not much significant novel design upon [1] as listed by [ZKNZ]. The iterative manner is prevalent in many pipelines and the guidance seems as a loss supervision, which is not something new. The AC agrees that authors bring GLOV to the VLM field, but the novelty still remains limited according to the current form. The authors shall further improve the current work and welcome to submit for the next venue.