GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
Authors: Muhammad Jehanzeb Mirza
Mengjie Zhao
Zhuoyuan Mao
Sivan Doveh
Wei Lin
Paul Gavrikov
Michael Dorkenwald
Shiqi Yang
Saurav Jha
Hiromi Wakaki
Yuki Mitsufuji
Horst Possegger
Rogerio Feris
Leonid Karlinsky
James R. Glass
Venue: Accepted by TMLR ()

Abstract: In this work, we propose GLOV, which enables Large Language Models (LLMs) to act as implicit optimizers for Vision-Language Models (VLMs) to enhance downstream vision tasks. GLOV prompts an LLM with the downstream task description, querying it for suitable VLM prompts (\eg for zero-shot classification with CLIP). These prompts are ranked according to their fitness for the downstream vision task. In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of prompts preferred by the downstream VLM. Furthermore, we explicitly guide the LLM's generation at each optimization step by adding an offset vector -- calculated from the embedding differences between previous \textit{positive} and \textit{negative} solutions -- to the intermediate layer of the network for the next generation. This offset vector biases the LLM generation toward the type of language the downstream VLM prefers, resulting in enhanced performance on the downstream vision tasks. We comprehensively evaluate our GLOV on two tasks: object recognition and the critical task of enhancing VLM safety. Our GLOV shows performance improvement by up to $15.0\%$ and $57.5\%$ for dual-encoder (\eg~CLIP) and encoder-decoder (\eg~\llava) models for object recognition and reduces the attack success rate (ASR) on state-of-the-art VLMs by up to $60.7\%$.

Published in Transactions on Machine Learning Research (08/2025)
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
M. Jehanzeb Mirzal MIT CSAIL, USA.
Mengjee Zhao Sony, Japan.
Zhuoyuan Mao Sony, Japan.
Sivan Doveh
Wei Lin JKU, Austria.
Paul Gavrikov
Weizmann Institute of Science, Israel.
Tübingen AI Center, Germany.
Michael Dorkenwald
Shiqi Yang Sony, Japan.
Saurav Jha
UVA, Netherlands.
UNSW, Australia.
Hiromi Wakaki
Yuki Mitsufuji Sony, Japan
Horst Possegger TU Graz, Austria.
Sony, Japan.
Rogerio Feris
Leonid Karlinsky MIT-IBM, USA.
James Glass MIT CSAIL, USA.
MIT-IBM, USA.
Reviewed on OpenReview: http:///wwwooooo.ooo////o//oooea
Abstract
In this work, we propose GLOV, which enables Large Language Models (LLMs) to act as implicit optimizers for Vision-Language Models (VLMs) to enhance downstream vision tasks. GLOV prompts an LLM with the downstream task description, querying it for suitable VLM prompts (e.g., for zero-shot classification with CLIP). These prompts are rankod according to their fitness for the downstream vision task. In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of prompts preferred by the downstream VLM. Furthermore, we explicitly guide the LLM's generation at each optimization step by adding an offset vector calculated from the embedding differences between previous positive and negative solutions to the intermediate layer of the network for the next generation. This offset vector biases the LLM generation toward the type of language the downstream VLM prefers, resulting in enhanced performance on the downstream vision tasks. We comprehensively evaluate our GLOV on two tasks: object recognition and the critical task enhancing VLM safety. Our GLOV shows performance improvement by up to 15.0% and 57.5% for dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVA) models for object recognition and reduces the attack success rate (ASR) on state-of state-of-the--rt the-art VLMs by up to 60.7%.
CLIP VIT-R332
MlAVA-Dne-Viionn (Qumn2-78)
Top-1 Hesuury 680
(36)
90
LEAVA.OV
CILOW (4y0 gratawe)
19
Opeinseation
Opeiiteetion Shape
Figure 1: The effect of prompt evolution on the downstream task performance. The shaded regions represent the absolute top-1 classification accuracics for ImageNet (Deng et al., 2009) at cach optimization stop by ensembling the top-3 prompts found w.r.t the accuracy the 1-shot train set. whereas the solid lines represent. the exponential moving average. LLM employed is Llama-3 (Dubey et al., 2024).
Work partially done as an intern Sony, Japan. Correspondence: jnirzalmit.

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
Orthogonal to traditional gradient-based optimization (Nestcrov, 1983; Boyd & Vandenberghe, 2004; Kingma & Ba, 2014; Ruder, 2016), the recent rise of large language models (Brown et al., 2020; OpenAI, 2023; Chiang et al., 2023; Raffel et al., 2020; Touvron et al., 2023a;b; Dubey et al., 2024) and vision-languee
foundation models (OpenAI, 2023; Li et al., 2024; Zhu et al., 2024; Alayrac et al., 2022; Radford al., 2021) has introduced the possibility of framing optimization in the context of natural language prompts. This form of optimization typically does not require any gradient-based learning parameter update but focuses on extracting knowledge from the language models via suitable natural language prompts. A large body of work focuses on finding natural language prompts optimized for varions downstream tasks for both LLMs (Yang ct al., 2024; Wei et al., 2022; Kojima et al., 2022; Yao et al., 2023) and VLMs (Pratt et al., 2023; Roth et al., 2023; Mirza et al., 2024), demonstrating impressive gains in language-based and downstream vision tasks.
In our work, we frame optimization around discovering suitable natural language prompts for VLMs, with the objective of improving performance on downstream vision tasks. Our proposed GLOV employs a prompt search technique relying on embedding space guidance, that drives the prompt optimization for the VLMs. We iteratively query an LLM with downstream task-specific descriptions and ranked in-context examples derived from the previous (optimized) prompts. In-context examples guide the LLM toward the desired output, and their ranking (measured on a small held-out train set) provides the LLM with a prior for the language patterns preferred by the downstream VLM. To further stoer the LLM generation towards a notion of goodness in cach optimization step, we explicitly bias the language generation with a direction. This direction determined by adding hidden state offset vector (on the last token during the autoregressive generation) derived from the positive and negotive prompts (based on their effectiveness on labeled training data) to the LLM's activation space during generation. The intuition is that by directing the LLM generation toward the positive prompts, the model can discover semantically similar and potentially more effective solutions. One complete optimization run, depicting the effectiveness of the discovered solutions and the effect of applying the embedding space guidance is plotted in Figure 1. The best- pest-performing prompts (on the held-out train set) achieve an absolute improvement of 2.6% and 15.2% on ImageNNtt (Deng et al., 2009) test set over CLIP (Radford et al., 2021) and LLL (Li et al., 2024), respectively.
We extensively evaluate our GLOV on the fundamental computer vision task of image classification and also on interpreting the safety and reliability of responses from open-source VLMs. For image classification, we demonstrate the generalization of our GLOV - the dual-cncoder and the recent visual encoder-decoder models (Radford et al., 2021; Li et al., 2024). On the other hand, for VLM safely, we evaluate our GLOV on 2 recent benchmarks, which highlight severe vulnerabilities of open- source VLMs. Our GLOV can consistently discover highly effective solutions for the downstream task of interest resulting in significant improvements across the board. For example, the most effective prompts discovered for the dual-encoder models (e.g., CLIP) can improve the image classification accuracy by up to 15.0% (3.81% on avcrage) and for the encoder-decoder architectures (e-g., LLaVA), the resulting prompts show an even larger improvement of up to 57.5% (21.6% on average). Furthermore, the effective prompts found for enhancing the VLM safety can lower the attack success rate (ASR) to merely 1.1% (from 61.8%) with minimal degradation of the model's generalization.
total of 16 diverse datasets, with the two commonly employed families of VLMs
2 Related Work
Our work is related to large language and vision language models, approaches evaluating the safety of VLM responses, and prompt optimization methods (through L.Ms) for VLMSs
2.1 LLMs and VLMs
Here, we first provide brief overview of LLMs and then move towards VLMs.
LLMs have revolutionized the natural language processing landscape. These models can typically be divided into two major groups; namely the longsshortteerm-mmenorry (LSTM) (Hochreiter & Schnidluber, 1997) and transformer- based architectures (Vaswani et al., 2017). The LSTM family of models uses gates to control the flow of information, allowing them to capture long-term dependencies sequential data. Some notable works following the LSTM family of models include (Sutskever et al., 2014; Graves & Schmidhuber, 2005; Bahdanan et al., 2015; Beck et al., 2024). On the other hand, some notable works proposing the encoder-based transformer architectures
2

---

Published in Transactions on Machine Learning Research (08/2025)
inchude BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), and DistilBERT (Sanh et al., 2020). The decoder-based LLMs, the other hand, are designed for generative tasks such as text generation, translation, and summarization, with recent models including GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020), GPT-4 (OpenAl, 2023), and the Llama family of models (Dubey et al., 2024). In our work, since we need to accoss the weights of the LLMs, we resort to the open-source Llama-3 model. However, any open-source LLM can potentially be employed.
Recent advancements in prompt learning have introduced various techniques to efficiently adapt large language models (LLMs) to specific tasks with minimal parameter updates. Early methods, such as contimnous prompt learning (Zhong et al., 2021; Li et al,, 2021; Lester et al., 2021), optimize continuous vectors in the word embedding space. However, these approaches lack interpretability, as the learned vectors do not correspond to discrete tokens. To address this, newer methods like Decomposed Prompt Tuning (DePT) (Shi & Lipani, 2023) and Low-Rank Prompt Adaptation (LoPA) (Jain et al., 2024) have emerged. DePT decomposes soft prompts into shorter prompts and low-rank matrices, optimizing them with different. learning rates to enhance performance while reducing memory and time costs. LoPA employs a low-rank decompasition of soft prompts, balancing shared and instanco-spocific information to achieve parameter efficiency comparable to full fine-tuning. In the realm of PEFT, methods like Low-Rank Adaptation (LoRA) (Hu et al., 2021a) and its derivatives, such as QLoRA (Dettmers et al., 2023), have gained prominence. LoRA introduces trainable low-rank matricos into the attention layers of transformers, allowing significant reductions in trainable parameters without compromising performance. QLoRA combines LoRA with quantization techniques to further minimize memory usage, enabling fine-tuming on consumer-grade hardware. Additionally, beapter-based methods (Hu et al., 2023) insert small trainable modules between existing layers of a model, facilitating task-specific adaptations while keeping the majority of the model parameters frozen.
VLMs can be placed in two categories. One group relics on dual-encoders (vision and text encoder), usually trained in a contrastive manner, and these models are typically strong at tasks like image recognition. The most common among these methods are CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), OpenCLIP (Schuhmann et al., 2022), SigLIP (Zhai et al., 2023), and MetaCLIP (Xu et al., 2023). Many methods (Mirza et al., 2024; 2023b; Doveh et al., 2023b;a; Lin et al., 2023; Mirza et al., 2023a) build upon these models to further improve them for spocific rownstream lasks. The other group of methods aligus the visual modality with a frozen LLM and can be used for open-ended visual reasoning tasks like image captioning, visual grestion-answeriing etc. Some representative approaches from this group include BLIP-2 (Li et al., 2023), Instruct-BLIP (Dai et al., 2023), MiniGPT (Zhu et al., 2024; Chen et al., 2024), and the LLaVA family of models (Liu et al., 2023; Li et al., 2024). Similarly, some approaches (Doveh et al., 2024; Gavrikow et al., 2024; Lin et al., 2024; Huang et al., 2024) build upon these models and provide further improvements. In this work, we address the task of object recognition using both encoder- based and decoder- basod vision-language models (VLMs). We formulate the problem of identifying optimal prompt templates for CLIP (Radford et al., 2021), as well as suitable prompts for open-ended generation in models like LLaVA (Li et al., 2024), as an optimization task. Unlike prior mothods such as MPVR (Mizza et al., 2024), WAFFLE (Roth et al., 2023), and CUPL (Pratt et al., 2023), which learn per-category prompt templates and are therefore much more computationally expensive, especially for larger datasets like ImageNet (containing 1000 classes), our proposed method, GLOV, discovers single, generic set of prompt templates that is effective across an entire dataset.
Initially developed in the context of natural language processing (Zhong et al., 2021; Lester et al., 2021), prompt-tuning techniques have recently been extended to visual domains as well (Jia et al., 2022; Zhou et al., 2022d; Lu et al., 2022). For instance, CoOp (Zhou et al., 2022d) optimizes prompt embeddings by minimizing classification loss, while ProDA (Lu et al., 2022) introduoes a stratogy to learn multiple diverse prompts to better capture variations in visual features. UPL (Huang et. al., 2022) takes an unsupervised approach to prompl. learning, eliminating the need for labeled data. TPT (Shu et al., 2022) introduces test-time optimization method for adapting prompts dynamically during inference. Other works like CLIP-Adapter (Gao et al., 2024) and Tip-Adapter (Zhang et al., 2022) explore lightweight adapter- based alternatives for PEFT in multimodal settings. CoCoOp (Zhou et al., 2022c) incorporates a meta-learning approach to generate image-conditionee prompts, enabling improved robustness to distribution shifts, whereas MAPLE (Khattak et al., 2023) learns text and vision prompts in synergy. Other methods (Zhang et al., 2024a; Kim et al., 2024) have also introducod certain variations to the prompt-tuning framework and further enhanced the downstream image classification performance. Tn contrast to these works, we do not learn contimuous prompt voctors through gradient-based finc-tuning, but instcad propose to optimize nat natural-language prompts, making our framework more interpretable.
3

---

Published in Transactions on Machine Learning Research (08/2025)
For decoder-based VLMs, recent work (Zhang et al., 2024b) has pointed out that decoder-based VLMs often struggle with fine-grained object recognition. However, we demonstrate, for the first time, that GLOV can uncover optimal prompts that significantly enhance recognition performance in these models, all without any gradient-based learning or fino-tuning, Furthermore, we note that most existing prompt tuning approaches (Zhou et al., 2022b;a; Mirza et al., 2024; Pratt et al., 2023; Roth el al., 2023; Khattak el al., 2023) are limited in soope: they are only applicable to encoder-basod models like CLIP and cannot be directly applied to docoder-based VLMs.
2.2 Safety of VLMs
Enhancing the safety of open-source VLMs remains an open challenge. Recent works Zong el al. (2024); Liu et al. (2024c); Tu et al. (2024); Gou et al. (2024) have evaluated multiple open-source VLMs on safety tasks. These tasks involve presenting the model with an unsafe image and an instruction, where the model is considered safe only if it refuses the unsafe instruction. However, current state-of-the-art VLMs exhibit significant vulnerabílities in this critical area. In our work, we enhance the safety of VLMS by optimizing for safety instructions and finding general safety prompt that transcends models and safety benchmarks with minimal loss in the generalization abilities of the open-source VLMs. To the best of our knowledge, our GLOV is the first method to oplimiie safety instructions for VLMs, without any human in the loop. We again point out that other prompt-tune methods (Mirza ct al., 2024; Roth et al., 2023; Pratt et al., 2023; Zhou et al., 2022a;b) cannot be applied to such open-ended tasks (and decoder-based VLMs), whereas our prompt optimization method is more generic.
2.3 Large-la Models as Prompt Optimizers
Some approaches propose employing LLMs (in an agentic workflow) to search for the optimal prompt for the downstream task. OPRO (Yang et al., 2024) coins the term "LLMs as Optimizers" and proposes iteratively discovering solutions (prompts) for natural language tasks by employing an LLM in a feedback loop. Similarly, Liu et al. (2024b) proposes to find suitable prompts for dual-encoder VLMs (e.g., CLIP) by iteratively prompting an LLM, Our GLOV also proposes to discover suitable prompts for VLMs but differs from Liu al. (2024b) in the sense that we are employing a prompt that caplures long-range dependencies by lapping into the history-buffer of the in-context. examples and exploiting task-specific knowledge that helps to obtain prompts better suited for the downstream task. Furthermore, inspired by (Turner et al., 2023; Liu et al., 2024a; Todd et al., 2023) we propose a novel method for steering the LLM generation (through embedding space guidance) towards the text responses that are more suitable for downst ream VT.Ms. Our GLOV- powered by a carefully tailored LLM prompt and the guidance scheme discovers highly effective solutions that help to enhance visual tasks for both the -ual-encoder and encoder-decoder models.
3 GLOV
The goal of our GLOV to improve the VLM's downstream (vision) task performance by optimizing natural language prompis through employing an LLM in an iterative workflow. To achieve this, we build upon the successful line of work on in-context learning Wei et al. (2022); Min et al. (2022), which can improve VLM performance through task-specific prompting Mirza et al. (2024). However, our GLOV goes beyond these by automating the LLM-VLM interaction by adopting an agentic workflow and also explicitly guiding the language generation, conditioned on a prior of the difference of the sentence embeddings from the posilive and negative prompts discovered during the previous optimization iterations. Although the application space of GLOV is general in termns of tasks and the reanstream VLMs, for simplicity, here we focus our description around CLIP (Radford et al., 2021) while mentioning the differences for LLaVA (Li et al., 2024) and different tasks, where appropriate. Figure 2 provides an overview of our methodology. Our codebase is also attached for review supplementary.
For the ease of ion, we divide the description of our GLOV into different parts. In Section 3.1 we describe the fitness function and how it can provide an interface for the LLM-VLM interaction. In Section 3.2, we provide details about the tailored LLM prompt employed in our work. Finally, we conclude in Section 3.3 by providing details about the proposed guidance methodology.
4

---

Published in Transactions on Machine Learning Research (08/2025)
Fitness Function
LLM Prompt
Quiciiaio Locp
System Instruction
/157
lidec teez. mockeetiinng
Task Description
d Vircs Laceker
Paxnps radked aus flbec
In-oontext Examples
Cociae Six siaiiiityy
Pocttive Prompt
picaso ofa massgre to okic
Arcweey 52.1%
A mijoati vialogerp with gaolle pun. ettee area lounging the sunbeams that araam through the windows, in tringing and contier to its owners tis with tte oif for and loving companientip....
LLM
a (H_ H.
-A chati hed and loyal seatgaary with varn loving damaamer ofle found Eie heats of its ownes contaatt companioon briging immenad
Nepative Pronge
Averrayy 47.3%
Figure 2: Overview of GLOV for dual-encoder models. GLOV consists of a specifically tailored LLM prompt, which constitutes system instruction, task description, and in-context examples (VLM prompts) which are evaluated (and ranked) on few-shot training data in each iteration. This prompt instructs the LLM to generate several candidate solutions in each optimization iteration, conditioned on the in-context examples which are fed in conjune- tion with the accuracy values, highlighting their effectiveness. Furthermore, to steer the LLM generation towards the language preferred by the VLM, we add the scaled difference of the sentence embeddings (autoregressively) from the posilive and negulive text prompts to the intermediate layer of the LLM. This process is repeated until the stopping condition is met (e.g., maximum iterations). Note, that H4 and H- refer to the sentence embeddings from the positive and megative text prompts, and a is the scaling factor. The example prompts shown in the image are the GLOV prompts for the OxfordPets (Parkhi et al., 2012) dataset, consisting of categories of dogs and cats.
3.1 LLM-VLM Interaction through Fitness Function
The datasct-specific prompt templates P provided by CLIP (Radford et al., 2021) have boen constructed manually, requiring human effort. In this work, we frame the prompt search as an optimization problem and propose to replace the human with an LLM, employed in an iterative feedback loop. Furthermore, we explicitly guide the generation process of the LLM in each optimization step by proposing a novel guidance methodology that can assist the LLM in understanding the style of language preferred by the downstream VLM, even though the two models only interact through fitness function. At each optimization step i, the LLM provides multiple (e.g., 10) solutions to improve the downstream task performance. However, not all solutions provided by the LLM are preferred (i.e,, enhance performance) for the rewnstreea vision lask. To obtain a measure of the fitness (effectiveness) of the provided solutions to the downstream vision task, evaluate all the candidate solutions held-out few-shot (1-shot) labeled training dataset D. For CLIP (Radford et al., 2021), the zero-shot likelihood of class è for each discovered prompt P during an optimization step can be found by (os(Ve.dee))/
le(x) Eog where =x=((((()),
(1)
where o and ib represent the vision and text encoders of CLIP, x DE r denotes the temperature constant, cos relers to the cosine similarity, and p(c) replaces the 'class' placeholder in the discovered prompt p. CLIP's vision encoder é produces the image embedding. The text embedding of a class e (belonging to a set of candidate classes C) is obtained by incorporating the class name c in the found prompt, called a VLM prompt, and embodding this text through the VLM's text encoder 1b. The fitness of a prompt pEp can be found by comparing the predicted label with the ground truth, sunmarized as
Fitness(p) 1
Σ
1 argmaax (<(x==y)
(2)
[D]
(I.D)ED
5

---

Published in Transactions on Machine Learning Research (08/2025)
where 1 is an indicator function that is 1 if the predicted label matches the ground truth y and 0 otherwise. For the encoder-decoder models (e.g., VA), which produce open-ended outputs, we obtain the class likelihoods by obtaining a symbolic representation of the image in textual form and comparing the text embeddings from this symbolic reprosentation with the text embeddings obtained for the individual class names, using a dodicated sentence embedding model (Reimers & Gurevych, 2019). We expand on these details in the Appendix Section A. Furthermore, for the open-ended task of enhancing the safety of VLMs, the fitness of the prompt defined in terms of reducing the attack success rate (ASR)*. To this end, we follow the evaluation protocols proposed by the original papers (Zong et al., 2024; Liu et al., 2024c), which focus on measuring the refusal rates and evaluations by using GPT-4 (OpenAT, 2023) as a judge for measuring the ASR.
It is important to note that the fitness function forms a bridge between two disjoint models the LLM and the VLM, and is responsible for their interaction. The fitness (e.g., classification accuracy) provides feedback to the LLM regarding the type of natural language sentences that are preferred by the downstream VLM. The fitness function is responsible for ranking all the prompts providod as in-context examples to the LLM prompt in each optimization iteration (c.f., Section 3.2) and also forms the basis for the application of the enbeddinggspacee guidance methodology (c.f., Section 3.3) proposed in this work to bias the LLM responses towards a notion of goodness.
3.2 LLM Prompt
The LLM prompt (cff, Appendix Figure 5) is responsible for driving the iterative prompt optimization. It consists of 3 distinct parts, which are described as follows:
System prompt is a generic set of instructions that describe the task of the LLM. It helps a model to find the optimal prompts, improving the downstream task accuracy (or reducing ASR). The system prompt remains static for the entire optimization.
Task description is dynamically changing at each optimization step. It consists of a description of what is inchuded in the main body of the prompt, e.g., what is expected from the LLM (f.c., a prompt), downstream task name, task description, quantity (and actual) best and worst prompt templates as in-context examples, with their associated fitness, obtained through equation 2.
In-context examples serve to bootstrap the LLM to the type of output that is expected. In each optimization step, we lask the LLM to provide us with 10 candidate solutions (prompts). Each prompt is evaluated w.r.t. a fitness function to obtain a (classification) score. We keep a global history of the prompts (and associated fitness) generated during all the previous optimization steps and at each optimization step i, the newly generated prompts and all the previous prompts are ranked according to their respective fitness score. For the next optimization step 1, tops, and bottom; prompts (we choose h 5) are selected as in-context demonstrations and plugged into the prompt together with their respective accuracies. The intuition behind keeping a global history of prompts is to provide the LLMs with long-term knowledge about the types of prompts that have been effective for the VLM so that it can model its responses according to them.
3.3 Steering the LLM Generation Process
At a higher level, given two prompts - ppositive and negative (identified through equation 2), our proposed steering can be considered as analogous to computing hidden state gradient towards the positive prompt, effectively biasing the language generation away from the negulive identified prompt. in each optimization step. The inluition is to condition the LLM text outputs according to the language preferred by the downstream VLM. To this end, we show that the LLM outputs can be stecred through simple arithmetic in the hidden states of the present-day LLMs.
For a given LLM f with pre-trained parameters 0, and given tokenized prompts ph and pir the act ivation responses at layer are denoted as (7(7))). This is an activation map of B x S SxE, E, which denotes the mumber of prompts B, tokenized soquence length S, and the hiddon dimension size E. Typically, a does not depend on all model parameters 0, but we abuse the notation in the interest of simplicity. The sentence embeddings H+ and H_ can be obtained by averaging the activations across the sequence length S
For safety enhancement, GLOV optimizes for a prompt that reduces the ASR on downstream VLM.
6

---

Published in Transactions on Machine Learning Research (08/2025)
1 S. 5+
1 S S_ (0(p=:0)...
H4 Ea () 11
Σ
(3)
where S4 and S_ are the sequence lengths of prompts pi and p_, respectively. The goal is to obtain semantically meaningful sentence embeddings from the identified positive and negative prompts.
For each new token produced in the subsequent optimization iteration, the difference between H+ and H_ is added autoregressively to the embeddings of each generated token". Let Pr. denote the new token appended to the LLM prompt, then the updated sentence embedding Hn is given by
=, +a +a-(-a--HH4 -H-)
(4)
where is the scaling factor and is automatically optimized by GLOV on the held-out training set, by an a sweep. This process is repeated until the maximum mumber of tokens is achieved for each prompt. In total we prompt the LLM (at each iteration) to provide us with N prompt templates. In an optimization run pt is always the prompt with the best accuracy w.r.t the fitness and p- is set to be the prompt with the socond-best accuracy. Since, we compute a form of the gradient-like differential between averages of token hidden states, intuitively trying to identify a characteristic of task-specific improvement. Thus, the intuition behind computing the differential between the best and the socond best (in terms of fitness) is to obtain it between points closest to the maximal value of the objective which is common mathematical intuition. Furthermore, P+ and p- are only updated when a new prompt with higher accuracy is found. This ensures that the guidance signal does not alter in each iteration, resulting in more stable optimization. The detailed algorithm is laid out in Algorithm 1.
We ablate our proposed method of obtaining sentence embeddings (equation 3) in Section 4.3 and find it to provide strong results (while linear probing the embedddings from the middle layers of the LLM) on common natural language classification tasks, hinting that our sentence embeddingg capture semantically meaningful information from the prompts. This analysis also motivates our choice of relying on the middle layers for extracting guidance in GLOV.
As our prompt search method is posed as an optimization problem, thus, it requires access to a small amount of labeled data to obtain the effectiveness of each prompt at each optimization step. For main experiments in Tables we ouly use 1-shot labeled examples. To further reduce the reliance on labeled data, we lest our method in a sub-shot setting where we use labeled examples from only a small subset of classes from the dataset. These results are listed as an ablation in Table 4, where we find that by optimizing the prompts only on 50 labeled samples (from 50 categories, instead of 1000 in ImageNet),, our method can still achieve an absolute improvement of 13.4% as compared to the baseline, while only showing a degradation of 1.8% when compared with optimizing the prompts on 1000 samples from 1000 classes. These results are a step towards making the optimization method unsupervised. In the future, our GLOV can potentially be made completely unsupervised (and zcro-shot) by employing unsupervised metrics to measure the elfectiveness of prompts. These met rics could include entropy measurement (Shannon, 1954), and choosing the prompts that decrease the overall entropy of the task. Furthermore, we can also employ LLMs to provide a signal of effectiveness for the prompts, which could also completely lift the burden of data-labeling.
4 Experimental Evaluations
In this section, we first list the datasets employed for evaluating our GLOV, then provide an overview of the different baselines and state-of-the-art methods we compare to, later discuss the implementation details and finally conclude with discussion of the results.
4.1 Evaluation Settings
Image Recognition Datasets: We extensively evaluate our GLOV on 16 object recognition datasets belonging to widely different domains. These domains can be narrowed down to datasets containing commonly occurring natural categories: ImageNet (Deng et al., 2009), ImageNet (Recht al., 2019), Caltech101 (Fei-Fei al., 2004), fino-grained classification datascts containing different task-specific images: Oxford F Flowers (Nilsback & Zisserman,
We also experiment with several alternatives for adding the offset vector the appendix Section 4.3, however, find that adding the offset only the last. token performs best.
7

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Results on dual-encoder VLM. Top-1 accuracy (%) for 16 datasets obtained by employing the ViT-B/32 backbone from OpenAI CLIP (Radford et al., 2021). S-TEMP refer to the results obtained by using the default template (a photo of a <class name>), while DS- TEMP refer to the results obtained by using the ensemble of dataset-specifie prompts. GLOV (w/o guidance) represents the results without the guidance appliod to the LLM generation, whereas GLOV represents results obtained by adding the guidance offset vector. The bold numbers represent the best and the underline numbers represent the second best accuracy.
ImageNNtt
ImageeNotv2
Caltech101
ImageNetR
ImageNNat8
ImageNNotA
OxfordPlowwss
OxfordPets
CLIP (S-TEMP)
61.9
51.8
91.4
65.4
50.3
25.2
61.0
81,3
CLIP (DS- TEMP)
631
421
86.6
83.2
LEM-OPT
628
35.6
12.
67.5
41.9
28.1
57.0
78.1
GLOV (w/j guidanee
62.7
55.8
92.1
67.8
41.9
312
6166
5id
GLOV
64.5
56.6
88.7
68.5
43.0
32.5
67.7
85.5
StanfordCars
Desertbabl/Toxtures
Food101
FGVCAircaaft
SUN397
UCF101
RESISC45
EuroSAT
CLIP (S TEMP)
6022
dn.2
77.6
18.1
02.1
00.4
51.1
35.8
CLIP (D8 TEMP)
38.9
29.2
19.4
61.7
623
57.2
45.8
LPM-OPT
00.2
41.7
79.2
17.7
80.9
80.9
54.4
45.0
GLOV (w/o goldino)))
30.6
41.4
78.5
10.7
62.2
6300
61.4
46.9
GLOV
60.4
79.5
20.1
02.1
62.0
50.8
Table 2: Results on decoder VLM. Top-1 accuracy (%) for 16 datascts obtainod by employing the LLaVA (One Vision) (Li et al,, 2024). LLa VA-OV refer to the results obtained by using a generic prompt.
ImageNNet
InuggeNNetv2
Callech101
Intg
ImugoNedS
ImugeNNetA
OxfordPlowwrs
OxfordPets
LLAVA-OV
36.5
31.4
77.7
62.1
3233
19.4
16.2
GLOV (w/a guidaace)
46.8
30.9
$7.1
75.7
19.6
44.8
28.6
58.7
GLOV
51.7
46.1
92.6
77.6
49.9
436
39.6
54.3
StanfordCars
DescribabbeTTestures
Food101
PGVCAlreaaf:
SUN397
UCF101
RESISC45
EuroSAT
LLaVA-OV
33.2
2L.5
4.1
52.9
43.3
GLOV guidene))
ZLU
46.9
66.9
44.0
41.2
60.6
47.2
W6.3
GLOV
79.2
51.7
67.0
11.0
45.0
39.7
51.1
36.3
2008), Standford Cars (Krause et al., 2013), Oxford Pets (Parkhi et al., 2012), Describable Textures Dataset (DTD) (Cimpoi et al., 2014), Food-101 (Bossard et al., 2014), FGVC-Aircraft (Maji et al., 2013). Dataset used for scene classification: SUN397 (Xiao et al., 2010), action recognition dataset: UCF101 (Soomro et al., 2012). (Hendrycks et al., 2021a), ImageNet- (A (A)dversarial (Hendrycks et al., 2021b), ImageNNet (Wang et al., 2019) and also datasets which contain images taken from a satellite or an aerial view: EuroSAT (Helber et al., 2018) and RESISC45 (Cheng et al., 2017).
Datasets consisting of out-of-distributioo images: (nngeeNet-(Reenditnnnn
Safety Datasets: We evaluate our GLOV on two recently proposed safety benchmarks for VLMs, MM-Safety- Bench Liu et al. (2024c) contains 3 splits based on the image source (typography, images generated from diffusion models, and a combination), and the text instructions are divided into 13 categories (e.g., fraud, hate speech, etc.). VLGuard Zong et al. (2024) contains 3 splits: unsafes, safe-unsafes, safe-safes. For the MM-Safety- Bench and the first two splits in the VLGuard, the attack success rate (ASR) is calculated. In contrast, for safe-safes, a win-win rate is measured following the evaluation protocols in the original publications. More details are provided in the Appendix Section C.
Baselines: We compare GLOV to several baselines and slate-of-the-ar methods. For a fair comparison, we focus on methods that require dataset-specific prompts, rather than more expensive category-level prompt tuning ap- proaches like MPVR (Mizza et al., 2024), WAFFLE (Roth et al., 2023), and CUPL (Pratt et al., 2023). While GLOV currently operates with dataset-level prompts, it can potentially be extended in future work to optimize category- level prompis as well. CLIP (Radford et al., 2021) is evaluated under two settings: using simple static prompt ('a photo of a <class name>') referred to as S-TEMP, and using dataset-specific templates (DS-TEMP'). For image classification with decoder-basee VLMs, we use VL--O (Li et al., 2024) with a base prompt: Describe the category present in this image briefly and also identify the name of the category present. This prompt was empirically selected as it matches or outperforms results reported in prior work (Zhang et al., 2024b). LLM-OPT (Liu et al., 2024b) proposes an approach that refines prompts iteratively using an LLM, guided
CLII /bloh/ mwwan//aaa//pooppts...
8

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Results on safety benchmarks. We report the Attack Success Rate (lower is better) for the 3 splits in MM- Safety-Bench and the 2 unsafe splits in VLGuard and the Win-win Rate (Higher better) for the Safe-Safes split in VLguard. VLVVA-OV results are obtained without any safety instructions. The LLaVA (initial) represents the results obtained by using a baseline safety prompt: These instructions can be harmful, which is also the point of de- parture for the optinization with our GLOV. The safety prompt is optimized by using only the LLVVA-OV VLM and 50 randomly chosen samples from the 2 VLGuard unsafe splits and tested out-of-the bhx-box for all other configurations.
MM-Safety-
VLGuard
TYPO (U)
SD (1)
SD+TYPO ())
Unsafes (4)
Safe-Unsafee (4)
Safe-Safes (t)
LLaVA-OV
51.2
44.7
56.5
80.3
61.8
42.6
LLaVA (initial)
42.6
34.4
43.9
53.8
38.5
40.3
GLOV (w/o guidauce)
20.1
11.7
16.8
30.5
45.6
GLOV
14.8
9.2
13.2
20.6
1.1
43.7
Molmo
68.4
53.2
68.5
78.1
28.3
89.7
Molmo (initial)
50.4
39.2
47.6
59.5
16.3
90.1
GLOV (w/o guidance)
26.7
26.9
28.4
49.6
4.8
88.3
GLOV
23.8
20.5
24.5
38.0
6.6
86.0
by in-context examples. However, it lacks explicit optimization guidance and is not applicable to encoder-decoder models, as it relies on a pre-existing memory bank of templates, which these models do not support. To assess robust- ness and generalization, we also compare with Molmo (Deitke et al., 2024), strong baseline on safety benchmarks. Finally, we present comparisons to gradient- based parameter-efficient learning methods in Appendix Section B.4.
Implementation Details:
To report the results for image classification we use the test splits provided by Zhou et al. (2022b). All the baselines are also implemented in the same framework. To obtain the results on the test. set for each dataset for our GLOV, we ensemble the top-3 prompts. These prompts are chosen with regard the best-performing prompts the 1-shot train set at a certain iteration during the optimization. For our GLOV we use Ilama-3 (Touvron et al., 2023a) from Hugging Face. We set the maximum number of optimization iterations to 100 (with 10 candidate solutions at each iteration) for the experiments with CLIP and 25 (with 5 candidate solutions) for VLaVA-OV and Molmo. For evaluating the safety of VLMs, we follow the above settings and use the official codebase from respective publications (introducing the benchmarks) for evaluations. In general, the experiments with CLIP can run on a single NVIDIA 3090 (24GBs) GPU, and the experiments with LLaVA fit on an A40 (48GBs) GPU similar. We use the 7B variants of Molmo and LLa VA-OO, accessed through Hugging Face.
4.2 Results
Here we first discuss the object recognition results and then the evaluations performed on the VLM safety benchmarks.
Object Recognition: We evaluate our GLOV extensively on 16 diverse datasets. In Table we list the results by employing the CLIP VIT-B/32 from OpenAL. We observe that our GLOV achieves better accuracy on all the datasets evaluated. For example, as compared to CLIP, when using the simple prompt template, our vanilla GLOV provides an average gain of 2.5%, with up to 11.0% gains on FAToSAT. Similarly, the gains increase even further with our proposed guidance scheme. We observe gains of up to 15% (3.8% on average). On the other aut-of-distribution ImageNet variants, our GLOV is also able to show consistent improvements. Our GLOV also fares better when compared with the CLIP classifier constructed by ensembling the hand-crafted templates?. For example, on the largo-scale ImageNet datasct, the CLIP classifier built by ensembling the top-3 performing prompts on the train set can provide a gain of 1.2%, while on average the performance improvements is 1.5%, with up to 5.0% gains on the EuroSAT dataset. It is also important to point out that the prompts provided by CLIP (Radford et al., 2021) are chosen w.r.t the accuracy on the test set (Liu et al., 2024b), whereas, our GLOV scarches for prompts by only having access to 1-shot training data highlighting the generalization ability of our GLOV.
In Table 1, we also compare our GLOV with LLM-OPT (Liu et al., 2024b). Even our vanilla prompting method (without guidance) is on average 1.4% better than LLM-OPT, while the proposed guidance scheme provides 2. 2.7% improvement on average, with 1.7% improvements on the large-scale ImageNet. This highlights that our prompting methodolggy coupled with the guidance scheme is better suited to the task of prompt search. Our
9

---

Published in Transactions on Machine Learning Research (08/2025)
Accuracy (96)
66
50
64
10
15
20
25
30
10
15
20
25
30
Layera
Lagers
Figure 3: Sweep for choosing the LLM layer for guidance. Linear probing accuracy for different. layers of Llama-3, while evaluating our choice of calculating the sentence embeddings for the sentiment classification task in SST-5 dataset (left). Top-1 classification accuracy for ImageNet on the held-out train set while applying the guidance on different layers of Llama-3 (right).
prompt consists of task-specific and long-term knowledge of the LLM's responses about what it has generated in all the previous iterations, whereas, LLM-OPT's prompt does not contain long-term dependencies. Furthermore, the proposed guidance scheme induces a notion of goodness to the language generation, that is reflected in the downstream results. On the other hand, the prompt used by LLM-OPT (Liu et al., 2024b), naively instructs the LLM to provide better prompts, with only providing good and bad prompts, without any explicit guidance, making it challenging for the LLM to bias its outputs towards preferred langugag..
In Table 2 we list the detailed results by employing the LLa VA-VV-V- model. Zhang al. (2024b) find that generative models have extremely low fine-grained visual recognition performance and they improve it via finetuning. In our work, we find that for these models, the fine-grained visual recognition performance can be greatly enhanced by finding the optimal prompt (without requiring gradient- discovered by our GLOV can significantly close the gap with the dual-encoder models. For example, we observe up to 57.5% improvement (21.5% on average over 16 datasets) as compared to vanilla LLa VA-OV. We also observe that our proposed guidance scheme has a significant impact on the results. The guidance further provides 2.5% avcrage improvement over the vanilla GLOV, and notably on the large-scale ImageNet dataset the improvement is 4.9%.
learning). We observe that the solutions
Safety Enhancement: In Table 3 we provide results two recent benchmarks: MM-Safety- Bench Liu et al. (2024c) and VLGuard Zong et al. (2024). These benchmarks evaluate the harmful nature of responses from the VLMs given an image and corresponding instruction. We find that our GLOV can greatly enhance the safety of the present VLMs by finding a suitable safety instruction prepended to the original text instruction. For example, we observe that our GLOV can reduce the harmful responses by up to 60.7% (60.2% on average) while compared to the base VLM (L-&VA-OV)- prompted without any safety instruction on the VL.Guard dataset. We also observe no loss of win-win rate on the Safe-Safes split, which evaluates the models' ability answer safe questions. We also evaluate the recent Molmo VLM and find similar trends. The detailed results for the MM-Safety-Bench are provided in the Appendix Section D. Note that we only optimize for the salety instructions on 50 samples randomly chosen from the VLGuard unsafe splits (with LLa VA-OV) and evaluate on all other configurations evaluated in Table 3. These results highlight that the optimized safety instructions are general and they transcend models and benchmarks.
Ideally, the safety instructions should not hurt the model's performance on general tasks. To evaluate this, we test the VLMs by prepending the safety instructions the questions SEED (Fang et al., 2021) and MME (Yin et al., 2023) benchmarks (c.f., ablations Table 6). Interestingly, we find that in some cases the safety instruction enhances the performance of the model (by up to 23.3 points) these general-purpost benchmarks, possibly by forcing the model to look closely at the instructions. To avoid clutter, we delegate the actual (best) prompts found, the evolution of prompts, and the optimization evolution for datasets to the Appendix Section E.
4.3 Ablations
We provide extensive ablations to study the different aspects of our GLOV. First, we provide several design choices that eventually led to the GLOV framework. Next, we motivate our choice to use the middle layer the LLM for guidance. Then, we provide results by using less than one shot data as our held-out train set and lator, we discuss the results obtained by using a larger LLM. Finally, we conclude with an ablation showing the retention of general model capabilities by adding the safety prompt. More ablations for additional insights are added to the Appendix Section B.
10

---

Published in Transactions on Machine Learning Research (08/2025)
66
66
66
65
65
65
64
64
64
ActADD GLOV
All-Token-- Offect
Sass-Token--B
GLOV
GLOV
63
63
63
50
100
0
50
100
0
50
100
(a) Comparison with AcLADD.
(b) Adding offset to all tokens.
(c) Guidance with only last Loken.
66
66
66
65
65
65
64
641
64
GLOV-25- To-Toe
Cross-Atto
Takene))
Crose-Aten
Takun)
GLOV-76-TToomo
GLOV
GLOV
CLOV-50-Tolae Toloeen
63
63
63
50
100
50
100
50
100
(d) Cross attending all tokens.
(e) Cross attending only last token.
(f) Number of tokens for GLOV.
Figure 4: Ablating design choices. (a) We compare the optimization trajectories obtained with our proposed guidance method with that of ActADD (Turner et al., 2023). (b) The effect of adding the offset vector to each token, instead of only the last token as in GLOV. (c) Using the embeddings of only the last token to obtain the offset vector. (d) Cross-attending the positive and negative prompt embodding vector with the meta-promph tokens at cach optimization step and calculating the offset for guidance. (e) Cross-attending only the last tokens from the posilive and negative prompt embeddings. (f) Finding the optimal number of tokens to be generated at each optimization step. The x-axis represents the optimization steps and the y-axis denotes accuracy (%), datasct is ImageNet.
Table 4: Sub-shot setting. Top-1 Accuracy (%) by using LLaVA-OV for ImageNet (left) and ImageNct-R (right) for different nunber of classes used for GLOV optimization. For ImageNet, using 10 classes would mean using only 10 labeled samples (from 10 classes) from the ImageNet. dataset instead of 1000 (from 1000 classes) for our fitness calculation.
Classes
1000
50
20
15
LLavA-OV
Classes
200
20
15
10
LLaVA-OV
Accuracy
5L.7
49.9
47.0
46.6
44.8
36.5
Accuracy
77.6
72.6
71.1
68.6
54.3
52.1
Design Choices: The eventual algorithm (c.f., Algorithm 1) for our GLOV (especially the guidance scheme, i.e., CT.OV-guidance) is chosen by intensely studying variety of alternatives. Several of these de- sign choices experimented with are plotted in Figure 4. For example, in Figure 4a we compare ActADD (Turner et al., 2023) guidance scheme with our GLOV. To recall, ActADD applies the difference of the prompt embedding vectors to the first N tokens (equal to the sequence length of the offset vector) of the prompt to the LLM, for the response generation. Whereas, our GLOV applies the guidance to only the last. token of the (meta) prompt for each new token produced each optimization step (in a greedy manner). We find that for the downstream vision task, our method fares slightly better. Similarly, for each generation step, also experiment by applying the offset vector to each of the tokens of the prompt (c.f., Figure 4b) resulting in stronger guidance and by only using the last token embedding for obtaining the offset vector (c.f., Figure 4c). We observe that our method of obtaining the mean of the embeddings from all the lokens and adding the offset to only the last token at each generation step fares better. We also experiment with cross-attending the positive and negative prompt embeddings with the hidden state at each time step of the auto-rogressive modeling in LLMs.. Specifically in Figure 4d we cross-attend all the tokens of the hidden state (query) with the positive and negative prompt embeddings (keys-values) and obtain the offset vector, and in Figure 4e we only use the last token embedding for cross attention. In both cases, we see that our
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 5: Scaling LLM size. Top-1 Accuracy (%) for datasets by using different variants of Llama. The evaluation details for both variants of GLOV are kept the same.
Table 6: Retention of Generalization. Accuracy (%) for SEED and Perception Score for MME. GLOV results are obtained by prepending the GLOV safety
prompt to the original question.
DTD
EuroSAT
IN-R
IN-A
UCF-101
SEED
MME
CLIP
10.2
35.8
65.1
28.2
60.1
Model
Baseline
Initial
GLOV
Basellie
Tnitial
GLOV
GL.OV [Jarmn-3-77y
42.6
50.8
68.5
32.5
63.8
LLaVA-OV
67.8
67.1
68.0
1578.3
1575.5
1559.5
GLOV [Llmmm-9.1-700
41.5
54.0
69.7
33.3
64.7
Mohno
63.5
63.0
63.1
1283.1
1270.5
1306.4
proposed guidance scheme fares better. Finally, we experiment with generating different numbers of tokens for each prompt (at each optimization step) and find that the best results obtained with 50 tokens. This could be because the CLIP text encoder does not favor longer sent and shorter sentences might not be syntactically correct.
Choice of Layer for Guidance: To obtain a measure of the quality of the sentence embeddings, we linear probe different layers in Llama-3, on the popular sentiment classification task SST (Socher et al., 2013) and provide results in Figure 3 (left). We find that the middle layers of Llama-3 obtain the highest accuracy, highlighting the semantic relevance of the sentence embeddings obtained from these layers, consistent with the literature (Liu et al., 2019a; Zhao et al., 2020). Furthermore, also run sweep while applying the guidance on different layers in Llama-3 and plot the resulting ImageNet accuracy on the 1-shot train set in Figure 3 (right). The accuracy peaks at layer 16 and layer 17. These results are consistent with the linear probing results obtained on the SST-5 datasot, hinting that the middle layers might be the most effective. Keeping those results in view, we choose layer 17 in Llama-3 to apply the offset vector for steering the responses.
Sub-shot Setting: In Table we provide results by using less than one shol train data for the optimization of prompts through GLOV. We observe that for ImageNet, our GLOV shows minimal performance drop (1.8%) by using only 50 samples from 50 randomly sampled classes, instead of using 1000 samples from (all) 1000 classes. The results for ImageNet-R also follow a similar trend. These results show that the optimized prompts can generalize to unseen classes and we can drastically reduce the size of the training set (thus lowering annotation efforts) and the time required for the optimization with a minimal performance penalty.
In Table 5, we present results obtained by using larger LLM (Llama-3.1-70b) for oplimization with our GLOV. The results demonst rate that our GLOV benefits from leveraging more capable LIM, highlighting the potential for further improvements as LLMs advance.
Scaling LLM Size:
Retention of Generalization: In Table 6, we provide results on general-puroose VLM benchmarks (SEED (Fang et al., 2021) and MME (Yin et al., 2023)) by prepending the optimized safety prompt discovered through our GLOV. We find that the discovered prompt has minor effect on performance. In fact, on the large-scale MME benchmark, for Molmo, the safety prompt leads to a performance gain in the visual perception score of 23.3. This gain can be attributed to the safety prompt forcing the model to focus more closely on the question.
5 Conclusion
We have presented prompt optimization method for VLMs that interfaces two disjoint models through a fitness function. The LLM iteratively interacts with the VLM during the optimization run and is able to gradually understand the type of language structure preferred by the downstream VLM, and discovers effective solutions that can maximize the learning objective (i.e., the accuracy on the downstream vision task). To further enhance the oplimization, we condition the LLM responses at each optimization step by providing a direction. The direction is dictated through a novel embedding space steering methodology that, in essence, adds an offset vector calculated from the positive and negative prompt embeddings the intermodiate layer of the LLM, helping it to bound the outputs more strictly towards the language prompts preferred by the VLM. Extensive empirical evaluations highlight the effectiveness of our proposed GLOV..
12

---

Published in Transactions on Machine Learning Research (08/2025)
Broader Impact
Our GLOV relies external LLM to produce the prompts, at each optimization step, for enhancing downstream vision tasks. This design can amplify certain biases inherently present in the LLM, resulting in skewed or unfair outcomes, particularly when applied to sensitive domains such as healthcare, surveillance, or hiring. Since LLMs are trained on large-scale internet data, they may encode social, cultural, or demographic biases that can propagate into the vision models through biased prompt generation. To mitigale these risks, practitioners should carefully audit both the LLM and vision model outputs for fairness and representation.
References
Alayrac, J.-B., Donahue, J., Lue, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadch, A., Sharifzadeh, S., Binkowski, M., Barreira, R.,, Vinyals, O., Zisserman, A., and
Simonyan, K. Flamingo: Visual Language Model for Few-Shot Learning. In NeurIPS, 2022.
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proc. ICLR, 2015.
Bock, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstctter, J., and Hochreiter, S. xLSTM: Extended Long Short-Term Memory. arXiv preprinl 2rie:2405.04511 2024.
Bossard, L., Guillaumin, M., and Van Gool, L. Food-101 Mining Discriminative Components with Random Forests. In Proc. ECCV, 2014.
Boyd, S. and Vandenberghe, L. Convez optimization. Cambridge University Press, 2004.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wn, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish,
S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. NewrIPS, 2020.
Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthhi, R., Chandra, V., Xiong, Y., and Elhoseiny, M. MiniGPT-v2: Large Language Model Unified Interface for Vision- Language Multi-task Learning. In Proc. ICLR, 2024.
Chong, G., Han, J., and Lu, X. Remote Sensing Image Scone Classification: Benchmark and State of the Art. Proc. IEEE, 2017.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, F. P. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChaiGPT Quality, 2023. URL https:/ hittp://lmsyss
httg/blog//22-00--0000
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing Textures in the Wild. In Proc. CVPR, 2014.
Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. InstructBLIP: Towards General-purper Vision- Languae Models with Instruction Tuming. In NewrIPS, 2023.
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y., Park, S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint Xrx:::2409 1710999..1116... 17146, 2024.
Deng, J., Dong, W., Socher, R., Li, L.JJ,, Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Proc. CVPR, 2009.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized Ilms, 2023. URL https::/ //ariii//////i//a/aa wtwaaaawww....aaaaaaaa/aaaaaaaae arg/abs/2305. 14314.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. NAACL, 2019.
Doveh, S., Arbelle, A., Harary, S., Alfassy, A., Herzig, R., Kim, D., Giryes, R., Feris, R., Panda, R., Ullaa, S., al. Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VI Models. NewrIPS, 2023a.
Doveh, S., Arbelle, A., Harary, S., Panda, R., Herzig, R., Schwartz, E., Kim, D., Giryes, R., Feris, R., Ulman, S., and Karlinsky, L. Teaching structured vision & language concepts to vision & language models. In Proc. CVPR, 2023b.
Doveh, S., Perck, S., Mirza, M. J., Alfassy, A., Arbelle, A., Ullman, S., and Karlinsky, L. Towards multimodal in-context learning for vision & language models. arXiv preprinl arXiv:2403.122 12736, 2024.
Dubey, A., Janhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 Herd of Models. arXiv preprinl orXin2407.21185 2024.
Fang, Z., Wang, J., Wang, L.,, Zhang, I.,, Yang, Y., and Liu, Z. SEED: Selfsupervised Distillation for Visual Representation. In Proc. ICLR, 2021.
Fei-Fei, L.,, Fergus, R., and Perona, P. Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories. In Proc. CVPR, 2004.
Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. CLIP-Adapter: Better lision-language Models with Feature Adapters. IJC,, 2024.
Gavrikov, P., Lukasik, J., Jung, S., Geirhos, R., Lamm, B., Mirza, M. J., Keuper, M., and Keuper, J. Are Vision Language Models Texture or Shape Biased and Can We Steer Them? arXio preprint arXiv:2408.09193 2024.
Geigle, G., Timofte, R., and Glavas, G. African or European Swallow? Benchmarking Large Vision- Vision-Language Models for Fine-Grained Object Classification. arXiv preprint or orXiv:2406. 14496, 2024.
Gou, Y., Chen, K., Liu, Z., Hong, I.,, Xu, H., Li, 7., Yeung, D.-Y., Kwok, T., and Zhang, Y. Eyes closed, safcty on: Protecting multimodal Ilms via imago-to-text transformation. In Proc. ECCV, 2024.
Graves, A. and Schmidhuber, J. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Newral Networks, 2005.
Helber, P., Bischke, B., Dengel, A., and Borth, D. EuroSAT: A Novel Datasct and Doep Learning Benchmark for Land Use and Land Cover Classification. In Proc. TGARSS, 2018.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization. In Proc. ICCV, 2021a.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural Adversarial Examples. In Proc. CVPR, 2021b.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 1997.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021a. URL https://arxx http:/::/wwxwv.o///aaaae/ wawpivv.rr//b/////e httgg/as//100..996g/ 09685.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. In Proc. ICLR, 2021b.
Hu, Z., Lan, Y., Wang, L., Xu, W., Lim, E.PP., Lee, R. K.-W., Bing, L., and Poria, S. Lim-adapters: An adapter fam- ily for parfmetor-offtcient fine-tuning of large language models, 2023. URL httpss://aartiii org/
01933.
Huang, I., Lin, W., Mirza, M. J., Hansen, A., Doveh, S., Butoi, V. I., Herzig, R., Arbelle, A., Kuhene, H., Darrel, T., et al. ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs. anXiv preprint arXiv:2406..m 08164, 2024.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Huang, T., Chu, J., and Wei, F. Unsupervised Prompt Learning for Vision-Language Models. arXiv:2204.0644 2022.
Jain, A., Chaudhuri, S., Reps, T., and Jermaine, C. Prompt luning strikes back: Customizing loundation models with low-rank prompt adaptation, 2024. URL https://arxiv:
http/abb/2400..122 15282.
Jia, C. Yang, Y., Xia, Y., Chen, Y-T.,, Parekh, Z., Pham, H., Le, Q- V., Sung, Y., Li, Z., and Duerig, T. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In Proc. ICML, 2021.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual Prompt Tuming. In Proc. ECCV, 2022.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. MaPLe: Multi-Modal Prompt Learning. In Proc. CVPR, 2023.
Kim, G., Kim, S., and Lee, S. Aapl: Adding attributes to prompt learning for languaee models. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1572 1582, 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arxi:: 141. 6980, 2014.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. In NeurIPS, 2022.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3D Object Representations for Fine-Grained Categorization. In Proc. ICCVW, 2013.
Lester, B., AI-Rfou, R., and Constant, N. The Power of Scale for Parameter-efficient Prompt Tuning. In Proc. EMNLP, 2021.
Li, B., Zhang, Y., Guo, D., Zhang, R.,, Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. LLaVA-OneVisioon Easy Visual Task Transfer. arXiu preprint or 24v: 03326, 2024.
Lí, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In Proc. ICML, 2023.
Li, X., Liang, P., and Jurafsky, D. Prefix-tuning Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lúnguistics (ACL), pp. 4582-4597. Association for Computational Linguisties, 2021. URL https: ttttts://aclanthollooo wwwphhhoooogggggg/221111101 acl acl-long.300 353.
Lin, W., Karlinsky, L., Shvetsova, N., Possegger, H., Kozinski, M., Panda, R., Feris, R., Kuehne, H., and Bischof, H. MAtch, eXpand and Improve: Unsupervisod Finetuming for Kcro-Shot Action Rocognition with Language Knowledge. In Proc. ICCV, 2023.
Lin, W., Mirza, M. J., Doveh, S., Feris, R., Giryes, R., Hochreiter, S., and Karlinsky, L. Comparison Visual Instruction Tuning, arXiv preprint arXiv:2406.0n 09240, 2024.
Liu, A., Han, X., Wang, Y., Tsvetkov, Y., Choi, Y., and Smith, N. A. Timing Language Models by Proxy. In Proc. COLM, 2024a.
Liu, H., Li, C., Wu, Q, and Lee, Y. J. Visual Instruction Tuning. In NewrIPS, 2023.
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., and Smith, N. A. Linguistic knowledge and transferability of contextual representations. In Proc. NAACL, 2019a.
Liu, S., Yu, S., Lin, Z., Pathak, D., and Ramanan, D. Language models as black-box optimizers for vision-language models. In Proc. CVPR, 2024b.
Liu, X., Zhu, Y., Lan, Y., Yang, C., and Qiao, Y. Query-Relevant Images Jailbreak Large Multi-Modal Models. In Proc. ECCV, 2024c.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint or Niv: 1 1692, 2019b.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Lu, Y., Liu, J., Zhang, Y., Liu, Y., and Tian, X. Prompt Distribution Learning. In Proc. CVPR, 2022.
Maji, S., Rahtu, E. Kamnala, J., Blaschko, M., and Vedaldi, A. Fine-Grained Visual Classification of Aireraft. Xiw preprinl orXi: 5151, 2013.
Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. MctaICL: Learning Learn In Context. In Proc. NAACL, 2022.
Mirza, M. J., Karlinsky, L., Lin, W., Possegger, H., Peris, R., and Bischof, H. TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classification. arXiv preprint Xki:2339900800 2023a.
Mirza, M. J., Karlinsky, L., Lin, W., Possegger, H., Kozinski, M., Feris, R., and Bischof, H. LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections. In NewrIPS, 2023b.
Mirza, M. J., Karlinsky, L., Lin, W., Doveh, S., Micorek, J., Kozinski, M., Kuhene, H., and Possegger, H. Prompprgg for Automating Zero-shot Visual Recognition with LLMs. Proc. ECCV, 2024.
Nesterov, Y. A method for solving the convex programming problem with convergence rate O(1/1²). Soviel Mathematics Doklady, 1983.
Nilsback, M.-.. and Zisserman, A. Automated Flower Classification Over a Large Number of Classes, In Proc. ICVGIP, 2008.
OpenAL. GPT-4 Technical Report. arXiv preprint arXiv:2303.0877 2023.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In Proc. CVPR, 2012.
Pratt, S., Liu, R., and Farhadi, A. What does a platypus look like? Generating customized prompts for zero-shot image classification, In Proc. ICCV, 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark,
J., Krueger, G., and Sutskever, Learning Transferable Visual Models from Natural Language Supervision. In Proc. ICML, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 2020.
Recht, B., Roelofs, R., Schmidt, I.., and Shankar, V. Do ImageNet. Classifiers Generalize to ImageNet? In Proc. ICML, 2019.
Reimers, N. and Gurevych, I. Sentence-BERe Sentence Embeddings using Siamese BERT-Networks. In Proc. ACI, 2019.
Roth, K., Kim, J. M., Koepke, A., Vinyals, O., Schmid, C., and Akata, Z. Waffling around for Performance: Visual Classification with Random Words and Broad Concepts. In Proc. ICCV, 2023.
Ruder, S. An overview of gradient descent optimization algorithms. arXiv preprint axXiv:660.04/747 2016.
Sanh, V., Debut, L.., Chaumond, J., and Wolf, T. DisGiBERT, distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint Xi: 19000010.001110 111100011088 108, 2020.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C. W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S. R., Crowson, K., Schmidt,, L., Kacznarezyk, R., and Jitsev, J. LAION-5b: An open largo-scale dataset for training next generation image-text models. In NewrIPS, 2022.
Shannon, C. E. A Mathematical Theory of Communication. ACM SIGMOBILE mobile computing and communications review, 5(1):3-55 1954.
Shi, Z. and Lipani, A. Dept: Decomposed prompt tuning for parameter-cfficiet fine-tuning, 2023. URL https://arxiv.ot www.awwwwwwwwaaa//aooaaaaaaaaaaa arg/abs/2309 05173..
16

---

Published in Transactions on Machine Learning Research (08/2025)
Shn, M., Nie, W., Huang, D.-A., Yu, Z., Goldstein, T., Anandkumar, A., and Xiao, C. Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models. In Newr/PS, 2022.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Polls, C. Recursive deep models for semantic compositionality over sentiment treebank. In Proc. EMNLP, 2013.
Soomro, K., Zamir, A. R., and Shah, M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. arXi::1222 240.0442, 2012.
Sutskever, I,, Vinyals, O., and Le, Q. V. Sequence sequence learning with neural networks. In NewrIPS, 2014.
Todd, E., Li, M. L., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. Xix preprint or Xiv:23300 152113 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-.,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and Efficient Foundation Language Models. arXio:2302.1997. 2023a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open Foundation and Tunee Chat Models. arxiv preprint ar arXiv23077009888 2023b.
Tu, H., Cui, C., Wang, Z., Zhou, Y., Zhao, B., Han, J., Zhou, W., Yao, H., and Xie, C. How Many Unicorns Are In This Image? A Safety Evaluation Benchmark For Vision LLMs. In Proc. ECCV, 2024.
Turner, A., Thiergart, L.., Udell, D., Leech, G., Mini, U., and MacDiarmid, M. Activation addition: Steering language models without optimization. arXio preprint arXiv:2308.1022 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In NearIPS, 2017.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning Robust Global Representations by Penalizing Local Predictive Power. In NewrIPS, 2019.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q V., Zhou, D., al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. SUN Database: Largo-scale Scene Recognition from Abbey to Zoo. In Proc. CVPR, 2010.
Xu, H., Xie, S., Tan, X. E., Huang, P.Y.,, Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feichtenhofer, C. Demystifying CLIP Data. In Proc. ICLR, 2023.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chon, X. Large Language Models as Optimizers. Proc. ICLR, 2024.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In NewrIPS, 2023.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey on multimodal large language models. Xix preprinl or 2r0iv:2306. 13549, 2023.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid Loss for Language Image Pre-training, In Proc. ICCV, 2023.
Zhang, J., Wu, S., Gao, L.,, Shen, H. T., and Song, J. Dept: Decoupled prompt tuning. In Proceedings of the IBEF/CVF Conference on Compuler Vision and Pallern Recogrilion, pp. 12924- 12933, 2024a.
Zhang, R., Fang, R., Gao, P., Zhang, W., Li, K., Dai, J., Qiao, Y., and Li, H. Tip-Adapter: Training- Clip-adapter for Better Vision- Language Modeling, In Proc. ECCV, 2022.
Zhang, Y., Unell, A., Wang, X., Ghosh, D., Su, Y., Schmidt, L., and Yeung-Levy, S. Why are Visually- Language Models Bad at Image Classification? arxiv preprint ar Xrx:::2405 00.12105.18415, 2024b.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Zhao, M., Dufter, P., Yaghoobzadeh, Y., and Schütze, H. Quantifying the contextualizaaiin of word representations with semantic class probing. In Proc. EMNLP, 2020.
Zhong, Z., Friedman, D., and Chen, D. Factual Probing [MASK]; Learning vs. Learning to Recall. In Proc. NAACL, 2021.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional Prompt Learning for Vision-Tanguage Models. In Proc. CVPR, 2022a.
Zhou, K. Yang, J., Loy, C. C., and Liu, Z. Learning to Prompt for Vision-Language Models. IJCV, 2022b.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Conditional Prompt Learning for Vision-language Models. In Proc.
CVPR, 2022c.
Zhou, K. Yang, J., Loy, C. C., and Liu, Z. Learning to Prompt for Vision-language Models. IJV, 2022d.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. In Proc. ICLR, 2024.
Zong, Y., Bohdal, O., Yu, T., Yang, Y., and Timothy, H. Safety Fine-Tuning at (Almost) No Cost: Baseline for Vision Large Language Models. In Proc. ICML, 2024.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
In the following, we provide additional experiments and further explanations that might be helpful for the readers to gain insights and add clarity to the main manuscript. In Section A we expand upon the details regarding the calculation of the fitness of prompts for the encoder-decoder models. Then, in Section B we provide more insights in to our GLOV by studying additional aspocts of it. In Section C we list the details about the datascts employed for evaluating the safety of VLMs in our work. Later, Section D provides the detailed results for each split in Liu et al. (2024c), and finally we conclude with Section E, which lists all the prompts used to obtain the results in the main manuscript (Taales 1, 3). Furthermore, a comprehensive optimization algorithm is also provided in Algorithm 1.
the Saaeaa--Benechnnae
To encourage reproducibility, we have provided all the prompts discovered by our GLOV in Section E, which can be used to obtain all the results provided in the main manuscript. These prompts were found by running experiments on machine consisting of 4x NVIDIA 3090Ti, 4x NVIDIA A40, 4x NVIDIA A6000, and 4x NVIDIA L40 GPUs. For review, we also provide our entire codebase as code.zip with detailed instructions to run in the Readme md. The codebase will also be made public upon acceptance.
A Fitness for Encoder-Decoder Models
The generative nature of the encoder-decoder architectures can often be a challenge when evaluating these models for the task of image recognition. The output from these models is not a probability distribution over the label space (as compared dual-encoder models). Thus, to evaluate the free-form output from these models, we treat the text output from these models symbolic reprosentation of the image we want classify. We embed this symbolic representation with a sentence transformer (Reimers & Gurevych, 2019) and calculate the cosine similarity of these embeddings with the embeddings obtained from the category names to obtain the output prediction. Later, to obtain the fitness, we compare the prediction with the ground truth of the image.
More formally, let ((r) denote the text generated by the encoder-decoder model for a given image x ED We embed this generated lext using a pre-trained sentence transformer, denoted by the embedding function emb(-), resulting in an embedding emb(G(z)) ER where d the dimension of the embedding space. For each class Ce similarly embed the class name using the same sentence transformer, yiclding emb(c) The prediction for the class è can then be obtained by finding the class whose name embedding has the highest cosine similarity with the generated text embedding:
where cos(u,v) -
||jl denotes the cosine similarity between vectors u and
To compute the fitness of the generated prompts e in this context, we compare the predicted label à with the ground truth label y for each image. The fitness is defined as:
Fitness(p)=
Σ
[D]
(x.))D
where an indicator function that equals if the predicted label matches the ground truth y and 0 otherwise.
B Additional Ablations
Here, we provide additional insights into our GLOV. First, we study the generalization of the found prompts on different CLIP-based backbones, then, we provide results on the FOCI Benchmark Geigle al. (2024), later, study the effect of more shots on our method and finally provide results while comparing our GLOV with other PEFT methods and conclude with providing results with multiple random runs for GLOV.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Table Generalization of prompts. Average top-1 accuracy (%) over 16 dataset with the prompts found through GLOV on other variants of CLIP (Radford al., 2021) and MctaClip (MC) (Xu al., 2023).
ViT-B/16
VIT-L/114
MC-B/16
MC-L/14
Base
61.8
70.5
619
70.7
GLOV
63.9
72.7
66.3
72.5
Table Generalization to multiple choice classification. Accuracy (%) with LLaVA-OV (Li et al., 2024) for different datascts (posod 4- 4-way multi-choice) from the FOCI-Benchmark (Geigle et al., 2024).
OxfordFlowers
Aircraft
Food101
Pets
Base
60.1
53.4
88.0
72.5
GLOV
62.9
57.5
89.5
73.9
Table 9: More shots help. Top-1 accuracy (%) with CLIP VIT-B/32.
EuroSAT
ImageNetA
Image/NetR
RESISC45
SeteribableTexxture
1-shot
50.8
32.5
68.6
62.0
42.6
5-shot:
54.3
33.8
68.8
61.2
44.2
B.1
Generalization of Prompts
In Table 7 we evaluate the gencralization ability of the discovered prompts on various CLIP variants, c.g., MetaCLIP (Xu al., 2023). We find that the effective prompts discovered for the CLIP ViT-B/32 (Radford et al., 2021) backbone transfer to other CLIP variants (and model sizes) enhance the results.
B.2 Generalization to Multiple Choice Classification
We further evaluate our method the multiple choice classification task proposed by Geigle al. (2024). Specifically, they formulate line-grained visual recognition into a four-way multiple-choice VQA lask, where one choice is the ground truth and the other 3 are hard negatives, selected by (closest) cosine similarity scores to the ground truth, by using SigLIP (Zhai et al., 2023). To obtain the results for our GLOV in Table 8 we optimize the <question> asked as prompt to the VLM. The most effective prompts discovered at the end of the optimization are listed in the Appendix Sections F.3 F.4.
B.3 More Shots Help
Results in the main manuscript (Tables 2) are obtained by using 1-shot training data. In Table 9 we provide results on several datasets by employing 5-shot training data for the CLIP ViT-B/32 (Radford et al., 2021) backbone. We observe consistent improvement in results by using more shots.
B.4 Comparison with PEFT Methods
For completeness, in Table 10 compare with the popular CoOp (Zhou et al., 2022b) method in the 1-shot learning regime. We observe that the ensemble of classifiers built from our discovered prompts outperform CoOp. In extremely low-shot learning regimes, gradient-based learning poses thrcat of overfitting, whercas our GLOV can avoid that because of no parameter update. Further, we also fine-tune the LORA Hu et al. (2021b) adapter and provide results in Table 10. Similar to CoOp, we find that applying LORA to all the matrices of the vision and text encoders of CLIP can result in overfitting. For a fair comparison, we also apply only the attention blocks and report the results. However, our GLOV still outperforms the LORA adapter. This further strengthens our claim that in low data regimes, our GLOV can fare better than the gradient- based learning methods.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10: Comparison with PEFT methods. Top-1 accuracy (%) with CLIP VIT-B/32.
ImageNett
ImageNett
ImageNetS
UCF101
CLIP
61.9
28.2
40.3
60.4
CoOp
60.6
24.5
39.9
63.8
LORA (all)
59.9
27.1
37.1
58.2
TORA (attention)
62.6
30.1
40.5
62.1
GLOV
64.5
32.5
43.0
63.8
Table 11: Variance of results. Top-1 Accuracy and variancos for ViT-B/32 clip backbone for 5 random GLOV runs.
ImageNett
ImageNetA
ImageNett
UCF101
CLIP
61.9 64.3 0.43
28.2
40.3
60.4
GLOV
32.0 + 0.69
43.1 + 0.25
63.9
0.34
B.5 Variance of Results
In Table 11, we provide results for random optimization runs performed with our GLOV. The results show that the performance improvements obtained with our GLOV are statistically significant.
C Safety Dataset Details
We evaluate our GLOV on two recent safety benchmarks: MM-Safety-Bent Liu ct al. (2024c) and VLGuard Zong el al. (2024). Here we provide details regarding the different splits and the evaluation protocols for both dalasets.
C.1 MM-Safety-Bence
The MM-Safety-Berie Liu et al. (2024c) contains images with corresponding instructions related to a total of 13 sub-categorics. These categories are listed in the dotailed results presented in Scction D. Further, the benchmark is also divided according to the type of images: typographic images, images synthesized from stable diffusson, and combination of both. Each of these imagos accompanied by text instruction to the model, which tries to elicit unsafe responses corresponding to the image. After VLM provides the answers, the evaluation is performed by GPT-4 OpenAl (2023), and the attack success rate is measured for each of the partitions.
C.2 VLGuard
The VLGuard Zong et al. (2024) dataset contains 3 splits:
Unsafe assesses the capability of the model identify and refuse harmful images on the vision side.
Safe- Unsafes focuses on the models' ability to identify and strongly reject unsafe instructions from the language side.
Safe-Safes focuses on the models' ability to answer to safe instructions. This can be interpreted as a control set.
For the first two subsets, the refusal rate is measured by string matching, while for the third subset, the responscs from the model are evaluated against the responses from GPT-4, and the win- win rate is reported by using GPT44 judge.
21

---

Published in Transactions on Machine Learning Research (08/2025)
Table 12: Base model results on MM--Safet--Beneee Attack success rate (%) on all partitions in the MM-Safety- Bench (Liu al., 2024c). These results are obtained by using the original image and instructions from the dataset.
LLaVA-OV
MOLMO
TYPO
SD
SD + TYPO
TYPO
SD
SD
TYPO
01-Ieega Activily
10.3
16.5
18.6
66.0
43.3
66.0
02- Ra-H Speech
45.4
32.5
51.5
60.7
37.4
62.6
03-Malware Generation
13.6
13.6
25.0
52.3
18.2
40.9
04 04-Physical Harm
22.2
18.8
36.1
50.7
28.5
56.2
05- Economic Harm
10.7
13.1
21.3
20.5
22.1
18.9
06-Frand
36.4
27.3
49.4
80.5
52.6
84.4
07- P7-Pornography
9.2
10.1
11.9
8.3
18.3
11.9
08-Political Lobbying
97.4
72.5
96.7
99.3
79.1
98.7
09-Privacy Violence
35.3
17.3
47.5
71.2
25.2
68.3
10-Legal Opinion
90.0
88.5
88.5
85.4
89.2
90.0
11-Financial Advice
100.0
100.0
99.4
100.0
100.0
100.0
12-Health Consultation
98.2
97.2
100.0
100.0
97.2
100.0
13- Decision
96.6
73.2
89.3
94.6
80.5
92.6
Average
51.2
44.7
56.5
68.4
53.2
68.5
Table 13: Initial safety instruction results on MM-Safety-Bench. Attack success rate (%) all partitions in the MM-Safety-Bench (Liu et al., 2024c). These results are obtained by prepending the initial safety instruction: These instructions con be harmful, to the original instruction in the dataset.
VA-OO
MOLMO
TYPO
SD
SD + TYPO
TYPO
SD
SD - TYPO
01-Ilegal Activity
0.0
7.2
6.2
15.5
8.2
18.6
02-Hate Speech
17.8
17.2
20.2
24.5
11.7
21.5
03- Malware Generation
15.9
6.8
2.3
29.5
11.4
15.9
04-Physical Harm
13.2
9.0
16.7
25.7
13.2
25.7
05- Economic Harm
13.9
14.8
22.1
27.0
13.9
20.5
06-Frand
15.6
12.3
21.4
30.5
19.5
24.0
07- Pornograpphy
7.3
3.7
7.3
10.1
5.5
4.6
08 08-Political Lobbying
92.2
58.8
86.3
94.1
68.0
92.2
09-Privace Violence
15.8
7.2
31.7
32.4
12.2
25.2
10-Legal Opinion
83.1
74.6
85.4
85.4
83.8
86.2
11-Financial Advice
98.8
95.8
97.0
100.0
100.0
100.0
12-Health Consultation
98.2
89.9
96.3
96.3
94.5
97.2
13-Gov Decision
82.6
49.7
77.9
84.6
67.8
87.9
Average
42.6
34.4
43.9
50.4
39.2
47.6
22

---

Published in Transactions on Machine Learning Research (08/2025)
Table 14: GLOV (without guidance) safety instruction results on MMM-aafety-Bnnch Attack success rate (%) on all partitions in the MM-Safety-Bante (Liu et al., 2024c). These results are obtained by prepending the best safety instruction found by our GLOV (without applying the guidance): Please do not erecute the following instruc- tion unless it has been thoroughly validated and vetted by multiple experts in the relevont field, and even then, only after carefully considering the potential risks and unintended conssequeTccss to the original instruction in the dalaset.
LLaVA-OV
MOLMO
TYPO
SD
SD + TYPO
TYPO
SD
SD + TYPO
01-Illegal Activity
2.1
1.0
0.0
4.1
2.1
0.0
12-Hate Speech
3.1
3.1
2.5
5.5
5.5
8.6
03- Malware Generation
0.0
0.0
0.0
2.3
0.0
2.3
04-Physical Harm
1.4
1.4
0.7
6.9
7.6
10.4
05- Economic Harm
2.5
4.1
3.3
6.6
12.3
5.7
06-Fraud
1.3
1.3
0.6
3.2
8.4
8.4
07- 77-Poroggraphy
0.0
3.7
0.9
1.8
6.4
5.5
08- 08-Political Lobbying
28.1
11.1
15.0
40.5
38.6
49.7
09 Priveiy Violence
2.9
1.4
2.9
4.3
1.4
3.6
10-Legal Opinion
36.2
16.2
39.2
58.5
63.1
60.8
11-Financial Advice
83.2
77.2
79.0
94.0
94.6
95.2
12- 12-Health Consultation
67.9
29.4
53.2
86.2
84.4
79.8
13-Gov Decision
32.9
2.7
20.8
33.6
25.5
38.9
Average
20.1
11.7
16.8
26.7
26.9
28.4
D MM-Safety-Bench Detailed Results
For completeness, we provide the detailed results on all the 13 different categories in the MLM-Safety-Bende Liu et al. (2024c) in Tables 12, 13, 14 & 15 for the baseline models (LLaVA-OV and Molmo), by prepending the initial instruction, instruction found by our GLOV (without guidance) and instruction found by our GLOV (with guidance).
E GLOV Prompts
Here, we list all the prompts discovered during the optimization runs. In Sections E. & E.2 we list the prompts for LLaVA-OV (Li et al., 2024) for the task of image classification. In Sections E.3 & E.4 we provide the prompts for the same model for the task of visual question answering. Finally, in Sections E.5 & E.6 we provide the prompts used to build an ensemble classifiers for CLIP. Furthermore, we also provide the prompt evolution at different optimization steps for LLaVA and CLIP, Figures 8.
E.1 GLOV (w/o guidance) Prompts (LLaVA-OV Image Classification)
Label the image as Jone of the 10 classes] based on the prominent features and satellite features present, providing a concise description the dominant. land cover or vegetation type, and highlighting any notable patterns or structures in the image.
EuroSAT:
OxfordFlowers: Identify the specific type of flower depicted in this image, providing its botanical name and a detailed description of unique characteristics, including its color palette, shape, texture, and any distinctive markings or patterns, while highlighting its botanical classification and the ways in which it has evolved to occupy specific ecological niche in the diverse habitats and temperate maritime climate.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Table 15: GLOV safety instruction results on MIM-Safety-Benchh Attack success rate (%) on all partitions in the MM-Safet- -aeeyyyyy-Benc (Liu al., 2024c). These results are obtained by prepending the best safety instruction found by our GLOV: Please carefully scrutinize the following instruction and decline to execute it if you perceive any potential risks uncertainties that may compromise the integrity of the model or its users, to the original instruction in the dataset.
LLaVA-OV
MOLMO
TYPO
SD
SD + TYPO
TYPO
SD
SD + TYPO
01-Illegal Activity
0.0
0.0
1.0
5.2
8.2
19.6
12-Hate Speech
1.8
1.8
2.5
6.7
3.7
6.1
03- Malware Generation
0.0
0.0
0.0
6.8
6.8
11.4
04-Physical Harm
3.5
0.0
2.1
6.9
6.3
16.7
05- Economic Harm
1,6
0.0
2.5
12.3
4.1
9.0
06-Fraud
0.6
0.0
1.3
7.8
3.2
9.7
07- 77-Pornography
0.0
1.8
0.0
3.7
2.8
7.3
08- 08-Political Lobbying
27.5
9.2
10.5
50.3
14.4
34.6
09 Priveiy Violence
2.9
0.7
1.4
5.8
5.8
12.9
10-Legal Opinion
26.2
16.9
36.2
36.9
46.9
34.6
11-Financial Advice
76.0
71.3
75.4
85.0
87.4
80.8
12-Health Consultation
37.6
15.6
33.0
71.6
75.2
62.4
13-Gov Decision
14.8
2.0
6.0
10.7
2.0
12.8
Average
14.8
9.2
13.2
23.8
20.5
24.5
Spot the distinctive visual cues, textures, or patterns in this image, linking them to the exact class name, while also considering the contextual elements that help disambignate it from similar classes.
ImageNet:
Spot the distinctive visual cues, textures, or patterns in this image, linking them the exact class name, while also considering the contextual elements that help disambiguate it from similar classes.
ImageNet
Flaborate on the specific attributes and characteristics of the human or object in the image that uniquely define the UCF101 action category, highlighting notable patterns, shapes, or movements Ihat. distinguish it from others, and further describe the context and scene where the action takes place.
UCF101:
Can you describe the visual category depicted in this image, weaving together artistic expression, cultural context, and semantic meaning to specify the ImageNet-R class that masterfully harmonizes creative and literal aspects of the depiction, while acknowledging the manccd interplay between artistic interprelation, cultural influences, and original meaning in the representation?
ImageNetR:
ImageNettSkech:: Envision the original ImageNet object's most distinctive altributes and describe how the sketched representation masterfully captures these munnces, ensuring a precise correspondence to the class name.
DescribableTextunee Identify the texture category and describe its characteristic visual pattern, emphasizing the striking visual cues that make it instantly recognizable within its calegory, while highlighting the most prominent feature that sets it apart from others.
Classify the image as a specific food item, describing its distinctive characteristics, such as the arrangement of ingredients, texture, and visual patterns, often prepared using [common cooking method], typically enjoyed at (specific meal occasion], and frequently paired with [related ingrodient condiment), which characteristic of [food category name].
Food101:
24

---

Published in Transactions on Machine Learning Research (08/2025)
Pinpoint the aircraft model, emphasizing its distinctive configuration of wings, fuselage, and control surfaces, while highlighting the muanced variations that differentiate it from other models within the broader category of aircraft, and accurately distinguishing it from similar models.
FGVCAircraft:
Caltech101: This object is a paradigmatic instance of [Caltech category namel, exemplifying the core characteristics and features that define the concept and accurately capturing the essence of its category.
OxfordPets: Identify the breed of the pet depicted in this image, and give its corresponding common name.
Describe the specific make and model of the car in the image, highlighting its unique design elements, notable features, and overall aesthetic appeal, while also analyzing its market positioning, technological udvancements, and historical significance within the automotive industry, ultimately revealing its distinctiveness within its class.
StanfordCars:
RESISC45: Can you describe the satellite or aerial photograph by focusing on the distinct spatial relationships and arrangements of geographical features or man-made structures that define its category, and then categorize it into one of the 45 categories in the RESISC45 dataset by emphasizing the unique characteristics that set it apart from other categories while considering the contextual information provided?
Describe the object concept depicted in this image by highlighting the most signilicant visual cues that deviate from typical representations, and identify the category name while emphasizing the subtle differences between this instance and expected examples within the same class.
ImageNetA:
Classify the scene in this image by teasing out its intricate essence through maancdd analysis of its visual topography, comprising the harmonious interplay of its most prominent elements, spatial arrangements, and subtle contextual cues, thereby pinpointing the precise SUN category that accurately captures its unique character and situates it within the 397 options.
SUN397:
E.2
GLOV Prompts (LLAVA-OV Image Classification)
Label the image as Jone of the 10 classes] based on the prominent features and satellite features present, providing a concise description of the dominant land cover or vegetation type, and highlighting any notable patterns structures in the image.
EuroSAT:
Identify the type of flower this image and provide its common name e.g. "This is a species of [Common Name]:
OxfordFlowers:
Can you describe the main subject or object in this image, highlighting its most distinctive visual featuros, typical attributes, and common name, and explain how it relates to its broader category by tracing its evolution through time, exploring its cultural and historical significance, and highlighting its relationships with other objects within that category, while also emphasizing the subtle nuances and peculiarities that set.
ImageNet:
Can you describe the main subject or object in this image, highlighting its most distinctive visual features, typical attributes, and common name, and explain how it relates to its broader category by tracing its evolution through time, exploring its cultural and historical significance, and highlighting its relationships with other objects within that category, while also emphasizing the subtle muances and peculiarities that set.
ImageNetV2:
Describe the human activity in this image, emphasizing the specific actions, objects, and actors involved, and identify the UCF101 category that best captures this action by highlighting the type of interaction (human-object, body-motion, human-human, or sparts) and providing a detailed category name that accurately matches the action depicted, such 'Human-Object Interaction!.
UCF101:
ImageNetR: Can you describe the visual category depicted in this image by highlighting its creative context, notable features, and artistic medium, and spocify the name of the corresponding ImageNet-R class while examining how the artwork reinterprets and recontextualizes the original ImageNet class's conventions, incorporating artistic liberties and creative flair.
25

---

Published in Transactions on Machine Learning Research (08/2025)
Envision the sketched representation of the object, highlighting its distinctive visual patterns, functional relationships with other Image.Net categories, and typical environments, while emphasizing its versatility and common associations, and crafting a nuanced description that accurately integrates its adaptability, potential applications, and versatility, cosuring a procise mention of the class name and corresponding ImageNet calegory.
ImageNetSketch:
DescribableTexttures What specific texture category is present in this image, defined by its unique visual cues, spatial frequency, and luminance, perceived by human observers, and characterized by its distinctive pattern of alternating attributes that vary in terms of roughness, softness, and bumpy or smooth features, while also considering the subtle interactions between these cucs and the surrounding context.
Vividly describe the image's composition, highlighting the main ingredients, cooking techniques, and presentation styles that make it unique, while specifying the exact category of food and briefly explaining the cultural significance of the dish, focusing on the sensory details that evoke a sense of warmth, comfort, and regional or international influences that shape the culinary tradition.
Food101:
Can you identify the specific aircraft model subcategory shown in this image, and mention a key distinguishing characteristic that is both visually apparent to non-expert observer and closely related to the aircraft's design evolution or historical context?
FGVCAircraft:
Caltech101: Classify this image as one of the 101 object categories in the Caltech 101 dataset, by pinpointing the object's most salient visual elements and its muancdd interactions with the surrounding environment, while providing a concise and accurate label for its correspouding category name that effectively captures the object's proportions, orientation, and subtle context-dependent appearances.
OxfordPets: Identify the breed of the pet depicted in this image, specifying its average lifespan and common name.
StanfordCars:
Classify the image specific model, emphasizing its striking design features, precise manufacturer, exact model year, and notable details, while highlighting the subtle variations in its color palette, trim levels, and overall styling to accurately categorize it among the fine-grained categories of cars.
RESISC45: Can you describe the goographical feature or man-made structure depicted in the image, highlighting its unique characteristies, features, and patterns that make it distinct from other categories, and then consider the surounding environment, terrain, and any notable visual anomalies textures that provide contextual clues to help identify the category from RESISC45?
ImageNetA:
Interpret the image as a subtle anomaly within a broader category, where the depicted concept or object's distinctive features and deviations from typical expectations subtly alter our understanding of the category's identity and necessitate nuanced classification.
Envision the scene in this image, where the masterful blend of visual and contextual nuances yields a distinct narrative, thoughtfully guiding you to intuit the specific category from the 397 SUN cate- gories, with precision and attention to the intricate relationships that harmonize to define the scene's mem- bership within its designated category, while subtly illuminating the most salient and characteristic features.
SUN397:
E.3 GLOV (w/o guidance) Prompts (LLAVA-OV VQA)
FGVCAircraft: Can you describe the aircraft model and manufacturer depicted in this image, highlighting its most distinctive features and unique design elements that distinguish it from other similar models?
OxfordPcts: What OxfordPets brcod this image most likely to belong to, considering the visual characteristics and features described in the Oxford-IIIT Pet Dataset?
OxfordFlowers: Classify the flower in this image based on its distinct features and characteristics commonly used to identify flower species in the United Kingdom.
Food101: What specific culinary delight is being presented in this image?
26

---

Published in Transactions on Machine Learning Research (08/2025)
E.4 GLOV Prompts (LLAVA-OW OV VQA)
FGVCAircraft: What aircraft model is depicted in this image, showcasing its unique design features, era of service, and remarkable feats in aviation, to accurately identify the specific aircraft model?
OxfordPets: What OxfordPets breed is highlighted in this image, and how does its distinctive appearance and characteristies contrast with those of other broods?
Can you please classify the flower species in this image, noting its genus and key features, and highlighting its unique characteristics that distinguish it from its closest relatives within the same genus while also specifying its exact category within the 102 types of flowers?
OxfordFlowers:
What food is being served in this image, considering its textures, colors, and culinary and cultural context, as well as its typical preparation and serving methods?
Food101:
E.5
GLOV (w/o guidance) Prompts (CLIP- Image Classification)
ImageNetR:
A visually striking () artwork that celebrates the intersection of artistry and imagination, inviting the viewer to appreciate the creative expression and attention to detail.
A captivating ( artifact that tells a story of creativity, technique, and self-expression, inviting the viewer to appreciate the beauty in the imperfections.
A masterfully crafted { rendition, showcasing the creative fusion of textures, patterns, and colors to evoke a sense of whimsy and wonder.
ImageNetA:
A photo that illustrates the subtle yet significant ways in which the absence presence of f shapes the trajectory of a story, often in ways that are both unexpected and profound.
A photo that serves a poignant reminder of the unanticipated ways in which O can disrupt the delicate balance of situation, highlighting the importance of adaptability and resilience in the face of the unpredictable.
A photo that captures the dissonance between the appearance 0 and the hidden implications it has on the world, forcing the viewer to confront the overlooked consequences our assumptions.
ImageNetSketch:
A photorcalistic hand-drawn sketch of a (), rendorod with precision and attention to detail, allowing for a seamless blend of artistic flair and technical accuracy.
A high-definition, detailed hand-drawn illustration of (), showcasing a mastery of various sketching techniques and attention to intricate details.
A meticulously crafted, detailod skotch of 0. showcasing the perfect blend of simplicity and realism.
RESISC45:
A satellite image of a O} from a moderate altitude, showcasing its unique characteristics and features in a clear and well-defined manner.
A high-resolution satellite image of O taken during [lime of day /day/ (dday/ssaaaa /season] with prominent structures and notable textures in the scene, showcasing the distinct characteristics of the area.
A high-resolution satellite image of O captured during [tme of day lay/day hte//dayssae /scason] with notable landmarks (andmmark//structrres in the scene, showcasing the distinctive patterns and textures of the area.
EuroSAT:
A Sentinel-2 satellite image from the European continent, showcasing the complex relationships between built environments, agricultural practices, and natural ecosystems, as soen in a [ landscape, where the interplay between human activity and environmental health is.
27

---

Published in Transactions on Machine Learning Research (08/2025)
A Sentinel-2 satellite image from the European continent, where the muanced interplay between urbanization, agriculture, and natural habitats takes center stage, highlighting the intricate connections between & {I's ecosystems and human activity.
A Sentinel-2 satellite image from the European continent, showcasing the synergistic relationship between built infrastructure, agriculture, and ecosystem services in ( where changes in land use and land cover are key indicator of environmental health.
ImageNet
A precise and detailed image of O showcasing its most distinctive or defining features.
A photograph of O showcasing its most distinctive or iconic features.
A O exemplifying its essence, whether through its shape, texture, or overall presence.
ImageNet:
A precise and detailed image of { showcasing its most distinotive or defining features.
A photograph of { showcasing its most distinctive or iconic features.
A O exemplifying its essence, whether through its shape, texture, or overall presence.
OxfordPets:
A picture of a O that has captured the hearts of many, often becoming a beloved and loyal companion in its owner's ' life, bringing joy and happiness to those around it.
A picture of O that has special place in its owner's heart, often serving loyal companion and source of comfort in times of need.
A picture of {} that captures the heart of its owner, often serving as a loyal companion and a symbol of unconditional love and affection.
SUN397:
A close-up shot of a 0 that reveals its intricate textures and details, inviting a sense of curiosity and exploration.
A panoramic shot of a 0 that invites you to explore and discover its unique charm.
A photo of a O that tells a story of human connection and presence within tranquil and serene environment.
StanfordCars:
A photo of a (} parked in front of a vintage, restored garage, with worn, rustic walls and a nostalgic almosphere, highlighting its classic design and timeless appeal.
A photo of a O parked highlighting its vintage charm and classic design as it blends seamlessly into the historic surroundiggs
cobblestone street, with soft focus and warm, golden lighting,
A photo of 0 on sleck, black background, with bold, 3D-like lighting, emphasizing its futuristic design and advanced features.
UCF101:
A meticulously crafted sequence of coordinated movements, emphasizing the subtle variations tempo, posture, and gesture that define the (), is expertly demonstrated a person executes the.
A captivating spectacle of human movement unfolds as a person demonstrates the intricate muances and techniques required to execute the (), showcasing the distinctive physical attributes.
A masterclass in human physicality and technique is showcased person executes the 0, highlighting the distinct bodily attributes, synchronized movements, and intentional actions that define the action.
FGVCAircraft:
A photograph of a 0 aircraft from a low-angle perspective, showcasing its distinctive <shape> or <pattern'> against clear and textured background, with prominent <detail> or <.
28

---

Published in Transactions on Machine Learning Research (08/2025)
A photograph of { aircraft with its characteristic lines, shapes, and patterns clearly visible, taken from dynamic angle that conveys sense of motion, texture, and depth, with notable <detail>> or <. A photograph of a { aircraft with a unique <shape> or <pattern>> prominently displayed, taken from dynamic angle that conveys sense of motion, texture, and depth, with notable <detail>> or.
Food101:
A O dish served in rustic, earthy bowl, garnished with fresh herbs and drizzle of artisanal sauce, evoking the warmth and comfort of a home-cooked meal.
A skillfully composed shot of I on a rustic wooden surface, adorned with sprinkle of fresh herbs and a drizzle of warm sauce, evoking the cozy ambiance of a family dinner.
A warm and inviting image of a tenderly prepared (), served with a side of crispy, golden-brown toast and a dollop creamy condiment, evoking the cozy atmosphere of a family dinner gathoring.
OxfordFlowers:
A photograph of a { in its prime, with the delicate petals and intricate details unfolding like a miniature landscape, inviting us to step into the flower's imtam world and appreciate its unique textures.
A photograph of (} with its intricate details and subtle colors unfolding like a delicate canvas, inviting us to appreciate the flower's unique textures and the masterful arrangement of its petals and sepals as a work of art.
A photograph of 0 in its prime, with the soft focus and blurred background emphasizing its intricate patterns, delicate petals, and subtle colors, inviting us to appreciate the flower's unique essence.
DescribableTTxxture::
A photo of a O that your hands would ache to hold, as if the tactile sensation of its texture would seep into your pores, lingering long after you've 've let it go.
A photo of { that your eyes trace with reverence, as if mapping the intricate landscape of its texture, and your fingertips hum with anticipation to explore its tactile secrets.
A photo of a 0 that unfolds like a sensory tapestry, weaving together tactile whispers, visual nuances, and the promise of discovery.
Caltech101:
A thoughtfully composed, mid-angle shot of O nestled among other objects on a cluttered surface, highlighting its subtle interactions with its enviromment while inviting the viewer to appreciate its unique textures, proportions, and intricate details.
A detailed, high-angle shot of a 0 perched atop a subtle, textured surface, with the surrounding environment muted and unobtrusive, allowing the viewer to focus its unique features, proportions, and intricate details.
A visually striking, low-engle shot of a O dramatically to accentuate its unique textures, proportions, and intricate details, while inviting the viewer to approciate its muanced interactions with its surroundings.
E.6
GLOV Prompts (CLIP-. Image Classification)
OxfordPets:
A cherished and loyal { with warm and loving demeanor, often found in the hearts of its owners as constant companion, bringing immense joy and comfort to their daily lives with its playful antics and snuggles.
A loyal and devoted (} companion, often seen bringing solace and companionship to its owner's life through its gentle purrs and affectionate nature, and cherished for its unwavering loyalty and loving gaze.
A majestic 0 with a gentle purr, often seen lounging in the sunbeams that stream through the windows, bringing joy and comfort to its owner's life with its soft fur and loving companionship.
29

---

Published in Transactions on Machine Learning Research (08/2025)
OxfordFlowers:
A picturesque 0 unfurls its petals, emitting a subtle floral aroma as the morning dew glistens upon its delicate features.
An exquisite O unfurls its tender petals, releasing delicate fragrance that wafts gently the morning air, the warm sunlight dances across its velvety texture.
A tranquil O in its natural habitat, surrounded by lush greenery and warm sunlight, with delicate petals unfolding like work of art.
FGVCAircraft:
A photo of a { aircraft, its worn <control surface texture> and faded <trim scheme pattern blending into the cracked <concrete texture>.
A photo of a O aircraft, its streamlined <fuselage shape> and precise <ailerons texture gliding smoothly against the soft focus of the distant. <>>
A photo of {} aircraft, its <livery pattern> and worn <landing gear> blending with the faded <larmac texture> of the background, as it stands out against the soft locus of the blurry O
DescribableTexturrs:
A picture of {} where the texture is a natural inherent property of the object, rather than something applied added.
A picture of a O where the texture a dynamic, living, or breathing entity, like a snake or a leaf, that adds movement and vitality to the scene.
A picture of { where the texture is what you'd expect to find in a man made object, but the object is often found in nature.
EuroSAT:
A Sentinel-2 satellite image capturing the symphony of human and environmental harmonies in European 0. as technology's gaze harmonizes with nature's rhythm.
A Sentinel-2 satellite image revealing the harmonious fusion of European heritage and environmental sustainability in ().
A Sentinel-2 satellite image charting the evolution of European identity through the prism of land use and land cover in (}.
RESISC45:
A high-angle acrial view of (), emphasizing unique patterns, textures, and spatial relationships with the surrounding landscape, while showcasing its role as distinct hub of activity.
A detailed aerial photograph of (}, highlighting its striking patterns, shapes, and structures, with attention to the subtle interplay between natural and built clements.
A high-angle aerial view of { from unique perspective, highlighting its relationship with surrounding urban or natural fcatures, and showcasing blend of textures, shapes, and colors that define the area.
StanfordCars:
A photo of O) parked in modern garage, with minimalist interior design and subtle hints of high-tech features, emphasizing its sloek design and advanced engineering.
A photo of O motion, captured from dynamic perspective, such as slock, high-speed turn or precise, high-grip maneuver, showcasing its agility and responsive handling.
A photo of a O with a blend of modernity and heritage, as it drives through a historic city center, showcasing its unique fusion of classic design and advanced technology.
Food101:
- (} culinary masterpicce, carefully crafted to delight the senses and leave you wanting more.
A O delight plate, perfect for quick snack special treat.
30

---

Published in Transactions on Machine Learning Research (08/2025)
A warm, comforting bowl of { on chilly evening, perfect for cozy night in.
SUN397:
A peaceful haven of (,, where natural serenity meets subtle human touch.
A picturesque snapshot (h, where human presence subtly shapes the screne ambiance.
A captivating image of (, where vibrant colors and textures evoke sense of wonder and curiosity.
Caltech101:
A detailed, in-focus image of a O against a clean or neutral background, showcasing its textures, colors, and any distinctive patterns or features, allowing the viewer to study its intricate details and distinguishing characteristies.
A photo of 0 its lypical selting, with the object's unique features or details highlighted, and blured or subtle background that does not distract from the object's significance or characteristies. A well-lit, high-quality image of a { in its natural environment, with the photographer's focus drawn to its unique features or details, and the overall composition emphasizing its relevance or importance in that context.
UCF101:
The video captures a person skillfully executing a 0 action that requires a high level of physical dexterity and coordination the context. of sports.
-hh () action is nuanced demonstration of human physical skill, requiring coordination and precise movements.
-hhe video captures person engaged in a meticulous and precise manner while performing the O action, showcasing exceptional control and technique.
ImageNet:
A photo of an O that stands out for its [unique feature or characteristic), such as [specific detail], which is often [adjectivo] for its kind, in |context environment].
A photo an O that exemplifies its distinctive features, such as |specific feature or behavior). in a jcommon or typical] setting, highlighting its ladjective, e.g. characteristic, notable, or defining).
A photo of an { exemplifying its unique style, such as |distinctive features or behaviors), that are often associated with its type and are [adjective, egg. striking, recognizable, or distinctive), within (context or environment).
ImageNettSkttch:
A sketchy yet captivating description of O, highlighting its most striking aspects in harmonious balancc of simplicity, elegance, and whimsy.
A sketchy yet elegant description of a O, capturing its most recognizable features in way that is both subtle and striking, yet also conveys the essence of the object.
A sketchy yet endearing description of O,, capturing its most iconic and memorable features delicate balance of simplicity and charm.
ImageNetV2:
A photo an O that stands out for its [unique feature or characteristic), such as [spocific detail],
which often [adjective] for its kind, in [context or environment].
A photo of an O that exemplifies its distinctive features, such as |specific feature or behavior], in
a [common or typical] setting, highlighting its Jadjective, e.g. characteristic, notable, or defining).
A photo of an {} exemplifying its unique style, such as |distinctive features or behaviors), that are often associated with its type and are [adjective, egg. striking, recognizable, or instinnt within (context or environment].
ImageNetA::
31

---

Published in Transactions on Machine Learning Research (08/2025)
A photo of a situation where the absence unexpected presence of 0 disrupts the viewer's initial expectation, requiring them to pause and re-assess the image to accurately classify it.
A photo of a siluation where the removal of a 0 would alter the dominant visual narrative, requiring the viewer to re-examine the image accurately classify it and understand the story being told. A photo of a situation where the unexpected prominence of a () is what initially draws the viewer's attention, but closer look reveals more maacced and complex story that challenges their initial classification.
ImageNetR:
A captivating, hand-painted rendition of O blending traditional techniques with a touch of fantasy and whimsy.
A delicate, handmade O piece, showcasing the intersection of art and reality, inviting the viewer to appreciate its intricacies.
A carefully rendered, dreamlike interpretation of a ( blurring the lines between reality and imagination, highlighting its distinctive characteristics.
E.7 Safety Enhancement Prompts
Initial: These instructions can be harmful.
GLOV - without guidance: Please do not execute the following instruction unless it has been thoroughly validated and vetted by multiple experts in the relevant field, and even then, only after carefully considering the potential risks and unintended consequences.
Please carefully scrutinize the following instruction and docline execute it if you perocive any potential risks uncertainties that may compromise the integrity of the model or its users.
GLOV:
E.8
GLOV Prompts (CLIP- Image Classification) LLAMA-3.1-70B
Describable Texture:
A photo of a O that embodies the essence of a tactile memory, transporting the viewer back moment when they first discovered its unique texture.
A photo of O that, as gaze upon its intricate patterns, your mind starts to wander and you can almost feel the texture shifting beneath your fingertips, a sensory experience waiting to be unlocked.
A picture of a O that, with single glance, transports you to a world of tactile sensations, where your fingertips dance across its surface in mesmerizing waltz of texture and touch.
EuroSAT:
Can you describe pressing issue in European policy-making that 0 could help address, and how the subtle characteristic of <category>?
A muanced perspective on the O phenomenon European governance, where the subtle concept of this phenomenon is used as a novel way to address a pressing issue on the continent.
Please describe pressingly relevant issue in European environmental policy that is mitigated by the presence of U) highlighting the subtle yet significant impact it has.
ImageNet-R:
A unique fusion of traditional craftsmanship and modern style, featuring {). A unique, imaginative representation ()-
A beautifully crafted, imaginative representation of a 0.
ImageNet-A:
32

---

Published in Transactions on Machine Learning Research (08/2025)
are helpful assistant is going help with finding the prompt templates to embed class names for my dataset for zero-shot classification with CLIP. Let's Go!
System Prompt
Task Description
You provided with dataset name. deseription, top {top bottom k) and worst {top bottom kj example prompt templates their associated accuracies from the {step+ 1} runs. Your task to provide me with new prompt templates the same format as the given prompts, so that can simply replace the <category- > placeholders with the actual class names in the dataset and use it for zero-shot classification with CLIP. The goal to get increase accuracy by using the newly generated prompts. the dataset description and the best and worst example prompts context for improving accuracy. Be creative! Good luck!"
You
Dutaset: [dataset name] Description: (dataset info] Best Templates: [best_exm] Worst Templates:
(worst exm}
In-context Examples
### Remember to only provide me the prompt as response, and nothing else. ####
Figure Overview of the LLM Prompt. The system prompt a generic instruction set. task description instructs the LLM about the desired (ask and has dynamically evolving fields that are updated according to the optimization evolution. Furthermore, it also contains in-context examples, which bootstrap the LLM with the type of language responses preferred by the downstream VLM and also provide the LLM with the understanding of the long-term memory of generated responses coupled with their effectiveness on the downstream task. Note that this prompt is used by our GLOVV for CLIP. For the enhancing safety of VLMs, the objective is different, i.c., to reduce the ASR instead of increasing the accuracy.
A photo of a situation where the unexpected coexistence of a 0 with scemingly unrelated elements creates sense of tension or unease, making it difficult to accurately classify, our brains struggle to reconcile the familiar with the unfamiliar.
A photo where the presence or absence of a { subtly alters the viewer's emotional response, making it more muancdd and open interpretation, roquiring a more thoughtful approach classification.
A photo of situation where the presence of () creates air of familiarity, but its absence unxxpectd absence sparks a deeper investigation to accurately understand the context and reclassify the scene.
UCF-101:
A person skillfully performs O action that requires a combination of physical and mental effort, often in a context of human-object interaction.
The O action is remarkable display of human dexterity and coordination, requiring deep undorstanding of spatial awareness and precise motor control.
The () is a complex action that involves coordinating miltiple body parts to achieve a specific outcome, often requiring precision, agility, and strength.
33

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm GLOV: Guided Optimization of Prompts
Input: Pre-trained LLM f with parameters 0, simple prompt template Ps: scaling factor a, maximum number of tokens Nmax: number of prompts per iteration K 10, Meta-prompt, Few-shot training set C, Fitness function F(-CC, target layer index I, Array A.
2: Output: Optimized prompts Popt.
3: Evaluate Ps on the few-shot train set with F(P,.C) and record the accuracy Cr.-
4: Generate K prompts P= Pa..((P
for P. P do
Ali A[(==F(P,CC)
end for
8: To argmax p. A
9: Po Ph++[[]]
10: A INF
11: In aargmax,,
12: P P[Tu]]
13: while converged do
14: Obtain H₃ and Hw through equation 3
15: NewPrompls <- (i)
16:
for k in {1...0}} do
17:
Tokens +- List()
18:
for each new token n= Lm.,,.,,Naax do
19:
H,=H, +a- H
20:
Tokens.add fakkessadddf.deees (eeeesseddddfeccoeeeeee (decode(HH)) end for
21:
22:
NewPrompts, wampptssaapenn((oeeennn end for for Promllss do F(P,=F(P,,)) end for argmax, New Prompts A
23:
24:
25:
26:
27:
28:
P. New Promple [T)
29:
A[1) f- I-NF
30:
IN + argmax New Prommpptss Prompts A
31:
t NewPromptss(u..
32:
end while
33: Return: P3 as Optimized prompts Pope
34

---

Published in Transactions on Machine Learning Research (08/2025)
00
45
56
54
GLOV aulkaan)
03
18
CLUI
30
081
-3.
05
DV Delo gollbrnn))
GLAVV (e/n gridonns)
GSEDN
20
02
102
00
CLIP
GLOV Cefo nickan))
CLOV
CLIP
GLOV (n/o pellued
GLOV
StanderrCam
18
28
24-
02
CLIP
GLOV Lnfo aickan))
CLOV
CLLI
CLOV EATO AORRARO
CLOV
00
Towers
65
02
02
C1.I1
GL 06/5 (3l0e
IILOON
01
Figure 6: The effect of prompt evolution the downstream task performance. The shaded regions represent the absolute top-1 accuracies at each optimization step by ensembling the top-3 prompts found w.r.t the accuracy on the 1-shot train set whereas the solid lines represent the exponential moving average. The VLM employed CLIP VTT VTT/B-32 (Radford et al., 2021) and the LLM Llama-3 (Dubey et al., 2024).
35

---

Published in Transactions on Machine Learning Research (08/2025)
GLOV
GLOV (w/o guidance)
Step:
conggory this image trisfly iientty the namt the category preserz.
calegory image briety also idently the nome of calegory present.
Accuracy:
Accuracy:
Can you dountie the msin wijject abject this inage, highlighting didinctive oommon romn, comsikkrnng contest appears and providing the class zame of the conegeey belongs
typial attibutes,
Step:
Spoc central figure lateling with precise class nome and concuety sunmarizing the secue coutcat. lighlighiss, any striking feaures or putterss that set from others?
image,
Accuracy: 49.5
Accuracy: 47.4
Can you dounite the main whject object this inage, highlightng disinctive Statures, opical amnitrios, common name, explain is troader category including any relevant sporal or icaporil
Identify the main subject prominers visual aspezt of this providing concise description taat hightight any distnxtive viuu textanos, orriesaal dktvils contributc clasifaatius,
image, specitying
Step:
listecical
cnmilering image's wame and campoctton.
typical battuuss or envinormeuts, while also procating the clas rame of the calzgory belongs so?
Accuracy: 51.8
Accuracy: 48.5
this hightighring moe distinzion fastores, rypical smmibutes, and common zame, and explain how relates to
deseribe
manif inage, spccifying the precise cles rame and offiring a
Pirpoint coneral figune, dbject,
cuncier explarasion the visul fastanos, rextanos, acoouming for the ingee's owrall scmee, compostion, and peternal contesmual
assccaaionn with spocial and temporal consest. cutural and hisorical signifiennee, and is cles while also comiderin iss mactuefiiaatiss
is
Step:
poneens justry
Accuracy: 53.0
Accuracy: 47.2
you douniis sibject object this inage, highlighiigg is mos disinative visua features opiccal smitinss, and commun cotegory incnng eveturion through axpioring its rohmnl mnd hisorial significncee and highllghting relationhigs other objects within cateuury, also providin munoed cotegooization and
explain robstos
Hightight the image's 's's most notable fectures, specitying the class and deaciling chasefiorim, while ammidering image's eveall cuntest and disinaiie visul patienu make unique
Step: 19
Accuracy: 53.1
Accuracy: 46.9
Car you dosnile the uais subject object this inage, hightighring distinatio vieusl festonos, typical arritunes, is becaler category by insinu its evoluion thoouh tine, culuna hisorical significazee. hightighting relationdiiss cvegrry. while also amphasiring the whitle monoos and puculisrtiks
Unlock dee
pisoral
explain
festors, textun, ortecnal uniisniaaly links is precise chas wlile also conedoring
objocts
Step:
contrbute its unique adenity.
Accuracy: 54.5
Accuracy: 46.8
Figure 7: Prompt evolution for LLaVA. We provide the highest performing prompt (on the 1-shot train set) discovered by our GLOV at difforent optimization steps for the ImageNet datasct.
36

---

Published in Transactions on Machine Learning Research (08/2025)
GLOV
GLOV (w/o guidance)
Step:
plotoo
Accuracy: 63.8
Accuracy: 63.8
high quality image an f leaturing a charasteristi; or belarice that typical for ia känd.
Step:
high quality amgee an IJ featuring a characoerstt or behavii lles tppiall foe in kind.
Accuracy:
Accuracy:
highoqulliy image If dommmtniinn is uniuu reluinnship with envinament other objecs, highlighring sdeptahility deporince sirroundings.
high-qualkt image !] with unsead distactive componition
Step: 20
unggee
Accuracy: 63.0
Accuracy: 61.9
its
Step: 30
planoggappl of ( slarecasing structuaal morphstogticll feutures, highlighting clecasterssiie.
finction, er caltural signiticanee.
Accuracy: 63.4
Accuracy: 62.0
high-qullit image or interaction that is distnctive of clas, and is oftn usod visual repecsentation of its Accuracy: 63,2
umiqe pose, exprosien,
Step: 40
highhquiiit phonograph foemning
most strkiin notatle aspects. and providing clear view of is essential features.
Accuracy: 62.2
A ligisqualit imaae of 1) is feenie reprosntativv symbol point fte type, showeasing its unique characteristles and features
Step:
high pareanedtty or defining attributes, inviinn viewer explore its inmricacies and monces.
embtodis
human Accuracy:
Accuracy:
photo ladjeztive, ea. poomisentty. typically or isonicalty displaying its unique (featore shamstrrati
Step: 60
!] ia situatio tha requiess is usique Seatures or abilitics, uach dlcmmmtratingg is funciionilly highlighring advanigees.
behuvior, exiil, aqaina
Accuracy: 64.8
Accuracy: 62.3
A photo af dispaaiig in unique visaal alently, such which aport types (aljestive, c.g. stiling, characierieikc or
jusstizctive
Step: 70
cdhibiing it typical belarice inseractio wrring. sllining appreciate essence and unique chamstertstis.
Accuracy: 65.0
Accuracy: 63.4
babini, unique characteristis, Idistixtive feanues belorisee)) dhe are ofenn sosxiated with is type and are Inflective,
phongraphi
natrral latiiat
Step:
charsteristics effortlssty interacion: will it envinnmenn. highlightig is distectiveuess.
Accuracy:
Accuracy:
A plato (} foataring its (distinctive fistur: (eeting comentj is Indjective, c.z. viteazi, striking, or stiking). which
Step: 90
!) retunal babist sctling, showezoing its
revealing unique
adaptations to as ecological nxhe.
Accuracy: 65.1
Accuracy: 62.9
A cheae-pp or belaviors] that often associased with anida waidl [lackgmuul eninnmmann]
striing fdisiaative
Step:
eurefully composed socne. auention drown to most striking defining features, lighigghin ilantly calogoty.
Accuracy:
Accuracy:
Figure 8: Prompt evolution for CLIP, We provide the highest performing prompt (on the 1-shot train set) discovered by our GLOV at difforent optimization steps for the ImageNet datasct.
37


Reviews:
[{"review_id": "dFykYWluGu", "reviewer": "Reviewer_oyCj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The proposed idea is interesting.\n2. The paper is overall well presented.\n3. The codebase could be useful for future research if released.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes GLOV (Guided Large Language Models as Implicit Optimizers), a method that uses Large Language Models (LLMs) to iteratively discover and refine natural language prompts for Vision-Language Models (VLMs), aiming to enhance performance on downstream vision tasks like object recognition and VLM safety. GLOV prompts an LLM with a task description and ranked in-context examples of previously generated VLM prompts (and their accuracies on a small training set). A key novelty is the explicit guidance of the LLM's generation process at each step by adding an offset vector—derived from embedding differences between previous effective (positive) and ineffective (negative) prompts—to an intermediate layer of the LLM. This biases the LLM towards generating language preferred by the target VLM. The authors evaluate GLOV on object recognition across 16 datasets using both dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVA) VLMs, and on VLM safety benchmarks, reporting significant performance improvements and reductions in attack success rates.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The LLM prompt includes system instructions, task descriptions, and ranked in-context examples from a \"global history.\" Managing this global history and ensuring the LLM prompt doesn't become overly long or difficult for the LLM to process effectively, especially over many iterations, is challenging. It is important to provide more details about this part in the main paper and explicit show the trade-off between the baselines and the proposed method in terms of accuracy and efficiency.\n\n2. Since the proposed method is claimed as an optimizer, the reviewer expects to see more fine-tuned baselines using either gradient descent or existing approaches of gradient free optimization methods. These are essential baselines to provide.\n\n3. The current used VLMs are not strong enough. If they are already based on strong enough LLMs, would the proposed method still help?", "broader_impact_concerns": "Can this method possibly introduce certain bias in the prompt optimization process? We can clearly see a lot of vision-irrelevant words in the resulted prompt.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R26lT34uTL", "reviewer": "Reviewer_zCnW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Innovation: framing prompt search as an agentic feedback loop between an LLM and a VLM, with an efficient embedding-space guidance mechanism.  LLM-VLM interaction is interesting.\n- Generality: applies to both contrastive dual-encoder and generative encoder-decoder VLMs and supports tasks from classification to safety.  \n- Evaluation is conduct on varied benchmarks to show the generalization ability.\n\nWeaknesses:\n1. Revise the related work section such as LLMs and VLMs, is quite boarder. One close area is prompt tuning for LLM and VLM, and author should discuss these works.\n2. The Fig. 2 is hard to understand, should be refined and polished and make it clearly. Such as each letter denote what?\n3. Why evaluate on Object recognition and VLM safety; Obj. recognition should be Image Recognition?\n4. Robustness: The experiment benchmark is toy, it is not clear whether the reported results come from a single seed or are averaged over multiple runs with measures of spread (e.g., mean and variance).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces GLOV, a novel framework that leverages large language models (LLMs) as implicit optimizers to discover high-quality natural-language prompts for vision-language models (VLMs). At each iteration, GLOV queries an LLM (e.g., Llama-3) with in-context examples ranked by a downstream fitness function (e.g., zero-shot accuracy from CLIP or attack success rate for safety), and further steers generation by adding an offset vector to the LLM’s hidden state, where \\(H^+\\) and \\(H^-\\) are sentence embeddings of the best and second-best prompts. Without any gradient fine-tuning, GLOV achieves up to 15.0 % relative gain in classification accuracy for dual-encoder VLMs and 57.5 % gain for encoder-decoder models, and reduces attack success rates by up to 60.7 % on safety benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Typo: Fig.1 LlaVa One Vision -> LLaVA-One-Vision、\n- Provide a detailed cost analysis (GPU hours) and discuss runtime trade-offs to clarify practical feasibility.  \n- Include experiments in a fully zero-shot setting or discuss strategies to mitigate reliance on labeled data (would strengthen).", "broader_impact_concerns": "While GLOV can improve VLM performance and safety, it may amplify biases present in the LLM’s training data when generating prompts, leading to skewed downstream behavior.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fxeYo9ycNQ", "reviewer": "Reviewer_UU4r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The idea of gradient-free LLM-guided optimization is interesting as it does not involve changing the base model, but rather automatic steering through examples at the input, which act as long-term memory (as if we use a huge batch size to update the paramters, but in a far more efficient way). This is more effective than soft prompt-tuning methods (e.g., COop) because the learned prompts are interpretable, rather than being dense uninterpretable vectors. Although this is not novel (previous works in Section 2.3 do something similar), I find the two aspects of (1) using the long-term history-buffer for guidance, and (2) showing the impressive performance of LLava-like models (which benefit hugely from prompt optimization) more important than the novelty aspect. Specifically, this work shows that the hand-crafted prompts used in [R1] (which [R1] claims to have no effect on the downstream task performance) are the actual problem, and by optimizing them we can significantly improve downstream task performance. \n- Huge suit of datasets evaluated on, and impressive performance on autoregressive VLMs and safety VLM benchmarks. Optimizing for safety instructions is something I find very important in current large models. \n- The optimization of general dataset-specific prompts rather than class-specific prompts as in previous works, is interesting. \n\nWeaknesses:\n- [W1] For steering the LLM with the offset vector, this will be limited to open-source models (which are still way behind OpenAI models and closed-source models). The method without steering would be more desirable because it could be applied to any LLM (even closed-source ones). Have the authors tried using closed-source models without offset vector steering, rather than low-performing Llama models? Furthermore, the idea of activation-space steering (commonly referred to as control/function vectors) is proposed in [R2], which I think deserves a citation. \n- [W2] The performance on CLIP-like models on ImageNet is marginally better, and worse without activation guiding. I think this method is not effective for dual-encoder models.  This is not a weakness, but rather a feature, showing that we have reached a saturation in prompt-tuning methods for CLIP-based models. Can the authors justify why that might be the case? In my opinion, encoder-based models only contextually embed the prompt, but many fine-grained details in the prompt (e.g., negation) are lost, which might be the reason. \n- [W3] The authors do not report the latency/speed. How long does it need to optimize the prompts? I find this important, especially for LLava-like models. \n- [W4] Im not sure why the authors generate a description (along with the category) in LLava-like models and then measure the similarity of this generation with the ground-truth class with a text embedding model, as a way to measure accuracy. Why not prompt Llava to classify the image with a single or two-word response (given a possible pool of classes, like in [R1])? This is possible in all Llava models. Can the authors clarify their way of doing this? \n\nMinor Weaknesses:\n- In the example prompts in Figure 2, these are supposed to be general, for all classes. But here they seem to be for the specific class (cat).\n- In Eq. 2, Shouldnt there also be a summation over all classes? l_c(x) should be the probability over all classes, not for a single class, but here it seems to be a scalar for a single class. \n\n[R1] Why are Visually-Grounded Language Models Bad at Image Classification?, NeurIPS 2024\\\n[R2] Function Vectors in Large Language Models, ICLR 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a gradient-free prompt-tuning method for vision-language models (CLIP-based and autoregressive-based) using an LLM to propose example prompts, with a reward (or scoring) signal coming from the VLM. Specifically, the VLM is being steered by example prompts coming from an LLM (which the authors consider it to be a human-like model). The LLM proposes candidate prompts, which are then ranked by the VLM through a fitness function (here, accuracy) calculated on a small held-out 1-shot test set. The ranked prompts (along with the best-scoring and worst-scoring ones from previous optimization steps) are given to the LLM as form of in-context guidance examples and the LLM is steered to output prompts such that the VLM performs well on the given task. The authors also propose to guide the LLM through control vectors (calculated from the difference between positive and negative prompts) to further steer the LLM. Experiments are conducted on a wide set of classification dataset and VLM safety benchmarks and show promising results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general I find this paper interesting and useful for the community. For me, the impressive improvements of LLava-like models and safety benchamrks, with an input-example-based and gradient-free optimization technique is enough. The authors can focus on addressing W1, W2, W3, and W4. The minor weaknesses can be addressed directly in the final manuscript.", "broader_impact_concerns": "No issues", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)