Title: IPO: Interpretable Prompt Optimization for Vision-Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Prompt learning

Large language model

Interpretable Prompt Optimization
Abstract: Pre-trained vision-language models like CLIP have remarkably adapted to various downstream tasks. Nonetheless, their performance heavily depends on the specificity of the input text prompts, which requires skillful prompt template engineering. Instead, current approaches to prompt optimization learn the prompts through gradient descent, where the prompts are treated as adjustable parameters. However, these methods tend to lead to overfitting of the base classes seen during training and produce prompts that are no longer understandable by humans. This paper introduces a simple but interpretable prompt optimizer (IPO), that utilizes large language models (LLMs) to generate textual prompts dynamically. We introduce a Prompt Optimization Prompt that not only guides LLMs in creating effective prompts but also stores past prompts with their performance metrics, providing rich in-context information. Additionally, we incorporate a large multimodal model (LMM) to condition on visual content by generating image descriptions, which enhance the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance, while maintaining human comprehension. Extensive testing across 11 datasets reveals that IPO not only improves the accuracy of existing gradient-descent-based prompt learning methods but also considerably enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, our approach ensures that the prompts remain human-understandable, thereby facilitating better transparency and oversight for vision-language models.

IPO: Interpretable Prompt Optimization for Vision-Language Models

Abstract

Pre-trained vision-language models like CLIP have remarkably adapted to various downstream tasks. Nonetheless, their performance heavily depends on the specificity of the input text prompts, which requires skillful prompt template engineering. Instead, current approaches to prompt optimization learn the prompts through gradient descent, where the prompts are treated as adjustable parameters. However, these methods tend to lead to overfitting of the base classes seen during training and produce prompts that are no longer understandable by humans. This paper introduces a simple but interpretable prompt optimizer (IPO), that utilizes large language models (LLMs) to generate textual prompts dynamically. We introduce a Prompt Optimization Prompt that not only guides LLMs in creating effective prompts but also stores past prompts with their performance metrics, providing rich in-context information. Additionally, we incorporate a large multimodal model (LMM) to condition on visual content by generating image descriptions, which enhance the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance, while maintaining human comprehension. Extensive testing across 11 datasets reveals that IPO not only improves the accuracy of existing gradient-descent-based prompt learning methods but also considerably enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, our approach ensures that the prompts remain human-understandable, thereby facilitating better transparency and oversight for vision-language models.

1 Introduction

Vision-language models, trained on a diverse array of image-text pairs encapsulating a broad vocabulary of real-world concepts [1, 2, 3], have demonstrated notable adaptability across various downstream tasks [4, 5, 6, 7]. These models perform zero-shot image classification by filling in a predefined prompt template (e.g., ' a photo of a [CLASS] ') with specific class names for the text encoder. Despite their effective generalization to new tasks, the performance can be influenced by minor changes in the wording of prompt templates [8]. Instead of manually creating handcrafted prompts, recent developments in natural language processing [9, 10] and computer vision [8, 11, 12, 13] have proposed methods to learn a set of soft prompts with minimal labeled data. Despite the strides made in learning prompts, the current state of the art remains limited by its lack of interpretability and the overfitting problems on the base classes, which can be prohibitive in diverse and dynamic application environments. These limitations underscore the need for a more adaptable and user-friendly approach to prompt optimization in vision-language models.

Drawing from recent advancements in using large language models (LLMs) as optimization tools [14], our paper, for the first time, incorporates these capabilities into vision-language modeling. Unlike gradient descent-based methods [8, 11, 13], which often fail to provide explanations for the generated

Task introduction

+

Image description

+

Prompt history

CLIP

Loss

Take a breathtaking and

flawlessly detailed

photo of a {CLASS}.

Optimized prompt

CLIP

Loss: 0.11

Acc: 0.98

Scores

Top-k prompts

LMM

Image description

Default prompt

A photo of

a <CLASS>.

CLIP

Loss: 7.95

Acc: 0.65

Scores

Initial prompt

Optimized prompt

<CLASS>.

<CLASS>.

CLIP

Gradient descent optimizer

LLM

Scores

Interptretable prompt optimizer

Loss: 0.25

Acc: 0.75

Scores

Prompt Optimization Prompt

(a) Gradient-based prompt optimization.

(b) Interpretable prompt optimization.

prompts and tend to overfit on base classes, natural language-based methods enable LLMs to develop and refine solutions through continuous feedback iteratively. This approach improves interpretability in complex tasks like prompt optimization for vision-language models, making it easier for humans to understand the generated prompts. However, existing research on these methods [14, 15, 16] primarily addresses language tasks and has not yet explored their potential for integrating prompt optimization with an LLM in vision-language models.

To address these challenges, this paper proposes an interpretable prompt optimizer (IPO) for visionlanguage models that leverages the capabilities of LLMs to generate and refine text prompts dynamically. First, we design a Prompt Optimization Prompt to prompt LLMs to generate more effective prompts that improve the accuracy of CLIP and reduce the loss in base classes. Our Prompt Optimization Prompt also stores past prompts along with their corresponding accuracy and loss as episodic memory, thereby providing richer in-context information to enable LLMs to generate more effective prompts. Second, to incorporate image information within the Prompt Optimization Prompt, we propose using a large multimodal model (LMM) to generate descriptions of images in base classes that can be added to the Prompt Optimization Prompt. This integration facilitates a more intuitive interaction between the textual and visual modalities, which allows the Prompt Optimization Prompt to utilize image information, thereby generating datasetspecific prompts to enhance the generalization performance of CLIP. The framework of our IPO, illustrated in Figure 1, showcases the comparison between traditional gradient-based prompt optimization and the proposed interpretable prompt optimization. Third, the prompts generated by our optimizer are human-interpretable. For example, on the Food101 dataset [17], the initial prompt evolves from [' a photo of a [CLASS] '] to [' Categorize the image depicting a delicious and appetizing <CLASS> with remarkable visual qualities. ']. Our generated prompts perform 10.29% better in novel classes than the gradient-based method CoOP [8], reducing overfitting while maintaining interpretability.

We validated our IPO across 11 different datasets, demonstrating that it surpasses traditional gradientbased state-of-the-art methods in accuracy and excels in interpretability. Our approach generates human-comprehensible prompts that can be seamlessly integrated into existing vision-language models to enhance performance. We conducted rigorous comparative experiments to quantify the interpretability between gradient-based prompt learning and our method. We demonstrated the importance of specific keywords in our generated prompts and revealed that not all tokens learned through traditional prompt learning methods are essential.

2 Related work

Prompt learning in vision-language models. Prompt learning, originally introduced in the natural language processing community [18, 19, 20], involves applying a fixed function to input tokens to provide task instructions to the model. In the computer vision community, prompt learning has been explored in various forms, including textual prompt tuning [8, 11, 21, 22, 23, 24], and prefix tuning [13, 12, 25, 26, 27, 28, 29]. 1) Prompt tuning mainly involves treating text prompts as learnable parameters, using a small amount of data to fine-tune these parameters. As pioneered by CoOp [8]

and CoCoOp [11], which both fine-tune a CLIP vision-language model [30] for few-shot transfer by optimizing a continuous set of prompt vectors within its language branch. Bayesian prompt learning [21] formulated prompt learning as a variational inference problem and demonstrated its ability for unseen class generalization. 2) Prefix tuning primarily involves adding learnable tokens to the text encoder [31], vision encoder [12, 25], or both encoders [13, 27, 28, 32]. These tokens are fine-tuned using a small amount of data. Note that these methods do not optimize the initial text prompts. Instead, they focus on enhancing the model's understanding capabilities by integrating these additional, trainable tokens. Our method belongs to prompt tuning, but unlike previous approaches that use gradient descent to optimize prompts, we propose using LLMs to optimize prompts. Our method leverages the natural language capabilities of LLMs to iteratively refine feedback-based prompts, aiming to enhance both the effectiveness and the explainability of the prompts.

LLMs as prompt optimizers. Several recent works explore the role of LLMs as prompt optimizers for NLP tasks. Some use LLMs to directly optimize the task instruction for in-context learning [33, 34, 14]. Other studies use LLMs to mutate prompts for evolutionary algorithms [35, 36]. However, to the best of our knowledge, no existing studies have investigated how LLMs could be used to optimize text prompts within vision-language models. This approach could potentially open up new avenues for integrating and enhancing the capabilities of vision-language models through more effective and contextually appropriate text prompts.

Meta-prompting. Suzgun and Kalai [37] introduce meta-prompting to transform a single LLM into a versatile 'conductor' capable of managing and integrating multiple independent LLM queries. By using high-level instructions, meta-prompting guides the LLM in decomposing complex tasks into smaller subtasks. The core of OPRO [14] involves designing a meta-prompt for LLMs to optimize prompts for each task. This meta-prompt includes two key pieces of information: previously generated prompts with their corresponding training accuracies and a description of the optimization problem. Self-select [38] leverages meta-prompting to optimize instruction selection. It considers a set of provided templates and chooses the most suitable template. Meta-prompting is related to instruction tuning [39] as both techniques provide high-level guidance to improve the performance and adaptability of LLMs. However, while instruction tuning focuses on fine-tuning models with a variety of tasks to improve generalization, meta-prompting offers the advantage of dynamically guiding the model to decompose and manage complex tasks in real-time. Liu et al. [40] proposes a method that utilizes LLMs as black-box optimizers for vision-language models, iteratively refining prompts based on in-context examples. Their approach focuses on leveraging ChatGPT to improve prompt templates for visual classification tasks. Mirza et al. [41] explores a different aspect of prompt optimization by focusing on zero-shot vision-language models. Our Prompt Optimization Prompt is akin to meta-prompting, it stores past prompts along with their corresponding accuracy and loss, thereby providing richer in-context information to enable LLMs to generate more effective prompts. Different from prior meta-prompting, our Prompt Optimization Prompt generates prompts beyond LLMs for vision-language models.

3 Preliminaries

Contrastive Language-Image Pre-Training (CLIP). The goal of CLIP [30] is to develop an image encoder f I and a text encoder g T via contrastive pre-training with a large collection of paired images and captions. This process aims to map image-text pairs to a common semantic space. After the pre-training phase, CLIP is able to perform zero-shot visual recognition by treating classification as a task of matching images to text. Specifically, the placeholder term ' [CLASS] ' is used within a prompt template (e.g., ' a photo of a [CLASS] ') for the text encoder g T . Here, g T ( T i ) denotes the text features adapted for class i , and the probability of classifying class i from an image I is:

where ⟨ g T ( T i ) , f I ( I ) ⟩ represents the cosine similarity between the image feature f I ( I ) and the text feature g T ( T i ) specific to the i -th class, K is the total number of classes, and τ is the temperature parameter that is tuned during training.

Prompt learning improves the adaptability of the CLIP model by eliminating the need for manual prompt engineering. It facilitates the automatic generation of prompts using a limited number of examples from a downstream task. CoOp [8] presents a method where a set of M continu-

ous context vectors V = { v 1 , v 2 , . . . , v M } serve as the learnable prompt. The constructed prompt T i = { v 1 , v 2 , . . . , v M , c i } merges these learnable context vectors V with the class-specific token embedding c i , which is then processed by the text encoder g T ( · ) . In CoOp, the optimization of these static context vectors V aims to minimize the negative log-likelihood for the correct class token:

here, y i represents the one-hot encoded ground-truth label for class i . In downstream applications, the pre-trained model parameters are kept unchanged, which allows the learnable prompt vectors V to be optimized efficiently using only a small number of samples through the minimization of the cross-entropy loss.

4 Methods

Figure 1 depicts the comprehensive structure of our interpretable prompt optimizer. At each step of the optimization, the LLM generates candidate prompts for the vision-language task by considering both the description of the optimization problem and the feedback from previously evaluated prompts stored in the Prompt Optimization Prompt. These new prompts are then assessed and incorporated into the Prompt Optimization Prompt for future optimization cycles. The optimization process concludes either when the LLM can no longer generate prompts that improve the optimization scores, or when a predefined maximum number of optimization steps is reached. Next, we will detail the design of the Prompt Optimization Prompt and explain how image information is integrated into the Prompt Optimization Prompt.

Prompt Optimization Prompt design At the core of our optimizer is the design of the Prompt Optimization Prompt, which enhances the performance of the vision-language model by optimizing the prompts through the prompt LLM. Figure 2 shows an example of our Prompt Optimization Prompt. Our Prompt Optimization Prompt consists of the following components: (1) Instructions: These guide the LLM by clearly defining its task to optimize the prompt for achieving better performance in classification tasks. (2) Textual descriptions of training images: These descriptions provide the LLM with detailed information about the images, enabling it to generate dataset-specific prompts. (3) Previously generated prompts and corresponding scores: This component supplies in-context information, including past prompts and their performance metrics, allowing the LLM to refine its prompt generation more accurately. By incorporating these elements, our approach leverages the iterative refinement capabilities of LLMs to dynamically generate and optimize text prompts. The instructions ensure that the LLM understands the optimization goal, the textual descriptions offer rich image-related context, and the historical data aids in producing more effective and precise prompts.

Textual descriptions of training images For the textual descriptions of training images, we utilize a large multimodal model (LMM) to generate text descriptions for each training image. Specifically, we employ MiniCPM-V-2.0 [43] to generate descriptions of the content of images from base classes. In the appendix, we provide content descriptions for some images from each dataset generated using MiniCPM-V-2.0. We denote the extracted image textual features as f M ( · ) .

Additionally, we have attempted to directly optimize prompts using the LMM with the Prompt Optimization Prompt. Specifically, we input images from the base classes and the Prompt Optimization Prompt into the LMM, aiming for the LMM to generate better prompts. We experimented with six different LMMs: BLIP-2 [44], Qwen-VL-Chat-9.6B [45], FUYU-8B [46], MiniCPM-V-2.0 [43], and llava-llama3-8B [47]. Unfortunately, all six models failed to understand our Prompt Optimization Prompt and generated new prompts that were merely descriptions of the images, not the universal prompts we desired. This failure might be due to the fact that the training of these LMMs did not consider such a task. Note that image descriptions are not mandatory. In our 16-shot experiments, we omitted the image descriptions in the Prompt Optimization Prompt due to the limited text input length that the LLM can handle.

Episodic memory retrieval We utilize an episodic memory mechanism to retrieve past prompts and their corresponding scores, which include metrics such as loss and accuracy. Here, we denote the memory as M . During each iteration, we retrieve the top-20 prompts R ( M ) , based on their accuracy from M and use them as the current memory, denoted as m . Moreover, we consistently include the prompt ' a photo of <CLASS> ' in our history at every step, as this is a frequently used

Instruction 1

You need to perform image classification on the large-scale visual recognition dataset based on visual features.Here, <CLASS> represents

a class name from the large-scale visual recognition dataset, and it is essential to include <CLASS> in <INS>.

Below is the image description for each image:

Image descriptions

Image 0: The image showcases a close-up view of the texture on an object with distinct stripes, buttons and stitching details.

Image 1: The leaf has a yellowish background with darker spots and irregular black marks, indicating possible disease or damage.

(... more image examples. . . )

Prompt history

text prompt:

a photo of a <CLASS>.

scores:

loss: 1.6435546875, accuracy: 54.166664123535156

text prompt:

Identify the intricate details and unique patterns of the <CLASS> texture.

scores:

loss: 1.3935546875, accuracy: 70.83332824707031

(... more previous prompts and scores . . . )

Instruction 2

Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0,

while the accuracy varies from 0 to 100.

Instruction 3

Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The

prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above.

Below are the prompts created according to your guidelines:

(1) <INS> Analyze the intricate texture of <CLASS> with precision and attention to detail. </INS>

(2) <INS> Thoroughly analyze and classify the intricate <CLASS> texture with meticulous precision. </INS>

(3) <INS> Analyze, categorize, and classify the intricate <CLASS> texture with unparalleled precision and exceptional attention to detail. </INS>

(... more output prompts examples. . . )

and effective prompt within the CLIP framework [30]. Therefore, our optimization loss is defined as:

where I indicates the our designed instruction for LLM, ˆ T represents the new human-interpretable text prompt optimized by the LLM. Note that our optimizer is parameter-free, which differentiates it from traditional gradient-based prompt learning methods. Instead, we leverage the LLM to optimize the prompt, reducing L CE iteratively until convergence.

The input and output example in Figure 2 shows the structured information fed into the LLM, while output demonstrates the optimized prompts generated by the LLM. For more detailed examples of Prompt Optimization Prompt input and output, please refer to the appendix.

5 Experiments

5.1 Experimental setup

We validate the effectiveness of our approach on the base-to-new generalization benchmark for evaluating prompt learning in vision-language models [8, 11]. Across all experiments, we benchmark the models' performance in a 1-shot and commonly used 16-shot setting. To ensure consistency, all results from learning-based methods are averaged over three random seeds. We use the harmonic mean (H) as the average metric, which is a common approach in prompt learning for vision-language models.

Eleven Datasets. We follow CLIP [30] and CoOp [8] to use 11 image classification datasets, i.e. , ImageNet [48] and Caltech101 [49] for generic object classification, OxfordPets [50], StanfordCars [51], Flowers102 [52], Food101 [17] and FGVCAircraft [53] for fine-grained image recognition, EuroSAT [54] for satellite image classification, UCF101 [55] for action classification, DTD [42] for texture classification, and SUN397 [56] for scene recognition.

Six Baselines. To conduct a comparative evaluation, we utilize a number of established baselines including CLIP [30], Coop [8], CoCoOp [11], MaPLe [13], PromptSRC [28], and CoPrompt [32]. Note that all methods do not present 1-shot results in their publications, so we perform 1-shot experiments using their available code.

Training details. We use GPT-3.5 Turbo as our default optimizer, iterating 100 steps for each dataset to derive the final prompt. At each step, we generate five prompts and compare their accuracy with past prompts, storing the top-20 prompts in our history. Ultimately, we select the prompt with the highest accuracy as the final prompt. For generating image descriptions, we employ MiniCPM-V-2.0 [43] as the default LMM, using the prompt: '[ Please generate the description in detail, do not provide the class name in the description. ]'. We added image descriptions to the 1-shot Prompt Optimization Prompt but not to the 16-shot version due to the character input limitations of GPT-3.5 Turbo, which prevent adding detailed information for each class's images. All experiments were conducted on a GeForce RTX 3090. Our code is available at https: //github.com/lmsdss/IPO .

5.2 Results

Interpretable prompt analysis. To analyze the importance of specific words or phrases in text prompts, we display a comparative experiment on the Flower102 dataset. Specifically, in Table 1a, we use ' a photo of a <CLASS> ' as the prompt, which is the most commonly utilized format. By employing occlusion sensitivity analysis, we individually remove each word to test the importance of the remaining words. We find that using just ' <CLASS> ' as the prompt performs only 0.89% lower on the novel classes than ' a photo of a <CLASS> ', indicating that the CLIP model can generate more discriminative features using just the category name. Additionally, the prompt ' a photo <CLASS>. ' achieves the best performance. By comparing ' <CLASS> ' with ' photo <CLASS> ', we determine that the word photo is particularly significant on the novel classes.

In Table 1b, we present the results of the CoOP and CoCoOP models when some learned prompt tokens are removed to analyze which token is most crucial. Surprisingly, for both models, performance improves in novel classes when some or all tokens are removed. This indicates severe overfitting in base classes by these models, where the learned tokens are only applicable to base classes. Removing all tokens allows the models to retain some of the original performance of CLIP on novel classes. Additionally, these methods, which only learn tokens, make it challenging for humans to understand the specific meanings of each token, complicating interpretation.

In Table 1c, we also demonstrate the final prompt produced by our model: 'Identify the unique visual features of the <CLASS> flower accurately,' which achieves a performance of 79.6%, surpassing the original CLIP by 2.9% on novel classes. When comparing this optimal prompt with others, removing any words from it results in worse performance than the original prompt. Specifically, comparing ' Identify <CLASS> ' with ' <CLASS> ' reveals that including " Identify " boosts performance by 1.7% on novel classes. This highlights the importance of the word ' Identify ' in datasets like Flower102. We have included additional analyses on various datasets in the appendix.

Overall, the comparative experiments demonstrate that our prompt can be more easily interpreted and understood by humans, while also providing insights into the significance of certain key words

(a)

CLIP

(b)

CoOP and CoCoOP.

(c) This paper : IPO

(a)

Impact of large language model.

(b)

Impact of large multimodal model.

in the prompt. This understanding can guide us in identifying crucial words that enhance prompt effectiveness.

Effect of LLM and LMM choice. In Table 2, we analyze the effect of the choice of the LLM optimizer and LMM for generating image descriptions as Prompt Optimization Prompt inputs. We evaluate various models and report their average performance across 11 datasets. Using GPT-3.5turbo as the optimizer results in better prompt generation, especially improving performance on novel classes. The training speed of our IPO is heavily influenced by the computational efficiency of LLMs used. Since PaLM and GPT-3.5 are not open-source, we rely on their respective APIs to generate prompts. Consequently, our training speed depends on the API call latency and the computational complexity of these models. Alternatively, Phi2 and Phi3 are open-source, allowing us to generate prompts directly using their weights. Therefore, for researchers and practitioners seeking faster and more cost-effective training, we recommend utilizing open-source large models for prompt generation. When comparing different LMMs, we found that LLaVA-Llama-3-8B and MiniCPM-V-2.0 perform

almost identically on average. However, MiniCPM-V-2.0 shows better performance on novel classes. Before using LLMs, we first use LMMs to generate image descriptions, which are then used as inputs for the LLMs. In terms of computation cost, MiniCPM-V-2.0, with its smaller parameter size, generates descriptions more quickly. Therefore, we recommend this lightweight LMM as the image description generator for more efficient processing. In summary, the text comprehension capabilities of LLMs are crucial for determining the quality of the optimized prompts.

Interpretable prompts generated per dataset. Table 3 showcases the diverse prompts generated by our optimizer for each dataset. Our method enhances model accuracy by concentrating on the most relevant attributes, such as those in Flowers102 [52] and Food101 [17]. Consequently, it delivers high-quality text prompts that improve vision-language models. We encourage future researchers to leverage these interpretable prompts in their own downstream tasks.

Benefit of image description. In Table 4, we assess the impact of incorporating image descriptions into the Prompt Optimization Prompt on the performance of the optimizer. The inclusion of image descriptions enhances the model's performance. This improvement suggests that IPO, when generating new prompts, can effectively integrate information from the images themselves. As a result, the optimizer is able to produce prompts that are more specific to the data, thereby increasing the relevance and accuracy of the generated content. This highlights the importance of multimodal inputs in optimizing the prompting abilities of vision-language models.

Comparison with knowledge bank-based prompt learning methods. Our IPO leverages LLMs to optimize prompts, which conceptually aligns with previous bank-based prompt learning approaches. To assess its relative effectiveness, we compared IPO to traditional bank-based methods, specifically using L2P [57], which learns to dynamically prompt (L2P) a pre-trained model to learn tasks sequentially under different task transitions. We apply L2P within the visual prompt tuning (VPT) [12] framework, which also learns prompts in the visual space and applies them to few-shot vision- language model tasks. Similar to our IPO, VPT + L2P learns prompts within the visual space and applies them to few-shot VLM tasks. In this setup, VPT + L2P trains a prompt bank, allowing test samples to query the bank for suitable prompts during testing. The table 5 presents a comparison of VPT + L2P and our method across 11 datasets in the 16-shot setting. While L2P contributes to enhanced VPT performance, confirming

L2P's efficacy in the VLM domain, our IPO method still demonstrates superior results compared to VPT + L2P. This comparison, which we will include in the revised manuscript, reinforces that IPO 's effectiveness is not merely due to a memory retrieval mechanism but also benefits from prompt optimization through LLMs.

Comparison with state-of-the-art. Table 6 shows the comparative experiments of IPO against the state-of-the-art across 11 datasets in a 1-shot setting. Our method excels in the novel classes, surpassing the second-best performer, the original CLIP [30], by 2.78% in average performance. Other methods do not perform as well as CLIP in novel classes, indicating overfitting to base classes. For instance, CoCoOP [11] performs better in base classes but falls behind the original CLIP by 2.05% in novel classes. In particular, on the most challenging FGVCAircraft [53], and EuroSAT [54] dataset, the current state-of-the-art model, CoPrompt [32], performs poorly. This is because methods based on prefixtuning, like CoPrompt [32], PromptSRC [28], require substantial amounts of data for training to achieve adequate generalization. Consequently, on more challenging datasets, when data is scarce, it becomes difficult to fine-

tune these prefixes effectively. In contrast, our model outperforms other methods, exceeding the second-best by 1.78% in harmonic mean. Additionally, in Table 7, we compared our method's performance with other traditional gradient-based prompt learning methods on a 16-shot setting across all datasets, where our approach consistently performs well in novel classes. Demonstrating that our approach can mitigate overfitting and generalize better to novel classes.

6 Conclusion

In this paper, we presented a novel approach to prompt optimization for vision-language models, addressing the limitations of existing gradient-descent-based methods. By integrating large language models for dynamic text prompt generation and optimization, we introduced the IPO system. This system guides LLMs in crafting effective prompts while maintaining a record of past prompts and their performance metrics, offering valuable in-context information. Additionally, we incorporated large multimodal models to generate image descriptions, enhancing the synergy between textual and visual modalities. Our comprehensive evaluation across 11 datasets demonstrated that our method improves the initial accuracy of vision-language models compared to traditional gradient-descentbased prompt learning methods. Most notably, our approach significantly enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, IPO ensures that the prompts remain human-understandable, thereby facilitating better transparency and oversight for vision-language models. This improvement in interpretability is crucial, as it allows for more effective and trustworthy human-AI collaboration, making vision-language systems more reliable and accessible.

Limitation . Our IPO method is primarily designed for few-shot scenarios. However, when dealing with large domain-specific datasets, the need to generate extensive image descriptions, which can lead to substantial computational costs due to the large text inputs required for LLMs. Currently, our model uses an input length of approximately 5,000 tokens. When scaled to larger datasets, the input length may increase to around 50,000 tokens. Using GPT-4 with an 8k context length, the cost for our current input size (5,000 tokens) is approximately 0.15 dollars per input (0.03 dollars per 1,000 tokens). For the expanded input size of 50,000 tokens, the cost would rise to approximately 3.00 dollars per input. If we were to use GPT-4 with a 32k context length, the cost for the 50,000-token input would be approximately 3.00 dollars for the first 32,000 tokens and an additional 1.08 dollars for the remaining 18,000 tokens, totaling approximately 4.08 dollars per input. Since our IPO method requires 100 iterations during training, the costs would multiply accordingly when scaled to large inputs. In future work, we aim to investigate methods for LMM fine-tuning to enable the direct input of both images and text, thereby generating even more sample-specific prompts.

Broader Impact. This paper explores the use of an LLM as an optimizer for refining text prompts in vision-language models. We introduce a straightforward yet interpretable approach to prompt optimization, which holds potential for societal impact, particularly in vision-language tasks.

Acknowledgment

This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763. PMLR, 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , pages 4904-4916. PMLR, 2021.

Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training. NeurIPS , 35:7290-7303, 2022.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , pages 740-755. Springer, 2014.

Mengze Li, Tianbao Wang, Haoyu Zhang, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang, Wenming Tan, Jin Wang, and Peng Wang. End-to-end modeling via information tree for one-shot natural language spatial video grounding. In ACL , pages 8707-8717, 2022.

Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. Winner: Weakly-supervised hierarchical decomposition and alignment for spatiotemporal video grounding. In CVPR , pages 23090-23099, 2023.

Wenqiao Zhang, Haochen Shi, Jiannan Guo, Shengyu Zhang, Qingpeng Cai, Juncheng Li, Sihui Luo, and Yueting Zhuang. Magic: Multimodal relational graph adversarial inference for diverse and unpaired text-based image captioning. In AAAI , pages 3335-3343, 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134 , 2021.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP , 2021.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103.10385 , 2021.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , 2022.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119 , 2022.

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR , pages 19113-19122, 2023.

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.03409 , 2023.

Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Black-box prompt optimization: Aligning large language models without model training. arXiv preprint arXiv:2311.04155 , 2023.

Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning. arXiv preprint arXiv:2309.06553 , 2023.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In ECCV , 2014.

Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980 , 2020.

Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL , 8:423-438, 2020.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generalization. In ICCV , pages 15237-15246, 2023.

Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR , pages 5206-5215, 2022.

Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV , pages 15659-15669, 2023.

Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledgeguided context optimization. In CVPR , pages 6757-6767, 2023.

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17274 , 2022.

Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Objectagnostic prompt learning for zero-shot anomaly detection. arXiv preprint arXiv:2310.18961 , 2023.

Zheng Li, Xiang Li, Xinyi Fu, Xing Zhang, Weiqiang Wang, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. arXiv preprint arXiv:2403.02781 , 2024.

Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In CVPR , pages 15190-15200, 2023.

Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani, Shengcai Liao, and Cees GM Snoek. Any-shift prompting for generalization over distributions. arXiv preprint arXiv:2402.10099 , 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervision. In ICML , 2021.

Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vision-language models. arXiv preprint arXiv:2401.02418 , 2024.

Shuvendu Roy and Ali Etemad. Consistency-guided prompt learning for vision-language models. arXiv preprint arXiv:2306.01195 , 2023.

Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In ICLR , 2022.

Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495 , 2023.

Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532 , 2023.

Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797 , 2023.

Mirac Suzgun and Adam Tauman Kalai. Meta-prompting: Enhancing language models with task-agnostic scaffolding. arXiv preprint arXiv:2401.12954 , 2024.

Keshav Ramji and Alexander Kyimpopkin. Self-select: Optimizing instruction selection for large language models. In NeurIPS 2023 Foundation Models for Decision Making Workshop , 2023.

Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792 , 2023.

Shihong Liu, Samuel Yu, Zhiqiu Lin, Deepak Pathak, and Deva Ramanan. Language models as black-box optimizers for vision-language models. In CVPR , pages 12687-12697, 2024.

MJehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuhene, and Horst Possegger. Meta-prompting for automating zero-shot visual recognition with llms. arXiv preprint arXiv:2403.11755 , 2024.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint arXiv:2308.12038 , 2023.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. In ICML , pages 19730-19742. PMLR, 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 , 2023.

Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Tasırlar. Introducing our multimodal models, 2023. URL https://www. adept. ai/blog/fuyu-8b , 2, 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS , 36, 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR , 2004.

Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR , 2012.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV , 2013.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP , 2008.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR , 2010.

Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR , pages 139-149, 2022.

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision , pages 736-753. Springer, 2022.

Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegclip: Towards adapting clip for zero-shot semantic segmentation. In CVPR , pages 11175-11185, 2023.

Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR , pages 8256-8265, 2019.

Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. NeurIPS , 32, 2019.

Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In ICMM , pages 1921-1929, 2020.

Jiaxin Cheng, Soumyaroop Nandi, Prem Natarajan, and Wael Abd-Almageed. Sign: Spatialinformation incorporated generative network for generalized zero-shot semantic segmentation. In ICCV , pages 9556-9566, 2021.

Donghyeon Baek, Youngmin Oh, and Bumsub Ham. Exploiting a joint embedding space for generalized zero-shot semantic segmentation. In ICCV , pages 9536-9545, 2021.

Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR , pages 11583-11592, 2022.

Yassine Ouali, Adrian Bulat, Brais Matinez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision-language models. In ICCV , pages 15534-15546, 2023.

Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253 , 2022.

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

Base

82.69

80.47

82.28

84.26

Novel

63.22

71.69

75.14

76.10

H

71.66

75.83

78.55

79.97

79.92

80.51

80.21

(a)

Average over 11 datasets.

ViT-B/16

Base

Novel

H

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

(d)

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

93.67

95.20

95.43

95.33

95.29

97.69

97.76

97.30

95.21

98.23

OxfordPets

Base

Novel

88.33

90.70

90.71

90.67

90.92

(g)

82.26

91.29

92.05

91.53

93.08

Food101

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

(e)

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

(h)

Base

76.47

75.98

76.66

77.60

Novel

67.88

70.43

70.54

70.73

77.83

72.45

ImageNet

Base

Novel

78.12

70.49

72.94

78.27

60.40

73.59

74.00

74.97

73.42

75.71

StanfordCars

Base

Novel

40.44

33.41

37.44

42.73

22.30

23.71

35.61

37.87

41.21

41.42

FGVCAircraft

Novel

89.81

93.81

94.36

94.03

Base

98.00

97.96

97.74

98.10

97.32

95.23

Caltech101

Base

Novel

97.60

94.87

95.92

98.07

59.67

71.75

72.46

76.50

96.78

78.32

Flowers102

Base

Novel

80.60

79.74

80.82

82.67

81.25

(i)

65.89

76.86

78.70

78.47

80.92

SUN397

Source

Target

ImageNet

Caltech101

OxfordPets

StanfordCars

Flowers102

Food101

Aircraft

SUN397

DTD

EuroSAT

UCF101

Average

CoOp

71.51

93.70

89.14

64.51

68.71

85.30

18.47

64.15

41.92

46.39

66.55

63.88

CoCoOp

71.02

94.43

90.14

65.32

71.88

86.06

22.94

67.36

45.73

45.37

68.21

65.74

IPO

72.15

94.34

90.96

66.10

72.75

86.75

25.14

67.97

47.01

48.56

69.23

67.36

A Experiments on 16-Shots

We report the 16-shot performance of our IPO method across 11 datasets, providing detailed results for the Base, Novel, and H metrics in Table 8. Our IPO method consistently outperforms all other approaches on the novel classes and the H metric, highlighting its effectiveness in reducing overfitting.

B Experiments on cross-dataset

We conducted a comprehensive cross-dataset experimental evaluation using the standard 16-shot setting to assess the performance of our IPO method. The results, presented in Table 9, demonstrate that IPO consistently outperforms previous gradient-based prompt learning approaches. By applying our task-agnostic, LLM-driven prompt optimization technique, IPO achieved superior results across various datasets, showcasing its robustness and generalizability. These findings highlight the effectiveness of IPO in adapting to diverse tasks and domains, further reinforcing its advantage over traditional gradient-based methods in few-shot learning scenarios.

94.47

96.43

96.58

96.30

96.70

H

85.19

90.99

91.38

91.10

91.99

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

(b)

H

71.92

73.10

73.47

74.01

75.04

H

68.13

72.01

73.47

76.58

74.55

H

28.75

27.74

36.50

40.15

41.31

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

(c)

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

(f)

ViT-B/16

CoOp

CoCoOp

MaPLe

PromptSRC

IPO

H

93.73

95.84

96.02

96.02

96.26

H

74.06

81.71

82.56

85.95

86.58

H

72.517

78.27

79.75

80.52

81.08

C Impact of LLM

As demonstrated in Table 10, upgrading the LLM capacity yielded a similarly positive impact on performance. To explore how a more advanced LLM, like GPT-4, could generate more effective prompts for our model, we conducted additional experiments with both GPT-4 and GPT-4o. Specifically, when we enhanced the LLM to GPT-4o and paired it with the GPT-4o LMM, we observed a significant overall increase in the H-score by 1.77% compared to the initial setup using GPT-3.5-turbo alongside MiniCPM-V-2. This improvement underscores the advantages of employing larger, more capable models, as they facilitate greater task generalization and more robust performance. The findings suggest that scaling up model capacity in both the LMM and LLM components can lead to substantial gains in prompt quality and adaptability across various tasks, indicating a promising direction for further enhancing our model's versatility and effectiveness.

D Experiments on Segmentation Task with IPO

In prompt-based vision tasks like segmentation and detection, the design of the text prompt plays a pivotal role. Our task-agnostic method, IPO, can be seamlessly integrated into various vision tasks to optimize text prompts. For example, as shown in Table 13, we applied IPO to pre-trained semantic segmentation models [58, 59], where the original prompt used was 'a photo of a [CLASS].' By utilizing GPT-4o as both the LLM and LMM, we generated more effective, context-specific text prompts tailored to the open-vocabulary semantic segmentation task, leading to significant performance improvements. These results highlight the potential of IPO to enhance text prompt design in segmentation tasks, showcasing its adaptability and value. We plan to extend our exploration of IPO to other vision tasks in future studies, aiming to further validate its effectiveness in optimizing prompt construction across a broader range of applications.

IPO with GPT-3.5 Turbo, indeed, does not show an improvement on the large-scale ImageNet. This is because ImageNet has a large number of classes and samples, which results in longer LLM input when generating descriptions for each sample. GPT-3.5 Turbo has limited performance in handling long-text inputs. The table 12 shows the results on ImageNet when IPO uses GPT-4o, which has superior long-text understanding compared to GPT-3.5 Turbo. We found that IPO using GPT-4o leads to better performance improvements over other methods as well as a considerable improvement over IPO with GPT-3.5 Turbo.

E Experiments on segmentation task with IPO

In other prompt-based vision tasks, such as segmentation and detection, the design of the text prompt is crucial. Our method, being task-agnostic, can be easily embedded into any vision task to optimize

the text prompt. For instance, as shown in Table 13, we incorporated IPO into pre-trained semantic segmentation models [58, 59], where the original text prompt was "a photo of a [CLASS]." Using GPT-4o as the LLM and LMM, we crafted more effective text prompts specifically suited to the open-vocabulary semantic segmentation task, leading to enhanced performance and demonstrating the value of IPO in optimizing text prompts for this application. We intend to further investigate the use of IPO in other vision tasks in future work.

F Effect of batch size

The Table 14 compares the performance of GPT-3.5 turbo and GPT-4o across different batch sizes. We observed that as the batch size increases to 128, GPT-3.5 turbo's performance begins to decline due to its limited capacity for handling longer input texts effectively. In contrast, GPT-4o maintains strong performance even at larger batch sizes. However, using extremely large batch sizes with GPT-4o becomes cost-prohibitive. Therefore, we selected a batch size of 128 for our experiments. Although

even larger batch sizes could potentially improve performance further, the cost considerations become a critical factor.

G Impact of prompt history length

We evaluated the effect of varying prompt history lengths on model performance, as shown in Table 15. Our findings indicate that without prompt history, performance declines due to the absence of contextual information, making it challenging for the LLM to converge. As the history length increases, performance progressively improves, with convergence observed at n=20. Although using n=100 yields the highest average performance, the extended input length significantly raises API costs. As a result, we selected n=20 for our IPO, balancing performance gains with cost efficiency.

(a) Comparison with LFA across 11 datasets in 16-shot scenarios. (b) Comparison with PLOT on average accuracy across 11 datasets in 1-shot and 16-shot scenarios.

H Comparison with recent prompt learning methods

We conducted a comparative evaluation with LFA [66] and PLOT [67] under the same experimental conditions, as shown in Tables 16. Our IPO method consistently outperforms both LFA and PLOT across the benchmarks.

I Detailed Prompt Optimization Prompt

In Figure 3, 4, 5 and 6, 7, 8, we show detailed inputs and outputs of different training steps in our Prompt Optimization Prompt. We observed that each optimized prompt is unique, and both loss and accuracy exhibit a downward trend.

J Loss and accuracy curve.

To demonstrate that IPO can indeed serve as optimizer for prompt learning in vision-language models, we present the optimization process's loss and accuracy on ImageNet in Figure 9. As the training steps increase, the loss consistently decreases, and the accuracy gradually improves, proving the optimization capability of IPO. Additionally, IPO not only allows for interpretable prompt generation but also reduces the risk of overfitting during training.

Instruction 1

You need to perform image classification on the large-scale visual recognition dataset based on visual features.Here, <CLASS> represents

a class name from the large-scale visual recognition dataset, and it is essential to include <CLASS> in <INS>.

Below is the image description for each image:

Image descriptions

Image 0: The image showcases a plate of food featuring thinly sliced red meat, garnished with green herbs and possibly other spices

or seasonings.

Image 1: The image showcases a mouthwatering dish consisting of crispy grilled ribs with sticky sauce, accompanied by juicy apple

slices on the side.

(... more image examples. . . )

Prompt history

text prompt:

a photo of a <CLASS>, a type of food.

scores:

loss: 0.78955078125, accuracy: 82.35294342041016

Instruction 2

Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0,

while the accuracy varies from 0 to 100.

Instruction 3

Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The

prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above.

Initial Instructions

K More generated prompts

In Table 17, we provide detailed prompts for each dataset in 16-shot format. We encourage future researchers to utilize these interpretable prompts in their own downstream tasks.

L Image description with LMM

In Table 18, we provide descriptions of some training samples generated using Mini-CPM-V-2.0 on each dataset, which serve as input for image information in our Prompt Optimization Prompt. Note that we did not use this image information in the 16-shot setup due to the context length limitations of the language model.

M Prompt Optimization Prompt design

Our Prompt Optimization Prompt is a crucial component of our optimizer, serving to enhance the performance of the vision-language model by optimizing the prompts through the prompt LLM. Figure 2 displays an example of our Prompt Optimization Prompt. Initially, the instruction in the first segment of the Prompt Optimization Prompt defines the role of the LLM. It introduces two essential tokens, <INS> and <CLASS>, which represent the prompt and category, respectively. The primary function of this instruction is to inform the LLM of its role and the contents of the Prompt Optimization Prompt, enabling a more effective understanding of the Prompt Optimization Prompt. Note that in this section, our model does not involve image information, hence the Prompt Optimization Prompt here lacks statements like ' Here is a description of some features of the flowers

Instruction 1

You need to perform image classification on the large-scale visual recognition dataset based on visual features.Here, <CLASS> represents

a class name from the large-scale visual recognition dataset, and it is essential to include <CLASS> in <INS>.

Below is the image description for each image:

Image descriptions

Image 0: The image showcases a plate of food featuring thinly sliced red meat, garnished with green herbs and possibly other spices

or seasonings.

Image 1: The image showcases a mouthwatering dish consisting of crispy grilled ribs with sticky sauce, accompanied by juicy apple

slices on the side.

(... more image examples. . . )

Prompt history

text prompt:

a photo of a <CLASS>, a type of food.

scores:

loss: 0.78955078125, accuracy: 82.35294342041016

Instruction 2

Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0,

while the accuracy varies from 0 to 100.

Instruction 3

Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The

prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above.

STEP 1

text prompt:

Identify the <CLASS> in the image based on its unique visual features.

scores:

loss: 0.7109375, accuracy: 84.3137283325192

text prompt:

Identify the <CLASS> in the image and evaluate its visual features for classification.

scores:

loss: 0.69470703125, accuracy: 85.47058868408203

(... more previous prompts and scores . . . )

Instruction 1

You need to perform image classification on the large-scale visual recognition dataset based on visual features.Here, <CLASS> represents

a class name from the large-scale visual recognition dataset, and it is essential to include <CLASS> in <INS>.

Below is the image description for each image:

Image descriptions

Image 0: The image showcases a plate of food featuring thinly sliced red meat, garnished with green herbs and possibly other spices

or seasonings.

Image 1: The image showcases a mouthwatering dish consisting of crispy grilled ribs with sticky sauce, accompanied by juicy apple

slices on the side.

(... more image examples. . . )

Prompt history

text prompt:

a photo of a <CLASS>, a type of food.

scores:

loss: 0.78955078125, accuracy: 82.35294342041016

Instruction 2

Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0,

while the accuracy varies from 0 to 100.

Instruction 3

Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The

prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above.

STEP 100

text prompt:

Train a state-of-the-art deep learning model for advanced <CLASS> image classification with exceptional accuracy and minimal loss.

scores:

loss: 0.4296875, accuracy: 90.19607543945312

(... more previous prompts and scores . . . )

text prompt:

Identify the primary ingredient in the <CLASS> and describe its texture, color, and presentation.

scores:

loss: 0.49072265625, accuracy: 88.23529052734375

Below are the prompts created according to your guidelines:

(1) <INS> Classify the image based on visual features of a <CLASS>. </INS>

(2) <INS>  Identify the <CLASS> in the image and evaluate its visual features for classification. </INS>

(3) <INS>  Classify the image as a <CLASS> food item. </INS>

(4) <INS>  Classify the image featuring a <CLASS> with precision and accuracy. </INS>

(5) <INS>  Identify the <CLASS> in the image based on its unique visual features. </INS>

(6) <INS>  A photo of a <CLASS>, displaying vibrant colors and textures. </INS>

(7) <INS>  Identify the primary ingredient in the <CLASS> and describe its texture, color, and presentation. </INS>

(8) <INS>  Identify the type of food shown in the image and classify it accurately. </INS>

STEP 1

Below are the prompts created according to your guidelines:

(1) <INS>  Accurately classify the <CLASS> in the image using advanced visual recognition techniques. </INS>

(2) <INS>  Accurately classify the image for <CLASS> identification based on detailed visual analysis with

precision and minimal loss.</INS>

(3) <INS> Accurately classify the image to determine the <CLASS> based on distinctive visual cues. </INS>

(4) <INS>  Analyze the visual composition of the image to accurately classify the <CLASS> featured with precision. </INS>

(5) <INS>  Analyze the visual composition of the image to classify the <CLASS> accurately. </INS>

(6) <INS>  Analyze the distinct visual elements of the <CLASS> to achieve precise classification. </INS>

(7) <INS>  Analyze the unique visual features of the <CLASS> for precise classification with superior accuracy. </INS>

(8) <INS>  Utilize cutting-edge visual recognition algorithms to categorize the image accurately and efficiently.  </INS>

STEP 2

Below are the prompts created according to your guidelines:

<INS>  Train a state-of-the-art convolutional neural network for accurate and efficient classification of <CLASS> with exceptional precision and minimal loss. </INS>

<INS>  Implement state-of-the-art deep learning algorithms for superior <CLASS> image classification accuracy with unrivaled precision and minimal loss..</INS>

<INS> Develop an advanced neural network model for precise and efficient <CLASS> image classification with exceptional accuracy and minimal loss. </INS>

<INS>  Utilize state-of-the-art machine learning algorithms for precise and efficient <CLASS> image classification with exceptional accuracy and negligible loss. </INS>

<INS>  Develop an innovative neural network model for precise and efficient <CLASS> image classification with exceptional accuracy and minimal loss. </INS>

<INS>  Employ state-of-the-art deep learning techniques for precise and efficient <CLASS> image classification with unparalleled accuracy and minimal loss. </INS>

<INS>  Efficiently classify <CLASS> using cutting-edge deep learning techniques for unparalleled accuracy. </INS>

<INS>  Identify the primary ingredient in the <CLASS> and describe its texture, color, and presentation.  </INS>

STEP 100

(a)

Training loss.

(b)

Training accuracy.

5

LoSS

Smoothed Loss

4

training loss

m

2

0

0

20

40

60

80

100

steps

90.0

87.5

Smoothed Accuracy

85.0

accuracy

82.5

training

80.0

77.5

75.0

72.5

70.0

0

20

40

09

80

100

steps

in the image ' and the subsequent description of the image. In the next section, we will discuss how to integrate image information into the Prompt Optimization Prompt.

The subsequent instruction concerns the history of the prompts and their associated scores, which include metrics such as loss and accuracy. Our task follows CoOP [8], focusing on base classes with few samples. We calculate the loss and accuracy for these base classes using the generated prompts. Initially, we inform the LLM that the following details are past prompts along with their scores, and we clarify the range of values for loss and accuracy. What follows are the historical prompts and scores. This history functions similarly to episodic memory, to prevent the generation of suboptimal prompts while providing in-context information that enhances the creation of better prompts. Additionally, as the history of past prompts grows, we only retain the top 20 prompts based on accuracy to avoid information overload. The selection of these top 20 prompts is determined by their accuracy scores. Moreover, we consistently include the prompt ' a photo of <CLASS> ' in our history at every step, as this is a frequently used and effective prompt within the CLIP framework [30].

The final paragraph of the instruction specifies the ultimate goal of the task: to generate improved prompts based on the aforementioned instructions and the history of past prompts. The aim is to achieve lower loss and higher accuracy in this vision-language task, and it also outlines the format for the final generated prompt. These three components constitute the entire content of the Prompt Optimization Prompt, each playing a critical role for the LLM. It is important to note that in this Prompt Optimization Prompt, we do not consider the specific content of the images; we merely use images to calculate scores, and the details of the images are overlooked. In the following section, we will utilize a LMM to generate content from images and then incorporate it into the Prompt Optimization Prompt.

N More occlusion sensitivity analysis

In Tables 19-29, we present various prompts analyzed using occlusion sensitivity analysis across different models and datasets. We found that CLIP is particularly sensitive to photos. However, CoOP and CoCoOP exhibit severe overfitting on base classes, leading to poor performance on base classes when some tokens are removed, but improved performance on novel classes. In contrast, our optimized prompts show performance degradation to varying degrees when certain words or phrases are removed, indicating that the key words or phrases in our generated prompts are essential.

X

5

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope of this paper are claimed in the abstract. Detailed information can be found in the fourth paragraph of the introduction section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide a "limitation" subsection in the conclusion section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all experimental details in the experiment section. The detailed input and output of Prompt Optimization Prompt of our method can be found in Appendix and supplemental materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all experimental details in the experiment section. The detailed Prompt Optimization Prompt and Python codes of our method can be found in Appendix and supplemental materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We follow the standard experimental setup in IPO, where the data splits, is set as the same as previous works CoOP [8] and CoCoOP [11]. Detailed information can be found in Sec. 5

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Following the standard experimental setup, we repeat each experiment over 3 random seeds and report the mean of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the computing resources in experiments 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We reviewed and followed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide the potential broader impacts in the conclusion section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The data and models pose no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers that produced the code package and datasets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Details of the datasets/code/model are provided in the supplemental materials. Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method called Interpretable Prompt Optimization (IPO) for vision-language models.\nThe goal of IPO is to improve the performance and interpretability of vision-language models by dynamically generating and optimizing text prompts. The paper addresses the limitations of current prompt optimization methods, which often lead to overfitting and produce prompts that are not understandable by humans. IPO leverages large language models (LLMs) to generate effective and human-readable prompts (which is the main advantage of their method). It incorporates a Prompt Optimization Prompt that guides the LLMs in creating prompts and stores past prompts with their performance metrics, providing rich in-context information. Additionally, IPO integrates a large multimodal model (LMM) to generate image descriptions, enhancing the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance while maintaining human comprehension. The paper validates IPO across 11 datasets and demonstrates that it improves the accuracy of existing prompt learning methods and enhances the interpretability of the generated prompts.\n\nOverall, IPO ensures that the prompts remain human-understandable, facilitating better transparency and oversight for vision-language models.\n\nSTRENGTHS:\n- The paper tackles one of the most important problems in prompt tuning. Existing prompt tuning methods are not interpretable at all (as they optimize a sequence of vectors with a model specific loss function). This paper mitigates this issue by using a LLM as an optimizer -- which inherently generates text.\n\n- The prompt optimization template though is simple, can be scalable towards multiple tasks (as shown in the paper).\n\n- Thorough ablations on the language model and the multimodal language model has been provided in the paper.\n\nWEAKNESSES:\n- [Major]. While the paper obtains interpretable prompts, the method primarily obtains improved performances for the novel classes, but not for the base classes.  In fact for a large number of the datasets, the base classes performance is significantly low when compared to other methods. I believe that there should be a balance in the performance between the novel and base classes. To improve the paper, the authors are suggested to provide strong justifications explaining this.\n\n- [Minor]. This area has a multitude of papers so it might be tedious to compare with all the methods. However, I would suggest the authors to add a separate section having a discussion on the comparison of their method with more recent prompt-tuning methods such as LFA, PLOT, DFA etc. \n\n- [Minor]. I would also suggest the authors to extend the discussion of the limitations of the method. Currently, as I see, the method is suitable for few-shot scenarios, but what if one has access to a large number of domain samples? How can this method be made scalable?\n\nQUESTIONS:\nOverall, the paper provides a fresh view on prompt-tuning by making the process interpretable. In a way, the paper automates prompt-engineering for a domain by using a LLM as an optimizer. I am going with Borderline Accept, but happy to revisit my scores if the Weaknesses are addressed in the rebuttal.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-01"}

comments: SUMMARY:
This paper presents an Interpretable Prompt Optimizer (IPO) designed to improve the performance and interpretability of pre-trained visual language models such as CLIP. By dynamically generating textual cues using a large language model (LLM) and combining it with a large multimodal model (LMM) to generate image descriptions, IPO demonstrates better accuracy and interpretability than traditional gradient descent-based cue learning methods on multiple datasets.

STRENGTHS:
1）The paper proposes a new prompt optimization framework that combines the advantages of LLM and LMM.

2）Extensive testing on 11 datasets demonstrated the effectiveness of the method.

3）The paper is clearly structured with diagrams and charts to aid illustration.

4）Improved interpretability of visual language models is important for achieving better human-computer collaboration.

WEAKNESSES:
1）Line 150 :(2) -> (3)

2）There are no citations throughout the paper Table 6.

3）Lack of cross-dataset experimental evaluation to validate the generalizability of IPO.

4）The paper mentions the use of large language models such as GPT-3.5 Turbo, but does not discuss in detail the computational cost and efficiency of these models during training and inference. For resource-constrained environments, this may be an important consideration.

5）The importance of each component of the IPO is described in Section 5.2, but it lacks in-depth analysis, e.g., it could be mined for some more intrinsic reasons for the rise in points from a regularization perspective.

QUESTIONS:
1）Will this parameterless training time be shorter for IPO compared to the gradient descent approach?

2）How does the length of the prompt history affect the stability of training convergence？

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an interpretable prompt optimizer (IPO) which uses an LLM to iteratively optimize prompt templates that lead to improved zero-shot visual classification performance on CLIP.\n\nSTRENGTHS:\n1) The proposed method outperforms baselines on the novel classes in the evaluation on base-to-novel generalization benchmark\n\nWEAKNESSES:\n1) Line 91, “However,to the best of our knowledge, no existing studies have investigated how LLMs could be used to optimize text prompts within vision-language models”. There are actually related work that employed LLMs for optimizing prompts for visual classification in VLMs which are not covered in this paper, e.g.  \n[a] Liu et al. Language Models as Black-Box Optimizers for Vision-Language Models. CVPR’24  \n[b] Mirza et al. Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs. ECCV’24  \nAnalysis and proper comparison to these works should be conducted.\nHere [a] also uses an LLM (ChatGPT) to iteratively update the prompt templates for visual classification on CLIP with the good prompts and bad prompts passed as in-context examples. This weakens the novelty of the proposed method. \n2) In Table 5, why is the performance of the proposed method consistently worse than baselines on base classes? How does the method perform in a usual one-shot classification setting (instead of base-to-novel setting)? \n3) In table 1,  the metric “H” should be specified\n\nQUESTIONS:\n1) In the base-to-novel evaluation, 1-shot refers to the setting where one sample of each category in the base classes is provided during training while no sample of the novel classes is used in training? \n2) Line 165, “all four models failed to understand our Prompt Optimization Prompt”, here it should be four models or six models (as the experiments are conducted on six models)?\n\nLIMITATIONS:\nThe proposed method has improved performance on novel classes, but suboptimal performance on base classes where the prompts are optimized.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of optimizing text prompts for vision-language models, specifically focusing on the interpretability of these prompts. Traditional methods for prompt optimization rely on gradient descent, which often results in overfitting and produces prompts that are not human-readable. This paper introduces a new approach called Interpretable Prompt Optimizer (IPO), which leverages large language models (LLMs) to generate and optimize prompts in a dynamic and interpretable manner. The paper details the design of the IPO framework and provides extensive experimental results across 11 datasets. The findings demonstrate that IPO not only improves the accuracy of vision-language models compared to traditional gradient-based methods but also significantly enhances the interpretability of the generated prompts.\n\nSTRENGTHS:\n1. The paper introduces IPO, a novel method that uses LLMs to optimize prompts in a way that maintains human readability and interpretability. This approach contrasts with traditional gradient-based methods that often produce opaque prompts.\n\n2. The POP system stores past prompts and their performance metrics, allowing LLMs to generate more effective prompts through iterative refinement. This system enhances the contextual understanding and effectiveness of the generated prompts.\n\n3. The IPO method incorporates LMMs to generate image descriptions, which improves the synergy between textual and visual data. This leads to more accurate and contextually relevant prompts.\n\n4. The paper validates the effectiveness of IPO across 11 different datasets, demonstrating its superiority over traditional methods in terms of both accuracy and interpretability.\n\nWEAKNESSES:\nThe main weaknesses of this paper lies in the experiment can not support the effectiveness of the LLM optimizer, which is the most important contribution of the authods. IPO incorporates the Large Language Model (LLM) as an optimizer to learn an interpretable prompt pool, and everytime a new instance comes, prompts will be extracted from this prompt. This mainstream is similari to knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. Therefore, the experiments should contains:\n1) **A fair comparitive with some of the knowledge bank based prompt learning methods  such as L2P [1], AttriCLIP [2], DualPrompt [3]. To exclude the effectivness of the memory retrieval mechanism of IPO.**\n\n2) **The current comparitive experiment merely involves the 1-shot setting in Table 5. Please provides a more comprehensive experiment settings, such as 16-shots and full-sized tuning setting. Besides, the IPO methods do not draw an obvious performance improvement in benmark datasets in the case that IPO has an obviously longer prompt that the learnable method.** \n\n\n3) What's more, the IPO does not perform well in the large-scale generic datasets such as ImageNet. This is not intuitive, because the LLM should show a better performance in tghe generic datasets. Lastly, I am not agree with your average metric, because the data size of these 11 benchmarks have significant difference.\n\n4) Table 6 are not referred in the author's paper, and the detailed benchmark is also not elaborated.\n\n[1]  Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149.\n[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663.\n[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.\n\nQUESTIONS:\nAuthors explor using LLM as an interpretable optimizer for prompt learning, I think this is a contribution worth a weekly accept score, i.e., 6 scores in NeurIPS rating. However, this score should be build upon all below concers are addressed:\n1)  I can tolerate that the current performance are not that remarkable, but this tolerance should be built upon the experiments are fair. To make a fair comparison in the framework, authors should make comparison with some of the knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. So that we can exculde  the effectivness of the memory retrieval mechanism of IPO.\n\n2) Why Table 5 only contains the 1-shot comparive experiments ? Detailed 16-shots or training with more samples setting are needed. \nI will confirm the effectivness of the your IPO if you draw a remarkable performance. If not, just take a breath and provide enough evidence that LLM optimizer is effectiveness, such as sound losses, sound improvement in the training set, sound improvement in the test sets in different settings. I will agrre with the effectivness of the your IPO if the evidence are sound.\n\n\n[1] Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149. \n\n[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663. \n\n[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.\n\nLIMITATIONS:\nExperiments now are not sound and the authors do not provide enough evidnce to prove the effectiveness of IPO.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper introduces a method named IPO for VLMs which uses LLMs to dynamically generate and refine text prompts. The method is to improve the accuracy and interpretability of prompts. Experiments show that it can address issues like overfitting and lack of human comprehension in traditional gradient descent-based methods.

STRENGTHS:
The prompts generated using IPO enhance the performance of vision-language models across various vision task related datasets.
The process of generating prompts are interpretable which brings better transparency and controllability.

WEAKNESSES:
LMM seems contribute little to th final performance. Is the reason that used LMM is not strong enough to generate high-quality image caption? Also for different LLMs, the performance gap is not obvious and the author did not test on GPT4 or stronger models. 
The overall performance is not much better than optimized methods like CLIP.  
Could the authors provide more examples to prove the iterpretability advantage of this method?

QUESTIONS:
Will this method be generalized to other vision related tasks?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13