IPO: Interpretable Prompt Optimization for Vision-Language Models
Authors: Yingjun Du
Wenfang Sun
Cees G. M. Snoek
Venue: NeurIPS 2024 poster (2024)
Keywords: Prompt learning
Large language model
Interpretable Prompt Optimization
Primary area: machine_vision
TL;DR: This paper introduces a simple but interpretable prompt optimizer (IPO), that utilizes large language models (LLMs) to generate textual prompts dynamically.

Abstract: Pre-trained vision-language models like CLIP have remarkably adapted to various downstream tasks. Nonetheless, their performance heavily depends on the specificity of the input text prompts, which requires skillful prompt template engineering. Instead, current approaches to prompt optimization learn the prompts through gradient descent, where the prompts are treated as adjustable parameters. However, these methods tend to lead to overfitting of the base classes seen during training and produce prompts that are no longer understandable by humans. This paper introduces a simple but interpretable prompt optimizer (IPO), that utilizes large language models (LLMs) to generate textual prompts dynamically. We introduce a Prompt Optimization Prompt that not only guides LLMs in creating effective prompts but also stores past prompts with their performance metrics, providing rich in-context information. Additionally, we incorporate a large multimodal model (LMM) to condition on visual content by generating image descriptions, which enhance the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance, while maintaining human comprehension. Extensive testing across 11 datasets reveals that IPO not only improves the accuracy of existing gradient-descent-based prompt learning methods but also considerably enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, our approach ensures that the prompts remain human-understandable, thereby facilitating better transparency and oversight for vision-language models.

IPO: Interpretable Prompt Optimization for Vision-Language Models
Yingjun Du"", Wenfang Sun'; Cees G. M. Snoek' AIM Lab, University of Amsterdam University of Science and Technology of China
Abstract
Pre-trained vision-language models like CLIP have remarkably adapted to various downstream tasks. Nonetheless, their performance heavily depends the speci- ficity of the input text prompts, which requires skillful prompt template engineering. Instead, current approaches to prompt optimization learn the prompts through gra- dient descent, where the prompts are treated as adjustable parameters. However, during training
these methods tend lead to overfitting of the base classes and produce prompts that are no longer understandable by humans. This paper introduces simple but interpretable prompt optimizer (IPO), that utilizes large language models (LLMs) to generate textual prompts dynamically. We introduce a Prompt Optimization Prompt that not only guides LLMs in creating effective prompts but also stores past prompts with their performance metrics, providing rich in-context information. Additionally, we incorporate large multimodal model (LMM) to condition on visual content by generating image descriptions, which enhance the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance, while maintaining human comprehension. Extensive testing across 11 datasets reveals that IPO not only improves the accuracy of existing gradient-descent-base prompt learning methods but also considerably enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, our approach ensures that the prompts remain numan-understame and oversight for vision-language models.
thereby facilitating better transparency
Introduction
Vision-language models, trained on a diverse array of image-text pairs encapsulating broad vo- cabulary of real-world concepts [1, 2, 3], have demonstrated notable adaptability across various downstream tasks [4, 5, 6, 7]. These models perform zero-shot image classification by filling in predefined prompt template (c.g.. "a photo of [CLASS]") with specific class names for the text encoder. Despite their effective generalization to new tasks, the performance can influenced by minor changes in the wording of prompt templates [8]. Instead of manually creating hand- crafted prompts, recent developments in natural language processing [9, 10] and computer vision [8, 11, 12, 13] have proposed methods to learn set of soft prompts with minimal labeled data. Despite the strides made learning prompts, the current state of the art remains limited by its lack of interpretability and the overfitting problems on the base classes, which can be prohibitive in diverse and dynamic application environments. These limitations underscore the need for a more adaptable and usee-friendl approach to prompt optimization in vision-language models.
Drawing from recent advancements in using large language models (LLMs) as optimization tools [14], paper, for the first time, incorporates these capabilities into vision-language modeling. Unlike gradient descent-based methods [8, 11, 13], which often fail to provide explanations for the generated
Equual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

prompt
Greiiereennee sptinde
Optinized prompt
prompt
Interpteetbble prompt optimizer
Optin and promet
photo
Zake Semechtaking and Flaxceensly demad led phono (CGAAG)
Shone Derrsster Promet
Trage desomptie
Soores
Scones
Proept hostery
CUP
Scores
Bcorcs
MOC R.M.
LERM
(a) Gradient-based prompt optimization.
(b) Interpretable prompt optimization.
Figure Comparison between traditional muddient-based prompt optimization (a) and our interpretable prompt optimization (b) for vision-language models. Traditional gradient descent-based prompt learning methods [8. 11] treat the text prompt learnable parameters V. By minimizing the loss through gradient descent on the training set, an optimized prompt V is obtained after I iterations, which is not interpretable by humans. In contrast, our interpretable prompt optimization leverages an LLM as optimizer to optimize the loss and accuracy. After I iterations, the resulting optimized top prompt is effective and meadabe..
prompts and tend overfit on base classes, natural language-basee methods enable LLMs develop and refine solutions through continuous feedback iteratively. This approach improves interpretability in complex tasks like prompt optimization for vision- language models, making it easier for humans to understand the generated prompts. However, existing research on these methods [14, 15, 16] primarily addresses language tasks and has not yet explored their potential for integrating prompt optimization with an LLM in vision-lanuuage models.
To address these challenges, this paper proposes an interpretable prompt optimizer (IPO) for vision- language models that leverages the capabilities of LLMs to generate and refine text prompts dy- namically. First, we design Prompt Optimization Prompt to prompt LLMs to generate more effective prompts that improve the accuracy of CLIP and reduce the loss in base classes. Our Prompt Optimization Prompt also stores past prompts along with their corresponding accuracy and loss as episodic memory, thereby providing richer in-context information to enable LLMs to generate more effective prompts. Second, incorporate image information within the Prompt Optimization Prompt, we propose using a large multimodal model (LMM) to generate descrip- tions of images base classes that can be added to the Prompt Optimization Prompt. This in- tegration facilitates more intuitive interaction between the textual and visual modalities, which allows the Prompt Optimization Prompt to utilize image information, thereby generating dataset- specific prompts to enhance the generalization performance of CLIP. The framework of our IPO, illustrated in Figure showcases the comparison between traditional gradient-based prompt opti- mization and the proposed interpretable prompt optimization. Third, the prompts generated by our optimizer are human-interpretabe prompt evolves from I'a photo of [CLASS]"] to ["Categorize the image depicting delicious and appetizing <CLASS> with remarkable visual qualities. "]. Our gen- erated prompts perform 10.29% better in novel classes than the gradient-based method CoOP [8], reducing overfitting while maintaining interpretability.
For example, on the Food101 dataset [17]. the initial
We validated our IPO across 11 different datasets, demonstrating that it surpasses traditional gradient- based state-of-the-at methods accuracy and excels in interpretabilityy Our approach generates suman-ccomprehensibb prompts that can be seamlessly integrated into existing vision-language models to enhance performance. We conducted rigorous comparative experiments to quantify the interpretability between gradient-based prompt learning and our method. We demonstrated the importance of specific keywords in our generated prompts and revealed that not all tokens learned through traditional prompt learning methods are essential.
2 Related work
Prompt learning in vision-language models. Prompt learning, originally introduced in the natural language processing community [18, 19, 20], involves applying a fixed function to input tokens to provide task instructions to the model. the computer vision community, prompt learning has been explored in various forms, including textual prompt tuning [8, 21, 22, 23, 24], and prefix tuning [13, 12, 25, 26, 27, 28, 29]. 1) Prompt tuning mainly involves treating text prompts as learnable parameters, using a small amount of data to fine-tune these parameters. As pioncered by CoOp [8]

---

and CoCoOp [11], which both fine-tune a CLIP rision-language model [30] for few-shot transfer by optimizing a continuous set of prompt vectors within its language branch. Bayesian prompt learning [21] formulated prompt learning as a variational inference problem and demonstrated its ability for unseen class generalization. 2) Prefix tuning primarily involves adding learnable tokens to the text encoder [31], vision encoder [12, 25], or both encoders [13, 27, 28, 32]. These tokens are fine-tuned using small amount of data. Note that these methods do not optimize the initial text prompts. Instead, they focus enhancing the model's understanding capabilities by integrating these additional, trainable tokens. Our method belongs to prompt tuning. but unlike previous approaches that gradient descent to optimize prompts, we propose using LLMs to optimize prompts. Our method leverages the natural language capabilities of LLMs to iteratively refine feedback- prompts, aiming to enhance both the effectiveness and the explainability of the prompts.
LLMs as prompt optimizers. Several recent works explore the role of LLMSs as prompt optimizers for NLP tasks. Some use L.MMs to directly optimize the task instruction for in-context learning [33, 34, 14]. Other studies use L.MSs to mutate prompts for evolutionary algorithms [35, 36]. However, to the best of our knowledge, no existing studies have investigated how LLMs could be used to optimize text prompts within vision-language models. This approach could potentially open up new avenues for integrating and enhancing the capabilities of vision-language models through more effective and contextually appropriate text prompts.
Meta- Meto-pronp Suzgun and Kalai [37] introduce meta-promp prompting to transform a single LLM into a versatile "conductor" capable of managing and integrating multiple independent LLM queries. By using high-level instructions, meta- prompttpopptngggggg guides the L.M in decomposing complex tasks into smaller subtasks. The core of OPRO [14] involves designing meta-prompt for L.MMs to optimize prompts for each task. This meta- peeta-prompt includes two key pieces of information: previously generated prompts with their corresponding training accuracies and a description of the optimization problem. Self-select [38] leverages meta-prompting to optimize instruction selection. It considers a set of provided templates and chooses the most suitable template. Meta-prompting is related to instruction tuning [39] as both techniques provide high-level guidance to improve the performance and adaptability of LLMs. However, while instruction tuning focuses on mue-tuning models with variety of tasks to improve generalization, meta- prompting offers the advantage of dynamically guiding the model to decompose and manage complex tasks in real-time. Liu et al. [40] proposes a method that utilizes LLMs as black-box optimizers for vision-language models, iteratively refining prompts based on in-context examples. Their approach focuses on leveraging ChatGPT to improve prompt templates for visual classification tasks. Mirza et al. [41] explores different aspect of prompt optimization by focusing on zero-shot vision-language models. Our Prompt Optimization Prompt akin to meta-prompting, it stores past prompts along with their corresponding accuracy and loss, thereby providing richer in-context information to enable LLMs to generate more effective prompts. Different from prior meta-prompting our Prompt Optimization Prompt generates prompts beyond LLMs for vision- languale models.
3 Preliminaries
Contrastive Language-Image Pre- Training (CLIP). The goal of CLIP [30] is to develop an image encoder fr and text encoder gr via contrastive pre-training with large collection of paired images and captions. This process aims to map image-text pairs to common semantic space. After the pre-training phase, CLIP is able to perform zero-shot visual recognition by treating classification as a task of matching images to text. Specifically, the placeholder term "[CLASS]" is used within a prompt template (e.g.. "a photo of a [CLASS]") ") for the text encoder gr. Here, (T, denotes the text features adapted for class i, and the probability of classifying class i from an image I is:
(T)) p(==i exp(eepplos (T,) (T))/T)
where (er (T;), (I)) represents the cosine similarity between the image feature fr (I) and the text feature gr (T)) specific to the i-th class, K is the total number of classes, and is the temperature parameter that is tuned during training.
Prompt learning improves the adaptability of the CLIP model by eliminating the need for man- ual prompt engineering. It facilitates the automatic generation of prompts using a limited number of examples from downstream task. CoOp [8] presents a method where a set of M continu-
3

---

ous context vectors (=(u1, U2,....... VM serve as the learnable prompt. The constructed prompt ( U2..... UM. ) merges these learnable context vectors V with the class-specific token embedding c, which is then processed by the text encoder 9r(-). In CoOp, the optimization of these static context vectors V aims to minimize the negative log- eeglikkli for the correct class token:
CCE(V)= Σ yi log (T)()),
(2)
here, y/ represents the one-hot encoded ground-truth label for class i. In downstream applications, the pre-trained model parameters are kept unchanged, which allows the learnable prompt vectors V to be optimized efficiently using only a small number of samples through the minimization of the cross-entropy loss.
4 Methods
Figure depicts the comprehensive structure of our interpretable prompt optimizer. At each step of the optimization, the LLM generates candidate prompts for the vision-languaa language task by considering both the description of the optimization problem and the feedback from previously evaluated prompts stored in the Prompt Optimization Prompt. These new prompts are then assessed and incorporated into the Prompt Optimization Prompt for future optimization cycles. The optimization process concludes either when the LLM can no longer generate prompts that improve the optimization scores, or when a predefined maximum number of optimization steps is reached. Next, we will detail the design of the Prompt Optimization Prompt and explain how image information is integrated into the Prompt Optimization Prompt.
Prompt Optimization Prompt design At the core of our optimizer is the design of the Prompt Optimization Prompt, which enhances the performance of the vision- laagua- model by optimizing the prompts through the prompl LLM. Figure 2 shows an example of our Prompt Optimization Prompt. Our Prompt Optimization Prompt consists of the following components: (1) Instructions: These guide the LLM by clearly defining its task to optimize the prompt for achieving better performance in classification tasks. (2) Textual descriptions of training images: These descriptions provide the
LLM with detailed information about the images, enabling it to generate dataset-speciff prompts. (3) Previously generated prompts and corresponding scores: This component supplics in-context information, including past prompts and their performance metrics, allowing the LLM to refine its prompt generation more accurately. By incorporating these elements, our approach leverages the iterative refinement capabilities of LLMs to dynamically generate and optimize text prompts. The instructions ensure that the LLM understands the optimization goal, the textual descriptions offer rich image-related context, and the historical data aids in producing more effective and precise prompts.
Textual descriptions of training images For the textual descriptions of training images, we utilize large multimodal model (LMM) to generate text descriptions for each training image. Specifically, we employ MiniCPM-V-2.00 V-2.0 [43] to generate descriptions of the content of images from base classes. In the appendix. we provide content descriptions for some images from each datasct generated using MiniCPM-V-2.0. We denote the extracted image textual features as (.).
Additionally, we have attempted to directly optimize prompts using the LMM with the Prompt Opti- mization Prompt Specifically, we input images from the base classes and the Prompt Optimization Prompt into the LMM, aiming for the LMM to generate better prompts. We experimented with six different LMMs: BLIP-2 [44], Qwen-VL--hann VL-Chat--9666 6B [45], FUYU-SB [46], M-nCCPM-V-2.0 [43], and llava-Ilama3-80 -B [47]. Unfortunately, all six models failed to understand our Prompt Optimization Prompt and generated new prompts that were merely descriptions of the images, not the universal prompts we desired. This failure might be due the fact that the training of these LMMs did not consider such a task. Note that image descriptions are not mandatory. In our 16-shot experiments, we omitted the image descriptions in the Prompt Optimization Prompt due to the limited text input length that the LLM can handle.
Episodic memory retrieval We utilize an episodic memory mechanism to retrieve past prompts and their corresponding scores, which include metrics such as loss and accuracy. Here, we denote the memory as M. During each iteration, we retrieve the top-20 prompts R(M), based on their accuracy from M and use them as the current memory, denoted as m. Moreover, we consistently include the prompt "a photo of <CLASS>" in our history at every step, as this is frequently used

---

INPUT
Instruction
You need perform image di classiication the la scale visual recogrition dataser based visual features Hero, <CLABE> represents class name from the large scale visual resogntion dataset, and it essontial to indlude <CLASS- 4INS>.
Below the agg desorition for each imgg::
Image descriptions
Image The mage showcases c coodeuu up view texture object with distinct strppe, buttons and strtching details.
Imag The leaf has yollowroh background with darker spots and imoqullr black marks, indicating poss bre discase or damage.
more imge exampies. )
Instruction
Below are some previous prompts their scores. The scores consist loss and accurocy, The loss value equal to greater than wh the accuratty var from 0 100.
Prompt history
prompt
16400040075, 190804
text prompt
Idietify the intiiaala debails and unique patteens the <CLASS> lectur.
scores:
loss: 1.30006408755 sccuracy: 830088044070001
previcus grompte and scores...
Instruction
prompt
promots above, with accuracy the prompts above. </INS promot condse, effective, genorally applicable proctoms above.
prompt begin
OUTPUT
Bellow prompts created acconding your guldeilnes:
Anatyze HCLASS-
detail,
Thoroughty anayye and
<GLASS>
Analyzo,
the <CLASS> unperalleted excectional detall. <1NB>
prompts examples...
Figure 2: An example of Prompt Optimization Prompt with input and output the DTD [42] dataset. The red text represents instructions given the large language model, the blue text denotes the image descriptions generated by the large multimodal model, and the green text indicates top-20 previously generated prompts retrieved from episodic memory along with their corresponding scores. yellow indicates the output prompt.
and effective prompt within the CLIP framework [30]. Therefore, optimization loss is defined as:
LCe Σ y log m,,
where Z indicates the our designed instruction for LLM, f represents the new human-interpretabb text prompt optimized by the LLM. Note that our optimizer is parameter-fie free, which differentiates from traditional gradient-based prompt learning methods. Instead, leverage the LLM to optimize the prompt, reducing LCF iteratively until convergence.
The input and output example in Figure 2 shows the structured information fed into the LLM, while output demonstrates the optimized prompts generated by the L.MM. For more detailed examples of Prompt Optimization Prompt input and output, please refer the appendix.
5

---

5 Experiments
5.1 Experimental setup
We validate the effectiveness of our approach on the base-to-new generalization benchmark for evaluating prompt learning
laniuge models [8. 11]. Across all experiments, we benchmark the models' performance in 1-shot and commonly used 16-shot setting. To ensure consistency, all results from learning-bai baaiid methods are averaged over three random seeds. We use the harmonic mean (H) as the average metric, which is a common approach in prompt learning for vision-languae models.
Eleven Datasets. We follow CLIP [30] and CoOp [8] to use 11 image classification datasets, i.e., ImageNet [48] and Caltech [49] for generic object classification, OxfordPets [50], Stanford- Cars [51], Flowers 102 [52], Food101 [17] and FGVCAircraft [53] for fine-grained image recognition, EuroSAT [54] for satellite image classification, UCF101 [55] for action classification, DTD [42] for texture classification, and SUN397 [56] for scene recognition.
Six Baselines. To conduct comparative evaluation, utilize a number of established baselines including CLIP [30], Coop [8], CoCoOp [11], MaPLe [13], PromptSRC [28], and CoPrompt [32]. Note that all methods do not present 1-shot results in their publications, so perform 1-shot experiments using their available code.
Training details. We use GPT-3.5 Turbo our default optimizer, iterating 100 steps for each dataset to derive the final prompt. At each step, we generate five prompts and compare their accuracy with past prompts, storing the top-20 prompts in our history. Ultimately, we select the prompt with the highest accuracy as the final prompt. For generating image descriptions, we employ MiniCPM-V-2.0 [43] as the default LMM, using the prompt: "[Please generate the description in detail, do not provide the class name in the description.1 to the 1-shot Prompt Optimization Prompt but not to the 16-shot version due the character input limitations of GPT-3.5 Turbo, which prevent adding detailed information for each class's images. All experiments were conducted on a GeForee RTX 3090. Our code is available at https: / //github Lommmmmdss/IIO.
We added image descriptions
5.2 Results
Interpretable prompt analysis. To analyze the importance of specific words phrases in text prompts, we display a comparative experiment on the Flower102 dataset. Specifically, in Table la, use "a photo of <CLASS>" as the prompt, which is the most commonly utilized format. By employing occlusion sensitivity analysis, we individually remove each word to test the importance of the remaining words. We find that using just "<CLASS>" as the prompt performs only 0.89% lower on the novel classes than "a photo of a <CLASS>", indicating that the CLIP model can generate more discriminative features using just the category name. Additionally, the prompt "a photo <CLASS>." " achieves the best performance. By comparing "<CLASS>" with "photo <CLASS>", we determine that the word photo is particularly significant on the novel classes.
In Table 1b, we present the results of the CoOP and CoCoOP models when some learned prompt tokens are removed analyze which token is most crucial. Surprisingly, for both models, performance improves in novel classes when some or all tokens are removed. This indicates severe overfitting in base classes by these models, where the learned tokens are only applicable base classes. Removing all tokens allows the models to retain some of the original performance of CLIP on novel classes. Additionally, these methods, which only learn tokens, make challenging for humans understand the specific meanings of each token, complicating interpretation.
In Table 1c, we also demonstrate the final prompt produced by our model: "Identify the unique visual features of the <CLASS> flower accurately." which achieves a performance 79.6%, surpassing the original CLIP by 2.9% on novel classes. When comparing this optimal prompt with others, removing any words from it results in worse performance than the original prompt. Specifically, comparing "Identify <CLASS>" with "<CLASS>" reveals that including "Identify" boosts performance by 1.7% on novel classes. This highlights the importance of the word "Identify" in datasets like 100 We have included additional analyses on various datasets in the appendix.
Overall, the comparative experiments demonstrate that our prompt can be more easily interpreted and understood by humans, while also providing insights into the significance of certain key words

---

Prompts
CLIP [30]
Prompts
CoOP [S]
COCoOP [11]
Base
Novel
Base
Novel
H
Base
Novel
H
photo of <CLASS>.
69.34
76.72
72.84
None
61.75
75.83
68.07
61.75
75.83
68.07
<CLASS>.
61.75
75.83
68.07
Token
76.02
73.38
74.68
75.83
76.92
76.37
<CL.SSS>
63.04
75.02
68.51
Token
70.07
74.26
72.10
76.75
76.98
76.85
photo <CLASS>,
64.72
76.74
70.22
Token
72.88
T1.23
77 07
Tokca
70.85
67.75
69.27
76.51
77.64
77.07
of <CLASSS>
63.02
76.17
Token 1.2
75.04
74.23
74.63
76.02
71.23
76.62
photo <CLASSS>
66.52
77.25
71.48
photo of <CLASS>,
61.33
77.12
68.32
Token 1,4
75.92
68.81
72.20
76.49
78.22
77.35
of <CLASS>
63.37
75.91
69.08
Token 1, 3
72.55
74.04
73.29
76.51
75.53
76.02
photo <CLASSS>
59.15
77.25
70.00
Token
Token 1,3,4
74.82
76.25
70.52
66.15
72.61
70.84
76.04
75.53
71.92
78.37
76.92
76.97
photo af <CLASS>
76.72
72.74
Token 3,
70.91
67.93
69.39
75.25
71.71
76.46
(a) CLIIP
(b) CoOP and CoCoOP.
Prompts
IPO
Base
Novel
H
Identify the unique visual features of the <CLASS> flower accurately. <CLASS>.
74.17
79.65
76.81
61.75
75.83
68.07
Visual features of the <CLASS>.
64.42
74.64
69.15
Identify unique visual features <CLASS>. Identify unique visual features the <CLASS> accurately.
63.28
77.21
69.55
65.67
77.35
71.03
The unique visual features <CLASS> flower. Identify the unique <CIASSS> Identify the <CLASS>. Identify <CLASS>, Features the <CLASS> flower accurately. Visual features of the <CLASS> flower.
66.42
76.67
71.18
67.22
77.73
72.09
68.88
76.35
72.42
69.53
77.52
73.31
72.03
77.82
74.81
71.93
78.55
75.09
The unique visual features of the <CLASS> flower accurately, Identify the unique visual features of the <CLASS> flower.
72.67
78.72
75.57
73.82
79.11
76.37
(c) This paper: IPO
Table Comparison various prompts with occlusion sensitivity analysis across different models on the Flower102 dataset [52]. The shaded areas in the table indicate the original performance of each method. Bold blue refers to the result of the best prompt for each model. The original CLIP model shows particular sensitivity the word photo. In contrast, the tokens learned CoOP and CoCoOP affect especially base class performance. while removing these Iearned tokens improves novel class performance. By contrast, with interpretable prompt optimization, every word makes a meaningful contribution to both base and new classes. We provide results for more datasets in the appendix.
Models
Base
Novel
H
Models
Base
Novel
H
Phi2-2.7B
71.15
75.43
73.22
FUYU-8B
70.98
75.45
73.14
PaLM 2-1.
71.32
75.93
73.55
BLIP--2
71.95
76.52
73.16
PaLM 2-L-IT
71.13
76.16
73.56
Qwen-Vie V1.-Chat--9.66
71.23
77.08
74.03
Phi3-7B
71.43
76.68
73.96
LLaVA-Llamu-3-s8
73.17
75.24
74.19
GPT-3.5-turbo
71.76
77.00
74.29
MiniCPMM--22228BB
71.76
77.00
74.29
(a) Impact of large language model.
(b) Impact of large multimodal model.
Table 2: Effect of L.M and LMM choice. obtain hest results with GPT-3.5-turbo as the L.I optimizer, and MiniCPM-V-2.00 for generating image descriptions for the Prompt Optimization Prompt.
in the prompt. This understanding can guide us in identifying crucial words that enhance prompt effectiveness.
Effect of LLM and LMM choice. Table 2, we analyze the effect of the choice of the L.M optimizer and LMM for generating image descriptions Prompt Optimization Prompt inputs. We evaluate various models and report their average performance across II datasets. Using GPT-3.5- turbo as the optimizer results in better prompt generation, especially improving performance on novel classes. The training speed of our IPO is heavily influenced by the computational efficiency of LLMS used. Since PaLM and GPT-3.5 are not open-source, we rely on their respective APIs to generate prompts. Consequently, our training speed depends on the API call latency and the computational complexity of these models. Alternatively, Phi2 and Phi3 are open-source, allowing us to generate prompts directly using their weights. Therefore, for researchers and practitioners seeking faster and more cost-effective training, we recommend utilizing open-source large models prompt generation. When comparing different LMMs,, found that LL-VA-Llama 3-BB and MiniCPM-V-D C-2-V-200 perform
7

---

Dataset
Best Prompt
ImageNet
Take high-quality photo of <CLASS>.
1a
Categorize the <CLASS> shown the image. Take a well-compoo composed photo of a <CLASS> with optimal lighting, fo- cus, and minimal distractions. Capture the pet's unique characteristics. including expression and posture, lo ensure a clear and distinct image. Describe the distinguishing characteristics of the <CLASS> in the image. Identify the unique visual features of the <CI CCLASS> flower accurately. Identify the primary ingredient the <CLASS> and describe texture, color, and presentation.
OxfordPets
StanfordCars
Flowers 102
Food101
Capture comprehensive range well-lit, high-resolution images of an <CLASS> from various angles, meticulously showcasing its specific de- sign features with perfect clarity and precision for unparalleled accuracy airerafl
FGVCAireraft
SUN397 DTD
A photo of a <CLASS>, a type of large-scale scene.
Clussify the intricate <CLASSS> texture
Analyze the <CLASS> vehicles the satellite image with state-of sate-of-the- art algorithms for precise classification and optimal elficiency.
EuRoSAT
Capture a high-quality, well-lit image of person flawlessly demon- strating the <CLASS> action with impeccable visual representation to achieve unmatched.
UCF101
Table 3: Interpretable prompts generated by our method each dataset in I-shot scenarios.
almost identically average. However, MiniCPM-VV V-2.0 shows better performance on novel classes. Before using LLMs, we first use LMMs to generate image descriptions, which are then used as inputs for the LLMs. In terms of computation cost, MuniCPM-V-2.0. with its smaller parameter size, generates descriptions more quickly. Therefore, we recommend this lightweight L.MM as the image description generator for more efficient processing. In summary, the text comprehension capabilities of L.LSs are crucial for determining the quality of the optimized prompts.
Interpretable prompts generated per dataset. Table 3 showcases the diverse prompts generated by our optimizer for each dataset. Our method enhances model accuracy by concentrating on the most relevant attributes, such as those in Flowers 102 [52] and Food101 [17]. Consequently, it delivers high-quality text prompts that improve vision-language models. We encourage future researchers to leverage these interpretable prompts in their own downstream tasks.
Benefit of image description. In Table 4, we assess the impact of incorporating image descriptions into the Prompt Optimization Prompt on the performance of the optimizer. The inclusion of image descriptions enhances the model's performance. This improve- ment suggests that IPO, when generating new prompts, can ef- fectively integrate information from the images themselves. As a result, the optimizer is able to produce prompts that are more specific the data, thereby increasing the relevance and accu- racy of the generated content. This highlights the importance of multimodal inputs in optimizing the prompting abilities of Aision-language models.
Base
New
II
CLIPP
69.34
74.22
71.70
W/o LMM w/ LMM
71.12
76.03
73.49
71.76
77.00
74,29
Table 4: Benefit of image description.
Comparison with knowledge bank-based prompt learning methods. Our IPO leverages LLMs to optimize prompts, which conceptually aligns with previous bank-based prompt learning approaches. To assess its relative effectiveness, we compared IPO traditional bank-based methods, specifically using 1.2P [57], which learns to dynamically prompt (L.2P) a pre-trained model to learn tasks sequentially under different task transitions. We apply L2P within the visual prompt tuning (VPT) [12] framework, which also learns prompts in the visual space and applies them to few-shot vision- language model tasks. Similar to our IPO, VPT
L2P learns prompts within the visual space and applies them to shot VLM tasks, In this setup, VPT L2P trains a prompt bank, allowing test samples query the bank for suitable prompts during testing. The table presents comparison of VPT L2P and our method across 11 datasets the 16-shot setting. While I.2P contributes to enhanced VPT performance, confirming
Model
Base
Novel
VPT [12]
69.34
74.22
71.70
VPT [12] LPP
71.12
76.03
73.49
VPT [12]+IPP IPO
71.76
77.00
74.29
Table 5: Comparison with knowledge bank- based prompt learning methods.
8

---

VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
CLIP
69.34
74.22
71,70
CLIP
72.43
68.14
70.22
CLIP
96.84
94,00
95.40
CoOp
72.08
66.71
69.29
CoOp
73.20
67.43
70.20
CoOp
90.63
85.20
87.83
CoCoOp
72.85
72.17
72.51
CoCoOp
73.90
69.07
71.40
CoCoOp
96.37
93.13
94.72
MaPLe
70.85
71.57
71.21
MaPLe
74.03
68.73
71.28
MaPLe
96.40
94,10
95.24
PrompiSRC
73.38
71.47
72.41
PromptSRC
73.27
68.87
71.00
PromptSRC
97.30
95.57
96.43
CoPrompt
70.44
70.11
70.27
CoPrompt
73.97
70.87
72.39
CaPrompt
97.60
95.57
96.57
IPO
71.76
77.00
74.29
IPO
74.09
69.17
71.54
IPO
96.53
95.39
95.95
(a) Average over 11 datasets,
(b) ImageNet
(c) Caltech
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
H
CLIP
91.17
97.26
94.12
CLIP
63.37
74.89
68.65
CLIP
72.08
77.80
74.83
CoOp
93.73
96.23
94.96
CoOp
61.80
68.33
64.90
CoOp
83.97
67.10
74.59
CoCoOp
93.47
96.27
94.85
CoCoOp
65.27
73.73
69.24
CoCoOp
75.57
77.00
76.28
MaPLe
90.83
96.00
93.34
MaPLe
66.00
73.67
69.62
MaPLe
77.10
76.97
77.03
PromptSRC
93.73
97.33
95.50
PromptSRC
67.93
73.73
70.71
PromptSRC
85.57
74.83
79.84
CoPrompt
92.37
96.37
94.33
CoPrompt
64.17
71.50
67.64
CoPrompt
72.90
72.93
72.91
IPO
94.48
97.93
96.43
IPO
63.83
75.45
69.16
IPO
74.17
79.65
76.81
(d) OxfordPets
(e) StanfordCars
(1) Flowers102
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
H
CLIP
90.10
91.22
90.66
CLIP
27.19
36.29
31.09
CLIP
69.36
75.35
72.23
CoOp
87.90
88.03
87.96
CoOp
27.37
27.60
27.68
CoOp
71.47
72.47
71.97
CoCoOp
88.73
89,60
89.16
CoCoOp
29.77
31.23
30.48
CoCoOp
73.67
75.50
74.57
MaPLe
89.13
90.67
89.89
MaPLe
28.33
29.00
28.66
MaPLe
74.33
76.37
75.34
PrompiSRC
88.30
91.03
89.64
PromptSRC
10.93
6.73
8.33
PrompiSRC
75.60
77.07
76.33
CoPrompt
88.40
90.60
89.49
CoPrompt
10.10
4.87
6.57
CoPrompt
76.37
78.77
77.55
IPO
89.78
91.59
90.67
IPO
31.43
36.32
33.70
IPO
72.25
77.53
74.80
(g) Food101
(h) FGVCAircraft
(i) SUN397
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
H
CLIP
53.24
59.90
56.37
CLIP
56.48
64.05
60.03
CLIP
70.53
77.50
73.85
CoOp
60.80
47.53
53.35
CoOp
69.13
50.33
58.25
CoOp
72.50
63.57
67.74
CoCoOp
58.70
52.70
55.54
CoCoOp
71.13
62.87
66.75
CoCoOp
74.73
72.80
73.75
MaPLe
58.20
54.17
56.11
MaPLe
50.20
51,20
50.70
MaPLe
74.83
76.43
75.62
PromptSRC
63.17
55.60
59.14
PromptSRC
73.27
67.00
70.00
PromptSRC
78.13
78.37
78.25
CoPrompt
62.77
60.40
61.56
CoPrompt
59.27
51.60
55.17
CoPrompt
76.93
77.73
77.33
IPO
55.45
62.47
58.75
IPO
64.97
82.13
72.54
IPO
72.43
79.35
75.73
(j) DTD
(k) EuroSAT
(I) UCF101
Table Comparison with existing state-of-ther ane-the-art methods for base-to-novel generalization using 1-shot learning. Except for CLIP, the results other methods are based on our reimplementation of their official code. Our proposed IPO exhibits robust generalization capability and achieves significant improvements on novel classes across II datasets.
L2P's efficacy in the VLM domain, our IPO method still demonstrates superior results compared to VPT + L2P. This comparison, which we will include in the revised manuscript, reinforces that IPO 's effectiveness is not merely due to a memory retrieval mechanism but also benefits from prompt optimization through LLMs.
Comparison with state-of-the-art. Table 6 shows the comparative experiments of IPO against the state-of-the-art across 11 datasets in 1-shot setting. Our method excels in the novel classes, surpassing the second-best performer, the original CLIP [30], by 2.78% in average performance. Other meth- ods do not perform as well as CL.IP in novel classes, indicat- ing overfitting base classes. For instance, CoCoOP [1 performs better in base classes but falls behind the origi-
Base
Novel
H
CLIP [30]
69.34
74.22
71.70
CoOP [8]
82.69
63.22
71.66
CoCoOp [11]
80.47
71.69
75.83
MaPLe [13]
82.28
75.14
78.55
nal CLIP by 2.05% in novel classes. In particular, on the
PromptSRC [28]
84.26
76.10
79.97
most challenging FGVCAircraft [53], and EuroS [54] dataset, the current of-te-of-the-ar art model, CoPrompt [32], performs poorly. This is because methods based on prefix- tuning, like CoPrompt [32], PromptSRC [28], require substantial amounts of data for training to achieve ade- quate generalization. Consequently, on more challenging datasets, when data is scarce, it becomes difficult to fine-
CoPrompt [32]
84.00
77.23
80.48
IPO (1-shot)
71.76
77.00
74.29
IPO (16-shot)
79.92
80.51
80.21
Table 7: Comparison with gradient-base prompt learning methods for 16-shots across 1l datasets.

---

tune these prefixes effectively. In contrast, our model outperforms other methods, exceeding the second-best by 1.78% in harmonic mean. Additionally, Table we compared our method's performance with other traditional madient-bbsed prompt learning methods on a 16-shot setting across all datasets, where our approach consistently performs well in novel classes. Demonstrating that our approach can mitigate overfitting and generalize better to novel classes.
6 Conclusion
In this paper, presented novel approach to prompt optimization for vision-language models, addressing the limitations of existing gradient-descent--aee methods. By integrating large language models for dynamic text prompt generation and optimization, we introduced the IPO system. This system guides LLMs in crafting effective prompts while maintaining a record of past prompts and their performance metrics, offering valuable in in-context information. Additionally, large multimodal models to generate image descriptions, enhancing the synergy between textual and visual modalities. Our comprehensive evaluation across 11 datasets demonstrated that method improves the initial accuracy of vision-language models compared to traditional gradient-descent- based prompt learning methods. Most notably, our approach significantly enhances the interpretability of the generated prompts. By leveraging the strengths LLMs, IPO ensures that the prompts remain uuman--undrrttndabble thereby facilitating better transparency and oversight for vision-language models. This improvement in interpretability is crucial, as it allows for more effective and trustworthy human-Al collaboration, making vision-language systems more reliable and accessible.
incorporated
Limitation. Our IPO method is primarily designed for few-shot scenarios. However, when dealing with large domain-speciff datasets, the need to generate extensive image deseriptions, which can lead to substantial computational costs due to the large text imputs required for LLMs. Currently, our model uses an input length of approximately 5,000 tokens. When scaled to larger datasets, the input length may increase to around 50,000 tokens. Using GPT-4 with an 8k context length, the cost for current input size (5,000 tokens) is approximately 0.15 dollars per input (0.03 dollars per 1,000 tokens). For the expanded input size of 50,000 tokens, the cost would rise to approximately 3.00 dollars per input. If were use GPT-4 with 32k context length, the cost for the 50, 50,000-token input would be approximately 3.00 dollars for the first 32,000 tokens and an additional 1 1.08 dollars for the remaining 18,000 tokens, totaling approximately 4.08 dollars per input. Since our IPO method requires 100 iterations during training, the costs would multiply accordingly when scaled to large inputs. In future work, we aim to investigate methods for LMM fine-tuning to enable the direct input of both images and text, thereby generating even more sample-specific prompts.
Broader Impact. This paper explores the use of an LLM an optimizer for refining text prompts in vision-language models. We introduce a straightforward yet interpretable approach to prompt optimization, which holds potential for societal impact, particularly in vision-language tasks.
Acknowledgment
This work is financially supported by the Inception Institute of Artificial Intelligence, the University of Amsterdam and the allowance Top consortia for Knowledge and Innovation (TKIs) from the Netherlands Ministry of Economic Affairs and Climate Policy.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763. PMLR, 2021.
[2] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lisoonlanguagee representation learning with noisy text supervision. In ICML, pages 4904-4916. PMLR, 2021.
[3] Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Fine-grained grained semantically aligned vision-language pre-training.
Qi Tian, and Siliang Tang. Fine- NeurlPS, 35:7290-7303, 2022.
[4] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[5] Mengze Li, Tianbao Wang. Haoyu Zhang, Shengyu Zhang, Zhou Zhao, Jiaxu Miao, Wenqiao Zhang. Wenming Tan, Jin Wang, and Peng Wang. End-to-end modeling via information tree for one-shot natural language spatial video grounding. In ACL, pages 8707-8717, 2022.
[6] Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. Winner: Weakly-supervised hierarchical decomposition and alignment for spatio- temporal video grounding. In CVPR, pages 23090-23099, 2023.
[7] Wenqiao Zhang, Haochen Shi, Jiannan Guo, Shengyu Zhang, Qingpeng Juncheng Li, Sihui Luo, and Yueting Zhuang. Magic: Multimodal relational graph adversarial inference for diverse and unpaired text-based image captioning. In AAAI, pages 3335-3343, 2022.
[8] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for langun-l models. arXiv preprint arXiv:2109.00 134, 2021.
[9] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[10] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103. 10385, 2021.
[11] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
[12] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12119 2022.
[13] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pages 19113-19122, 2023.
[14] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. arXiv preprint arXiv:2309.034000 2023.
[15] Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Black-box prompt optimization: Aligning large language models without model training. arXiv preprini arXiv:231 1ae::23/1...4444 1.04155, 2023.
[16] Hao Sun. Offline prompt evaluation and optimization with inverse reinforcement learning. arXiv preprint arXiv:2309.06553 2023.
[17] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food- fond-101-man minniing discriminative components with random forests. In ECCV, 2014.
[18] Taylor Shin, Yasaman Razeghi, Robert L I Logan IV, Eric Wallace, and Sameer Singh. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint 2xXiv:2010.15980 2020.

---

[19] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL, 8:423 4383443.220 2020.
[20] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[21] Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generalization. In ICCV, pages 15237-15246, 2023.
[22] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR, pages 5206-5215 2022.
[23] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In ICCV, pages 15659-15669, 2023.
[24] Hantao Yao. Rui Zhang, and Changsheng Xu. Visual- prompt tuning with knowledge- guided context optimization. In CVPR. pages 6757-6767, 2023.
[25] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan. and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 17274, 2022.
[26] Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Object- agnostic prompt learning for zero-shot anomaly detection. arXiv preprint arXiv:2310.189611 2023.
[27] Zheng Li, Xiang Li, Xinyi Fu, Xing Zhang, Weiqiang Wang. and Jian Yang. Promptkd: Unsupervised prompt distillation for vision-language models. arXiv preprint arXiv:2403.02781, 2024.
[28] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, -suan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In CVPR, pages 15190-15200, 2023.
[29] Zehao Xiao, Jiayi Shen, Mohammad Mahdi Derakhshani. Shengeai Liao, and Cees GM Snock. Any-shift prompting for generalization over distributions. arXiv preprint arXiv:2402. 2024.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, and Jack Clark. Learning transferable visual models from natural language supervision. In ICML, 2021.
[31] Muhammad Uzair Khattak, Muhammad Ferjad Naeem, Muzammal Naseer, Luc Van Gool, and Federico Tombari. Learning to prompt with text only supervision for vasion-language models. arXiv preprint urXiv:2401.024111 2024.
[32] Shuvendu Roy and Ali Etemad. Consistency-guin -uided prompt learning for vision-language models. arXiv preprint 230.002306..1995, 2023.
[33] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In ICLR, 2022.
[34] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.032. 03495, 2023.
[35] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint 2XXiv:2309.08332 2023.
[36] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rock- täschel. Prompthreeder: Self-referential self-improvement via prompt evolution. arXiv preprint 27X777230..1677, 2023.
12

---

[37] Mirac Suzgun and Adam Tauman Kalai. Meta-prompting: Enhancing language models with task-agnostic scaffolding. arXiv preprint arXi::240112295, 2024.
[38]
Keshav Ramji and Alexander Kyimpopkin. Self-select: Optimizing instruction selection for large language models. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
[39] Shengyu Zhang. Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308. 10792, 2023.
[40] Shihong Liu, Samuel Yu, Zhiqiu Lin, Deepak Pathak, and Deva Ramanan. Language models as black-hox optimizers for lasion-language models. In CVPR, pages 12687-12697, 2024.
[41] M Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuhene, and Horst Possegger. proa-igg for automating zero-shot visual recognition with llms. arXiv preprini arXiv:2403. / 1755, 2024.
[42] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures the wild. In CVPR, 2014.
[43] Jinyi Hu, Yuan Yao, Chongyi Wang. Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint arXiv:2308. 20..2330.....,... 2023.
[44] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICMI., pages 19730-19742. PMLR, 2023.
[45] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arX 2023.
[46] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Tasılarr. Introducing our multimodal models, 2023. URL https://www. adept. ai/blog/ficyu-8b.22 2. 2023.
[47] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2023.
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248-255, 2009.
[49] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR, 2004.
[50] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
[51] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013.
[52] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, 2008
[53] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint urXiv:1306.5155 2013.
[54] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[55] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 121.iv::112.0000 2012.
13

---

[56] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In CVPR, 2010.
[57] Zifeng Wang, Zizhao Zhang, Chen- Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, pages 139-149, 2022.
[58] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open-vocabulary semantic segmentation with pre-trained vision vision-language model. In European Conference on Computer Vision, pages 736-753. Springer, 2022.
[59] Ziqin Zhou, Yinjie Lei, Bowen Zhang, Lingqiao Liu, and Yifan Liu. Zegelip: Towards adapting clip for zero-shot semantic segmentation. In CVPR, pages 11175-11185, 2023.
[60] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, pages 8256-8265 2019.
[61] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. Zero-shot semantic segmentation. NeurIPS, 32, 2019.
[62] Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang. Context-aware feature generation for zero-shot semantic segmentation. In ICMM, pages 1921-1929, 2020.
[63] Jiaxin Cheng, Soumyaroop Nandi, Prem Natarajan, and Wael Abd-Almageed, Sign: Spatial- information incorporated generative network for generalized zero-shot semantic segmentation. 2021.
In ICCV, pages
[64] Donghyeon Baek, Youngmin Oh, and Bumsub Ham. Exploiting a joint embedding space for generalized zero-shot semantic segmentation. ICCV, pages 9536-9545, 2021.
[65] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai. Decoupling zero-shot semantic segmen- tation. In CVPR, pages 1155-11192, 2022.
[66] Yassine Ouali, Adrian Bulat, Brais Matinez, and Georgios Tzimiropoulos. Black box few-shot adaptation for vision-language models. In ICCV, pages 15534-15546, 2023.
[67] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang, Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022.
14

---

VIT-B/16
Novel
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
CoOp
82.69
63.22
71.66
CoOp
76.47
67.88
71.92
CoOp
98.00
89.81
93.73
CoCoOp
80.47
71.69
75.83
CoCoOp
75.98
70.43
73.10
CoCoOp
97.96
93.81
95.84
MaPLe
82.28
75.14
78.55
MuPLe
76.66
70.54
73.47
MuPLe
97.74
94.36
96.02
PromptSCC:
84.26
76.10
79.97
PrompiSRC
77.60
70.73
74.01
PrompSRC
98.10
94.03
96.02
IPO
79.92
80.51
80.21
IPO
77.83
72.45
75.04
IPO
97.32
95.23
96.26
(a) Average over 11 datasets.
(b) ImageNet
(c) Caltech101
VIT-B/16
Novel
VIT-B/16
Base
Novel
H
VIT-B/16
Base
Novel
H
CoOp
93.67
95.29
94,47
CoOp
78.12
60.40
68. 13
CoOp
97.60
39.67
74.06
CoCoOp
95.20
97.69
96.43
CoCoOp
70.49
73.59
72.01
CoCoOp
94.87
71.75
81.71
MaPLe
95.43
97.76
96.58
MaPLe
72.94
74.00
73.47
MaPL
95.92
72.46
82,56
PromptSRC
95.33
97.30
96.30
PromptSRC
78.27
74.97
76.58
PromptSRC
98.07
76.50
85,95
IPO
95.21
98.23
96,70
IPO
73.42
75.71
74.55
IPO
96.78
78.32
86.58
(d) OxfordPets
StanfordCars
(f) Flowers 102
VIT-B/16
Base
Novel
H
V/T6
Base
Novel
H
ViT-B/16
Basc
Novel
CuOp
85.19
CoOp
40.44
CoOp
72.517
CoCoOp
90.70
91.29
90.99
CoCoOp
33.41
23.71
27.74
CoCoOp
79.74
76.86
78.27
MaPLe
90.71
92.05
91.38
MaPLe
37.44
35.61
36.50
MaPLe
80.82
78.70
79.75
PrompiSRC
90.67
91.53
91.10
PrompiSRC
42.73
37.87
40.15
PromptSRC
82.67
78.47
80.52
IPO
90.92
93.08
91.99
IPO
41.21
41.42
41.31
(g) Food101
(h) FGVCAircraft
(i) SUN397
VIT-B/16
Novel
VIT-B/16
Base
Novel
VIT-B/16
Base
Novel
CoOp
79.44
41.18
54,24
CoOp
92.19
54.74
68.69
CoOp
84.69
56.05
67.46
CoCoOp
77.01
56.00
64.85
CoCoOp
87.49
60.04
71.21
CoCoOp
82.33
73.45
77.64
MaPLe
80.36
59.18
68.16
MaPLe
94.07
73.23
82.35
MaPLe
83.00
78.66
80.77
PromptSRC
83.37
62.97
71.75
PromptSRC
92.90
73.90
82.32
PromptSRC
87.10
78.80
82.74
IPO
82.14
66.81
73.69
IPO
94.25
80.11
86.61
IPO
85.32
80.92
83.06
(i) DTD
(k) Eu
(I) UCF101
Table 8: Comparison with existing statc-of-thh-ar methods for basc-to-nova generalization using 16-shots learning. Our proposed IPO exhibits robust generalization capability and achieves significant improvements novel classes across datasets.
Source
Target
ImageNer
Calezh1011
OxferPatss
StanfordCars
Fowwsi022
Foed101
Aircrafi
SUNW97
EunSAT
UCT101
Avarage
DTD
CoOp
71.51
93,70
89.14
64.51
68.71
18.47
64.15
41.92
46.39
CoCoOp
71.02
94.43
90. 14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
IPO
72.15
48.56
Table 9: Cross-dataset generalization. Accuracy (%) evaluation for prompts learned from source dataset. Our IPO consistently outperforms existing prompt learning methods.
A Experiments on 16-Shots
We report the 16-shot performance of our IPO method across 11 datasets, providing detailed results for the Base, Novel, and H metries in Table 8. Our IPO method consistently outperforms all other approaches on the novel classes and the H metric, highlighting its effectiveness in reducing overfitting.
B Experiments on cross-dataset
We conducted a comprehensive cross-dataset experimental evaluation using the standard 16-shot setting to assess the performance of our IPO method. The results, presented Table 9, demonstrate that IPO consistently outperforms previous gradient-based prompt learning approaches. By apply- ing our task-agnostic, LLM--riven prompt optimization technique, IPO achieved superior results across various datasets, showcasing its robustness and generalizability. These findings highlight the effectiveness of IPO adapting diverse tasks and domains, further reinforcing its advantage over traditional gradient-based methods in few-shot learning scenarios.
15

---

LLM
Params
LMM
Params Base
Novel
H
GPT-3.5-turbo
175B
MiniCPM V-2
2.8B
71.76
77.00
74.29
GPT-4
175B
MiniCPM-Va V-2
2,8B
72.67
77.62
75.06
GPT-4-0
175B
MiniCPM-V-2
2.8B
72.91
78.13
75.42
GPT-3.5-turbo
175B
GPT-4o
500B 1T
72.78
77.92
75.26
GPT-4
500B IT
GPT-4o
500B IT
72.93
78.01
75.38
GPT-4-o
500B IT
GPT-4o
500B 1T
73.41
78.93
76.06
Table 10: Impact of large language model.
LLM
Params
Base
Novel
CLIP
69.34
74.22
71.70
w/o LMM
71.12
76.03
73.49
w/MiniCPM-Va V-2
2.8B
71.76
77.00
74.29
w/GPT-40
500B
1T
72.78
77.92
75.26
Table 11: Impact of large language model.
C Impact of LLM
As demonstrated in Table 10, upgrading the LLM capacity yielded a similarly positive impact performance. To explore how a more advanced LLM, like GPT-4, could generate more effective prompts for model, conducted additional experiments with both GPT-4 and GPT-4o. Specifi- cally, when we enhanced the LLM to GPT-4o and paired it with the GPT-4o LMM, we observed a significant overall increase in the H-score by 1.77% compared the initial setup using GPT-3.5-turbo alongside MiniCPM-V-2. This improvement underscores the advantages of employing larger, more capable models, as they facilitate greater task generalization and more robust performance. The findings suggest that scaling up model capacity in both the LMM and LLM components can lead substantial gains prompt quality and adaptability across various tasks, indicating promising direction for further enhancing our model's versatility and effectiveness.
D Experiments on Segmentation Task with IPO
In prompt-basee vision tasks like segmentation and detection, the design of the text prompt plays pivotal role. Our task-agnostic method, IPO, be seamlessly integrated into various vision tasks to optimize text prompts. For example, as shown in Table 13, we applied IPO to pre-trained semantic segmentation models [58, 59], where the original prompt used was 'a photo of [CLASS). By utilizing GPT-4o as both the LLM and LMM, we generated more effective, context-specific text prompts tailored to the open-vocabulary semantic segmentation task, leading significant performance improvements. These results highlight the potential of IPO to enhance text prompt design in segmentation tasks, showcasing its adaptability and value. We plan extend our exploration of IPO to other vision tasks in future studies, aiming to further validate its effectiveness in optimizing prompt construction across a broader range of applications.
IPO with GPT-3.5 Turbo, indeed, does not show an improvement on the large-scale ImageNet. This because ImageNet has large number of classes and samples, which results in longer LLM input when generating descriptions for each sample. GPT-3.5 Turbo has limited performance in handling long-text inputs. The table 12 shows the results ImageNet when IPO uses GPT-4o, which has superior long-text understanding compared to GPT-3.5 Turbo. We found that IPO using GPT-4o leads to better performance improvements other methods well considerable improvement IPO with GPT-3.5 Turbo.
E Experiments on segmentation task with IPO
In other prompt-based vision tasks, such as segmentation and detection, the design of the text prompt crucial. Our method, being task-agnostic, can be casily embedded into any vision task to optimize
16

---

Model
Base
Novel
H
CLIP
72.43
68.14
70.22
CoOp
73.20
67.43
70.20
CoCoOp
73.90
69.07
71.40
MaPLe
74.03
68.73
71.28
CoPrompt
73.97
70.87
72.39
IPO w/ GPT-3.5
74.09
69.17
71.54
IPO w/ GPT-4o
76.14
72.13
74.09
Table 12: Performance large-scale generic datasets.
Methods
pAce
mloU (S)
mloU (U)
hloU
SPNet [60]
78.0
15.6
26.1
ZS3 [61]
77.3
17.7
28.7
CaGNet [62]
80.7
78.4
26.6
39.7
SIGN [63]
75.4
28.9
41.7
Joint [64]
77.7
32.5
45.9
Zegformer [65]
86.4
63.6
73.3
Zsseg [58]
90.0
83.5
72.5
77.5
ZegCLIP [59]
94.6
91.9
77.8
84.3
Zsseg + IPO
91.2
84.7
73.2
78.6
ZegCLIP + IPO
95.3
92.7
78.7
85.1
Table 13: Experiments segmentation tasks.
the text prompt. For instance, as shown in Table we incorporated IPO into pre-trained semantic segmentation models [58, 59], where the original text prompt was "a photo of [CLASS]." Using GPT-4o as the LLM and LMM, crafted more effective text prompts specifically suited to the open-vocabulary semantic segmentation task, leading to enhanced performance and demonstrating the value of IPO optimizing text prompts for this application. We intend further investigate the use of IPO in other vision tasks in future work.
Effect of batch size
The Table 14 compares the performance GPT-3.5 turbo and GPT-4o across different batch sizes. We observed that the batch size increases to 128, GPT-3.5 turbo's performance begins to decline due to its limited capacity for handling longer input texts effectively. In contrast, GPT-4o maintains strong performance even at larger batch sizes. However, using extremely large batch sizes with GPT-4o becomes cost- prohibbiieee... Therefore, we selected a batch size 128 for our experiments. Although
Model
Batch size
Base
Novel
H
IPO w/ GPT-3.5
73.11
68.08
70.51
IPO w/ GPT-4o
4
74.32
67.98
70.55
IPO w/ GPT-3.5
16
73.42
68.43
70.82
IPO w/ GPT-40
16
74.94
70.75
72.78
IPO w/ GPT-3.5
32
73.79
68.72
71.16
IPO w/ GPT-40
32
75.01
70.93
72.91
IPO w/ GPT-3.5
64
74.09
69.17
71.54
IPO w/ GPT-40
64
75.34
71.23
73.45
IPO w/ GPT-3.5
128
73.67
68.07
70.75
IPO w/ GPT-4o
128
76.14
72.13
74.09
IPO w/ GPT-3.5
256
73.11
67.81
70.36
IPO w/ GPT-4o
256
76.81
72.73
74.71
Table 14: Effect of batch size.
17

---

History length
Base
Novel
H
n 0
69.15
75.20
72.04
n
70.25
75,43
72.74
n 10
70.95
76.21
73.49
71.23
76.41
73.72
n 20
71.76
77.00
74.29
50
71.81
76.81
74.23
n 100
72.02
76.81
74.33
Table 15: Impact prompt history length.
even larger batch sizes could potentially improve performance further, the cost considerations become a critical factor.
G Impact of prompt history length
We evaluated the effect of varying prompt history lengths on model performance, shown in Table 15. Our findings indicate that without prompt history, performance declines due to the absence of contextual information, making challenging for the LLM converge. the history length increases, performance progressively improves, with convergence observed at n=20. Although using n=100 yields the highest average performance, the extended input length significantly raises API costs. As result, we selected n=20 for our IPO, balancing performance gains with cost efficiency.
Model
Base
Novel
H
Model
1-shot
16-shots
LFA [66]
83.62
74.56
78.83
PLOT [67]
65.45
76.20
IPO
79.92
80.51
80.21
IPO
74.29
80.21
(a) Comparison with I.FA across
datasets 16-shot (b) Comparison with PLOOT on average accuracy across 11 datasets in 1-shot and 16-shot scenarios.
scenarios.
Table 16: Comparison with recent prompt learning methods
H Comparison with recent prompt learning methods
We conducted a comparative evaluation with LFA [66] PLOT [67] under the same experimental conditions, as shown in Tables 16. Our IPO method consistently outperforms both LFA and PLOT across the benchmarks.
I Detailed Prompt Optimization Prompt
In Figure 3, 4, and 6, 7, 8, show detailed inputs and outputs of different training steps in Prompt Optimization Prompt. We observed that each optimized prompt is unique, and both loss and accuracy exhibit a downward trend.
Loss and accuracy curve.
To demonstrate that IPO can indeed serve optimizer for prompt learning in rision-language models, we present the optimization process's loss and accuracy on ImageNet Figure As training steps increase, the loss consistently decreases, and the accuracy gradually improves, proving the optimization capability of IPO. Additionally, IPO not only allows for interpretable prompt generation but also reduces the risk of overfitting during training.
18

---

INFUT
Instructions
Instruction
classification large-acale recognition name largo-scale visual recagnition cataset, essential 4INS>.
<CLASS>
Balow is the image description for each image:
Image descriptions
Image image seasonings.
plate featuring thinly gamished green possibly spices
Image 1: The image showcases mouthwatoring dish consisting crispy grilldd riss with sticky souce, accompanied by Juicy apple slices side.
[.. mora image examples.... ..)
Instruction
Below are some previous prompls with their scores. The scores consist loes and accuracy. The value is equel to or greater than while the accuracy from
Prompt history
prompt
photo <CLASS>, a type food.
scores:
78855078122 accuracy: 62.35294342041166
Instruction
prompts above, lower higher accuracy than prompts <INS> above. prompt should begin with <INS> and end with <INN>. The prompt should be concise, effeclive, and generally applicable to problems above.
promot
Figure 3: An example of our Prompt Optimization Prompt with input with initial instruction on the Food101 [17] dataset.
K More generated prompts
In Table 17, we provide detailed prompts for each dataset in 16-shot format. We encourage future researchers to utilize these interpretable prompts in their own downstream tasks.
L Image description with LMM
In Table 18, provide descriptions of some training samples generated using Mini-CPM-V-22 --200 on each dataset, which serve as input for image information our Prompt Optimization Prompt. Note that we did not this image information in the 16-shot setup due the context length limitations of the language model.
M Prompt Optimization Prompt design
Our Prompt Optimization Prompt is a crucial component of our optimizer, serving to enhance the performance of the vision-language model by optimizing the prompts through the prompt LLM. Figure 2 displays first segment of the Prompt Optimization Prompt defines the role of the LLM. It introduces two essential tokens, <INS> and <CLASS>, which represent the prompt and category, respectively. The primary function of this instruction to inform the LLM of its role and the contents of the Prompt Optimization Prompt, enabling more effective understanding of the Prompt Optimization Prompt. Note that in this section, our model does not involve image information. hence the Prompt Optimization Prompt here lacks statements like "Here is description of some features of the flowers
example of our Prompt Optimization Prompt. Initially, the instruction in the
19

---

STEP
Instruction
dassification the large-scas recogntton large-ecale visual recognition
features.Herg <CLASS> represents
Below is the image description for each image:
Image descniptions
showcases food featuring sliced garrished with and other seasoningg.
Image 1: The image showcases a mouttwatering dish consisting of crispy grilled rise with sticky sauce, accompanied by juicy apple slices the side.
(.. more image examples...
Instruction
Below are some previous prompts with their scores. The scores consist of loss and acouracy. The loss value is equal to or greater than while the varies from 100.
Prompt history
prompti
photo <CLASS>, a type
scores:
loss: 0. 78923078125, accuracy: 1623344422221166
text prompt:
Identify the <CLASS> in the based on unique visual fecturas.
scores:
loss: 0.7109075, accuracy: 96.313788325622
text prompt:
Identiyy
Image and visual
scores:
loss: 03470703128 125, accuracy: 47888668440820
(... more previous promple and scores.... ...)
Instruction
Generate a prompt different from all the prompts <INSs above, with lower loss and higher accuracy than all the prompts <INS> above. The prompt should with <INS> and with <INS> should effective. apolicable problems above.
Figure 4: An example of our Prompt Optimization Prompt with input at step on the Food101 [17] dataset.
20

---

Instruction
You need to perform image ciessificatio on the large scale visual recogn tion dataset hased on visual foaturos, Hore, <CLASS> represents recogrition dataset, essential <CLASS> <INS>,
is image description Image:
Image descriptions
The showcsses plate of food fesluring thinly sliced red meal, garnished with herbe and possibly other seceorings.
imge 1: The image showcases a mouthwatoring dish consisting crispy grilldd riss with sticky sauce, accompanied by julcy apple slices on the side.
(... more image examples....
Instruction 2
Below are some provious prompts with their scoms. The scoras consist loss and acnuracy. The loss value is aqual to or graater than 0. while the acouracy varies from to 100.
Prompt history
prompt:
type
ecores:
loss: 75958078125, 25, sccuracy. 82. 32335444424440616
text prompt
Identify primary ingradient <CLASS> descrbe texturo, and
scores:
loss: 48002226662,, accuracy: 88 2992229222233775
text prompt:
Train state-of-lhe-rt deep learning advanced <CLASS> image classifiction excaptional accurady minimal
ecores:
loss: O 4296875, accuracy: 90. 19807543964333
(.. more pravious prompts scores ....
Instruction tion
Generate difforent
acouracy the KINSS>, prompt should generally applicable problems above.
<INS>
promot begin
Figure 5: An example of our Prompt Optimization Prompt with input step 100 the Food101 [17] dataset.
21

---

OUTPUT
STEP
Below the prompts created according to your guidelines:
(1) <INS> Classify image based on visual features of <CLASS>, </INS>
(2) <INS>
Identify the <CLASS> in the image and evaluate its visual features for classification, </INS>
(3) <INS> Classify the image <CLASS> food item. </IN>>
(4) <INS>
Classify the image featuring a <CLASS> precision accuracy. </INS>
(5) <INS> Identify the <CLASS> in the image based on its unique visual features. </INS>
(6) <INS> A photo of <CLASS>, displaying vibrant colors and textures. </INS>
(7) <N Identify primary ingredient in the <CLASS> describe its texture, color, and presentation. </NS>
8 <NSS> Identify the type of food shown in the image and classify accurately. </INS>
Figure 6: An example of our Prompt Optimization Prompt with output step on the Food101 [17] dataset.
OUTPUT
STEP
Below are the prompts created according to your guidelines:
(1) <INS>
Accurately classify the <CLASS> in the image using advanced visual recognition techniques. </NN>>
<INS> Accurately classify the image for <CLASS> identificaion based detailed visual analysis with precision and minimal loss. </NS>>
(3) <INS
Accurately classify the image to determine the <CLASS> based on distinctive visual cues, </INS>
(4) <INS>
Analyze the visual composition of the image to accurately daassify the <CLASS> featured with precision. <
(5) <INS>
Analyze the visual composition of the image to classify the <CLASS> accurately. </INS>
(6) <INS>
Analyze the distinct visual elements of the <CLASS> to achieve precise classification. </INS>
<INS>
Analyze the unique visual features of the <CLASS> for precise classification with superior accuracy. </INS>
<ISS> Utilize cutting-edge visual recognition algorithms categorize the image accuralely and efficiently. </NS>>
Figure 7: An example of our Prompt Optimization Prompt with output step 2 on the Food101 [17] dataset.
22

---

OUTPUT
STEP 100
Below are the prompts created according to your guidelines:
(1) <INS> Train state-of-the-de art convolutional neural network for accurate and efficient classification of <CLASS> with exceptional precision and minimal loss, </INS>
(2) <INS> Implement state-of-the-art deep leamiing algorithms for superior <CLASS> image dlassification accuracy with unrivaled predision and minimal loss..KIIS>
(3) Develop an advanced network model precise and efficient <CLASS> image diasslfication exceptional accuracy and minimal loss, </INS>
(4) <INS>
Utiiiee state-of-the-art machine learning algorithms for precise and efficient <CLASS> image classification with exceptional accuracy and negligible loss, </INS>
(5) <INS> Develop an innovative neural network model for precise and efficient <CLASS> image dasssfication with exceptional accuracy minimal loss. </INS>
(6) Employ state-of-the-ar deep learning techniques for precise efficient <CLASS> image classification unparalleled accuracy and minimal loss, <IINS>
(7) <INS>
Efficiently classify <CLASS> using outting-edge deep leamiigg techniques for unparalleled accuracy. </INS>
<NS>>
Identify the primary ingredient <CLASS> and describe color, presentation. </INS>
Figure 8: example of our Prompt Optimization Prompt with output step on Food101 [17] dataset.
Less
Artry
smooth Lass
07.
trahingg
20
20
03
300
(a) Training loss.
(b) Training accuracy.
Figure 9: Training loss and accuracy on ImageNet. Each dot represents the average loss and accuracy across up 8 generated prompts the single strip, with the shaded region indicating the standard deviation. Our findings demonstrate that IPO can effectively optimize prompt learning Languag models. Notably, the best performance was achieved at step 85.
in the image" and the subsequent description of the image. In the next section, we will discuss how to integrate image information into the Prompt Optimization Prompt.
The subsequent instruction concerns the history of the prompts and their associated scores, which include metries such as loss and accuracy. Our task follows CoOP [8], focusing on base classes with few samples. We calculate the loss and accuracy for these base classes using the generated prompts. Initially, we inform the LLM that the following details are past prompts along with their scores, and we clarify the range of values for loss and accuracy. What follows are the historical prompts and scores. This history functions similarly to episodic memory, to prevent the generation of suboptimal prompts while providing in-context information that enhances the creation of better prompts. Additionally, as history of past prompts grows, we only retain the top 20 prompts based on accuracy to avoid information overload. The selection of these top 20 prompts is determined by their accuracy scores. Moreover, we consistently include the prompt "a photo of <CLASS>" in our history at every step, this is frequently used and effective prompt within the CLIP framework [30].
23

---

Dataset
Best Prompt
ImageNet 1a
TLet's address problem, a photo a <CLASS>.
Take an awe-inspiring photograph <CLASS> beautifully captures its essence with exceptional clarity, vibrant colors, impeccable composi- tion, mesmerizing details.
Cupture a well-lit, high-resolution photo the <CLASS> with optimal focus and minimal distractions. Ensure proper composition and framing to highlight unique characteristics. Emphasize the pet's distinct traits by capturing its expression and posture accurately.
OxfordPets
StanfordCars Flowers102 102
Use higher resolution camera to capture the vehicle photo <CLASS>.. Capture a high-resolution image a <CLASS> in perfect lighting condi- tions, ensuring precise focus and clutter-free background for maximum classification accuracy and to highlight the flower's 's distinctive features. Categorize the image depicting a delicious and appetizing <CLASS> with remarkahle visual qualities.
Food101
Capture a comprehensive set of high-resolution images of an <CLASS> from various angles, ensuring optimal lighting conditions and precise focus for unparalleled accuracy in aircraft model recognition. A photo of <CLASS>, type of large-scale scene.
FGVCAireraft
SUN397
DTD
Capture image depicting the distinct pattern of <CLASS>. Construct a state-of-the-ee deep learning model on the Sentinel-2 satel-
EuroSAT
lite dataset for <CLASS> leveraging culling-odge techniques including attention mechanisms, transfer learning, ensemble learning.
Capture high-quality, well-lit image of person flawlessly demonstrat- ing the <CI. .SSS> action, ensuring impeccable visual representation for unmatched results.
UCF101
Table 17: Interpretable prompts generated by our method for each dataset in 16-shot scenarios.
The final paragraph of the instruction specifies the ultimate goal of the task: to generate improved prompts based on the aforementioned instructions and the history of past prompts. The aim is to achieve lower loss and higher accuracy in this vision-language task, and it also outlines the format for the final generated prompt. These three components constitute the entire content of the Prompt Optimization Prompt, each playing critical role for the LLM. important note that in this Prompt Optimization Prompt, we do not consider the specific content of the images; we merely use images to calculate scores, and the details of the images are overlooked. In the following section, we will utilize LMM to generate content from images and then incorporate it into the Prompt Optimization Prompt.
N More occlusion sensitivity analysis
In Tables 19-29, we present various prompts analyzed using occlusion sensitivity analysis across different models and datasets. We found that CL.IP is particularly sensitive to photos. However, CoOP and CoCoOP exhibit severe overfitting base classes, leading to poor performance base classes when some tokens are removed, but improved performance on novel classes. In contrast, our optimized prompts show performance degradation to varying degrees when certain words or phrases are removed, indicating that the key words or phrases in our generated prompts are essential.
24

---

Datasets
Images
Text description
The cheetah's fur has spots and stripes, its eyes black with white pupils. It looking straight into the camera while standing against a clear blue sky background.
Caltech101
The image prominently features a large white and red airplane with the Turkish flag logo, flying against backdrop of blue sky dotted with clouds.
The image features a black Audi TT parked outside the dealership, with its silver rims and elegant design.
StanfordCars
The car in the image is 2016 Acura TSX, distinguished by its sleek design and distinctive tail lights.
The flowers have a combination of pink and white colors with distinct leaves, creating an attractive appearance.
Flowers102
The flowers in the image are large, have yellow center with dark spots and petals that vary from pink red. They appear vibrant due their color contrast against natural backgrounds like leaves or rocks trees behind them.
The cat the image is light-colored, possibly an orange tabby with big yellow eyes and white whiskers.
OxfordPets
The image features a small white puppy with brown spots, which is likely to be an American Pit Bull Terrier breed.
The food in the image is delicious pastry with meat and cheese filling, covered by caramelized topping.
1oo
The image features a colorful and diverse salad with various ingredients such as onions, corn kernels, lettuce leaves (cabbage). peanuts or pumpkin seeds.
The aircraft in the image a large commercial airplane, likely used for passenger transportation. It has orange and white color scheme with prominent windows along its body to provide natural light inside during flights.
FGVCAircraft
The aircraft in the image is a Western Airlines Boeing 737, identified by its distinctive tail fin with "Western" written it. It has various markings including numbers and letters that are part of their registration or identification system used for aviation purposes.
The image showcases a commercial airplane cabin with rows of blue and white chairs, each equipped with cup holders for passengers' convenience.
SUN397
The image captures the grandeur of monument with an elaborate arch, surrounded by fountains and illuminated buildings at night.
The image showcases close-up up view the texture on an object with distinct stripes. buttons and stitching details.
DTD
The leaf has yellowish background with darker spots and irregular black marks, indicating possihle discase damage.
The fish, characterized by its light yellow body and distinct blue fins with a black spot on the tail fin's 's edge.
ImageNet
The image features a white chicken with red comb and orange legs, standing on wooden planks surrounded by straw or grass.
The image features a black car with four-wheeler design, possibly indicating that it is automobile.
EuroSAT
The image captures various cars, each with distinct features such as their color and shape. For instance, one car red in color while another has unique design that stands out from the rest of them.
A bahy in yellow shirt and blue jeans actively crawling on the tiled floor, while an adult wearing red pants stands nearby.
UCF101
A female gymnast is skillfully performing on a balance beam. showcasing her athletic abilities as she navigates the challenging obstacle.
Table 18: Image descriptions generated with Mini-CPM-V-2.0 [43].
25

---

Models
Prompts
Base
Novel
H
photo <CLASSS.
72.43
68.14
70.22
<CLASS>.
69.91
64.72
67.21
<CLASS>,
70.72
66.44
68.51
photo <CLASS>.
68.15
62.32
65.10
of <CLAS>>>>
67.84
64.93
66.35
CLIP [30]
photo <CLASS>.
69.91
64.15
66.91
photo of <CLASS>.
71.24
65.67
68.34
of a <CLASS>. photo of <CLASS>. photo of a <CLASS>.
69.37
66.49
67.90
70.99
64.26
67.46
72.64
68.02
70.25
Token 1, 3,4
73.20
67.43
70.20
None
69.91
64.72
67.21
Token
66.39
62.88
64.59
Token 2 Token
66.77
62.18
64.39
74.06
68.61
71.23
Token 4
66.96
64.07
65,48
CoOP [8]
Token 1,
67.15
62.93
64.97
Token 3,
73.92
69.18
71.47
Token 1,
68.99
65.60
67.25
Token 1, 2,
73.15
66.54
69.69
Token 2,
69.84
64.14
66.87
Token 1, 4
73.52
68.64
71.00
Token 2, 3,
73.71
68.08
70.78
Token 1, 4
73.90
69.07
71.40
None
69.91
64.72
67.21
Token Token
74.01
67.85
70.80
74.00
68.25
71.01
Token 3
73.52
68.58
70.96
CoCoOP [11]
Token
73.92
68.23
70.96
Token 1,
73.29
67.45
70.25
Token 3,
73.34
67.68
70.40
Token 1,
73.66
67.68
70.54
Token 1, 2,
71.45
65.57
68.38
Token 2, 4
72.95
65.57
69.06
Token 3,
72.52
66.87
69.58
Token 2, 3,
72.52
66.77 74
69.51
Take high-quality photo of a <CLASS>.
74.09
69.17
71.54
<CLASS>.
69.92
64.71
67.21
Take photo <CLASSS>
72.43
68.43
70.37
High-quality photo of <CLASS>.
73.64
68.24
70.84
Photo <CLASS>.
72.65
68.08
70.29
IPO
Take high-quality photo of a <CLASS>. Quality photo of <CLASS>. High-quality <CLASS>, Take photo the <CLASS>. A photo of <CLASS>.
74.08
69.12
71.51
73.69
68.23
70.85
71.71
66.47
68.99
71.42
67.72
69.52
72.48
68.13
70.24
Table 19: Comparison of various prompts with occlusion sensitivity analysis across different models the ImageNet dataset.
26

---

Models
Prompts
Base
Novel
photo of <CLASSs.
96.84
94.00
95.40
<CLASS>.
90.47
92.19
91.32
<CLASS>>
91.08
93.45
92.25
photo <CLASS>.
89.67
90.62
90.14
of <CLASS>,
91.38
92.72
92.05
CLIP [30]
photo <CLASS>.
92.02
92.69
92.35
photo of <CLASS>.
91.93
93.32
92.62
of a <CLASS>. photo of <CLASS>. photo a <CLASS>.
96.14
94.72
95.42
91.32
94.45
92.86
97.05
94.36
95.69
Token 1, 2, 3,
90.63
85.20
87.83
None
90.47
92.19
91.32
Token
89.29
90.41
89.85
Token 2
89.71
90.83
90.27
Token
87.18
92.76
89.88
Token 4
91.75
91.45
91.60
CoOP [8]
Token 1,
89.86
92.97
91.39
Token 3,
89.97
92.26
91.10
Token 1,
92.84
92.04
92.44
Token 1, 2,
88.13
92.26
90.15
Token 2,
94.34
93.02
93.68
Token 1,
88.56
92.81
90.64
Token 2, 3,4
89.52
92.97
91.21
Token 2,
96.37
93.13
94.72
None
90.47
92.19
91.32
Token
96.36
93.38
94.85
Token
96.02
93.12
94.55
Token
96.97
93.38
95.14
CoCoOP [11]
Token 4
96.15
93.24
94.67
Token 1,
96.42
93.95
95.17
Token 3,
97.56
93.95
95.72
Token 1,
97.47
93.41
95.40
Token 1, 2,
95.54
92.87
94.19
Token 1,
96.63
93.24
94.90
Token 1,
97.41
93.24
95.28
Token 2, 3,
97.41
93.12
95.22
Categorize the <CLASS> shown in the image.
96.53
95.39
95.95
<CLASS>.
90.45
92.16
91.30
Categorize <CLASS>.
91.92
93.78
92.84
Categorize the <CLASS>.
95.44
95.05
95.24
<CLASS> shown in the image.
97.08
94.15
95.59
IPO
Shown in the image: <CLASS>, Categorize the <CLASS> in the image.
93.54
93.76
93.65
96.64
95.74
96.19
The <CLASS> shown the image.
97.05
95.06
96.04
The <CLASS> in the image.
93.54
94.32
93.93
Categorize the <CLASS> shown.
95.37
95.09
95.23
Image of a <CLASS>.
97.26
94.84
96.03
Table 20: Comparison various prompts with occlusion sensitivity analysis across different models on the 1altech101 dataset.
27

---

Models
Prompts
Base
Novel
H
photo <CLASS>.
89.42
96.81
92.97
<CLASS>.
88,30
88.69
a a<CLASS>.
87.77
90.76
photo <CLASS>.
84.69
87.57
<CLASS>
86.98
88.53
87.75
CLIP [30]
photoo <CLASS>.
87.19
91.16
89.13
photo <CLASS>.
88,62
90.38
89,49
of <CLASS>>
85.38
92.84
88,95
photo of <CLASS>.
88.94
91.39
90.15
photo <CLASS>.
89.90
96.36
93.02
Token 2,4
93.73
96.23
94.96
None
88,30
89.09
88.69
Token
81.34
86.80
83.98
Token 2
81.34
89.26
85.12
Token 3
93.83
94.80
94.31
Token 4
85.11
90.16
87.56
CoOP [8]
Token 1,
81.77
85.79
83.73
Token 3,
91.97
96.31
94.09
Token 1,
87.99
91.72
89.82
Token 1, 3
93.62
97.37
95.46
Token 1,
88.46
88.09
88.27
Token 1, 3.4
91.92
97.32
94.54
Token 2, 3.4
93.35
97.37
95.32
Token I, 2, 3,
93.47
96.27
94.85
None
88.30
89.09
88.69
Token
94.26
95.64
94.94
Token 2 Token
94.42
95.64
95.03
94.84
96.14
95.49
CoCoOP [1]]
Token 4
94.05
96.09
95.06
Token
94.15
95.75
94,94
Token
93.83
97.09
95.43
Token 1,
94.05
96.48
95.25
Token 1,
93.67
95.30
94.48
Token 1, 4
93.14
95.92
94.51
Token 3.
93.35
96.59
94.94
Token 3,
93.25
96.70
94.94
Take well-composed photo a <CLASS> with optimal lighting. focus. and minimal distractions. Cupture pet's unique characteristics, including expression and posture, ensure a clear and distinct image.
94.48
97.93
96.43
<CLASS>.
88,30
89.09
88.69
Take a photo of <CLASS>. Well-composed photo of <CLASS>.
90.06
96.59
93.21
89.90
96.70
93.18
Photo a <CLASS> with optimal lighting.
89.37
96.81
92.94
<CLASS> with optimal lighting. focus, and minimal distractions.
89.85
92.62
91.21
Capture the <CLASS>'s unique characteristics. Including expression
87.77
89.60
88.68
posture the <CLASS>.
86,98
87.97
IPO
Ensure a clear distinct image the <CLASS>. Unique characteristics the <CLASS>. Capture the expression of <CLASS>. Capture the posture the <CLASS>. Expression of <CLASS>. Posture <CLASS>.
81.77
84.00
89.10
87.90
88.20
89.11
87.83
91.11
89.44
87.56
90.32
88,92
89.79
92.67
91.21
Focus minimal distractions <CLASS>. Clear distinct image the <CLASS>.
90.27
96.31
93.19
83.57
85.91
84.72
Table 21: Comparison of various prompts with occlusion sensitivity analysis across different models on the OxfordPets dataset.
28

---

Models
Prompts
Base
Novel
photo of a <CIASSS>
63.37
74.89
68.65
<CLASS>.
62.72
73.58
67.72
<CLASS>.
61.37
72.82
66.61
photo <CLASS>. of <CLASS>,
61.79
73.78
67.25
62.04
73.98
67.49
CLIP [30]
photo <CLASS>>
62.57
73.98
67.80
photo of <CLASS>.
63.09
74.75
68.43
of a <CLASS>,
59.25
71.82
64,93
photo of <CLASS>. photo of <CLASS>.
63.82
75.41
69.13
63.04
74.45
68.27
Token 2,4
61.80
68.33
64.90
None
62.72
73.58
67.72
Token
61.42
72.05
66.31
Token
58.77
69.74
63.79
Token
62.37
68.31
65.21
Token 4
60.24
70.19
64.84
CoOP [8]
Token 1,
61.79
73.95
67.33
Token
59.57
68.46
63.71
Token 1,4
60.14
71.97
65.53
Token 2,
63.37
69.82
66.44
Token 2.
61.12
74.00
66.95
Token 3.4
60.79
70.29
65.20
Token 3.4
57.95
65.66
61.56
Token 1, 2, 3,
65.27
73.73
69.24
None
62.72
73.58
67.72
Token
65.22
75.27
69.89
Token
65.32
74.81
69.74
Token
65.09
74.05
69.28
CoCoOP [11]
Token
65.54
74.50
69.73
Token 2
64.57
74.94
69.37
Token 3,
64.67
73.93
68.99
Token 1,
64.84
74.85
69.49
Token 1, 2,
64.07
74.52
68.90
Token 1,
64.74
74.10
69.10
Token 3,
63.17
74.18
68.23
Token 3,4
63.29
74.18
68.30
Descrihe the distinguishing characteristics of the <CLASS> in the image.
63.83
75.45
69.16
<CLASS>.
62.72
73.58
67.72
Describe the <CLASS>
62.67
75.34
68.42
Distinguishing characteristics of the <CLASS>. Characteristics of the <CLASS> in the image.
63.52
74.40
68.53
63.72
74.35
68.63
Describe the <CLASS> in the image.
62.97
74.82
68.39
IPO
The <CLASS> in the image.
63.97
74.94
69.02
Describe distinguishing characteristics of the <CLASS> In the image. the <CLASS>>
63.97
74.15
68.68
63.67
75.19
68.95
The <CLASS>'s 's distinguishing characteristics.
61.34
72.39
66.41
The distinguishing characteristics of the <CLASS> in the image.
64.07
74.72
68.99
Table 22: Comparison various prompts with occlusion sensitivity analysis different models the
StanfordCars datasct.
29

---

Models
Prompts
Base
Novel
H
photo of <CLASS>.
69.34
76.72
72.84
<CLASS>.
61.75
75.83
68 07
<CLASS>.
63.04
75.02
68.51
photo <CLASS>.
64.72
76.74
70.22
of <CLASS>.
63.02
76.17
68.97
CLIP [30]
photo <CLASS>.
66.52
77.25
71.48
photo of <CLASS> of <CLASS>
61.33
77.12
68.32
63.37
75.91
69.08
photo of <CLASS>. photo <CLASS>.
59.15
77.25
70.00
69.16
76.72
72.74
Token I, 2, 3,
71.47
72.47
71.97
None
61.75
75.83
68.07
Token
76.02
73.38
74.68
Token
70.07
74.26
72.10
Token
69.62
72.88
71.21
Token
70.85
67.75
69.27
CoOP [8]
Token 1,
75.04
74.23
74.63
Token 3,
70.13
66.75
68.40
Token 1.4
75.92
68.81
72.20
Token 2,
72.55
74.04
73.29
Token 1,
74.82
70.52
72.61
Token 3,
76.25
66.15
70.84
Token 2, 3,
70.91
67.93
Token I, 3,
73.67
75.50
74.57
None
61.75
75.83
68.07
Token
75.83
76.92
76.37
Token
76.75
76.98
76.86
Token
76.74
77.27
77.00
CoCoOP [11]
Token
76.51
77.64
77.07
Token 1,
76.02
77.23
76.62
Taken 3,
76.07
78.75
77.39
Token 1,4
76.49
78.22
77.35
Token 2.
76.51
75.53
76.02
Token 1, 2,
76.04
77.92
76.97
Token 1,
75.53
78.37
76.92
Token 2, 3,
75.25
77.71
76.46
Identify the unique visual features of the <CLASS> flower accurately.
74.17
79.65
76.81
<CLAŠS>
61.75
75.83
68.07
Identify-<CLASS <CLASS>.
69.53
77.52
73.31
Identify the <CL.SSS>
68.88
76.35
72.42
Identify the unique <CLASS>. Identify unique visual features <CLASS>. Visual features of the <CLASS>.
67.22
77.73
72.09
63.28
77.21
69.55
IPO
64.42
74.64
69.15
The unique visual features <CLASS> flower. Features the <CLASS> flower accurately. Visual features of the <CLASS> flower. Identify the unique visual features of the <CLASS> accuratoly,
66.42
76.67
72.03
77.82
74.81
71.93
78.55
75.09
65.67
77.35
71.03
Identify the unique visual features of the <CLASS> flower.
73.82
79.11
76.37
The unique visual features of the <CLASS> flower accurately.
72.67
78.72
75.57
Table 23: Comparison various prompts with occlusion sensitivity analysis across different models on the Flowers 102 dataset.
30

---

Models
Prompls
Base
Novel
II
<CLASS>.
89.44
90.68
90.06
<CLASS>.
89.25
89.44
89.34
<CLASS>.
88.48
89,12
88,80
photo <CLASS>.
89.39
88.49
88
<CLASS>>>
89.51
90.01
89.76
CLIP [30]
photo <CLASS>.
88
photo of <CLASS>.
89.38
90.00
89.69
of <CLASS>.
89.31
90.17
89.74
photo of <CLASS>.
89.32
88.15
88.73
photo of a <CLASS>.
89.17
90.59
89.87
Token 2. 3.4
87
88.03
87.96
None
89
89, 44
89 34
Token
86.29
87.44
Token
88.99
87.99
Token
86.47
88.90
87.67
Token
89,27
88.95
89.11
CoOP [8]
Token 1.2
88.00
87.45
87.72
Token 3,
87.55
89.17
88.35
Token 1,4
89.12
89_36
Token 2.
86.38
88.91
87.63
Token 2,
88.84
89.58
89.21
Token 1,
87.73
89.67
88.69
Token 3,
87.41
89.41
88.40
Token 2. 4
88.73
89.60
89.16
None
89.25
89.44
89.
Token
88.77
89.55
89.16
Token
89.31
90.45
89.87
Token
89.57
90.83
90,20
CaCoOP [11]
Token
89.01
90.40
89.70
Token 1,2
88.63
89.55
89.09
Token 3,
89.08
90.68
89.87
Taken 1,4
88.49
89.57
89.03
Token 2,
88.40
89,86
89,12
Token 1, 2.
88.24
89.59
88.91
Token 1,
88.37
90.15
89.25
Token 2, 3,
88.53
90,28
89.40
Identify the primary ingredient in the <CLASS> and describe its texture, color, and presentation.
89.78
91.59
90.67
<CLASS>.
89.25
89, .44
89.
Identify <CLASS>.
Primary ingredient in the <CLASS>.
80.55
82.15
81.34
Identify the primary ingredient the <CLASS>>
88.99
90.44
89.71
Describe the <CLASS>'s texture.
89.32
90.39
89.85
Describe the <CLASS>'s color.
87.75
88.82
88 28
IPO
Describe <CLAS5>'s presentation.
89.07
88.77
Texture <CLASS>> Color the <CLASS>. Presentation the <CLASS>.
88.86
89.99
87.90
90.31
89.09
88.74
90,98
89,85
The <CLASS>'s primary ingredient. Identify the primary ingredient in the <CLASS> and describe its texture. Identify the primary ingredient
87 87.44
87.89
87.66
89.35
91.06
90.20
<CLASS> describe its color.
90.60
89,57
Identify primary ingredient <CLASS> and describe its presentation.
89.77
91.51
90.63
Table 24: Comparison various prompts with occlusion sensitivity analysis across different models on the Food101 dataset.
31

---

Models
Prompts
Base
Novel
H
a photo <CLASS>. <CLASS>.
27.73
33.17
30.21
25.81
31.43
28.34
a <CLASS>.
25.81
30.89
photo <CLASS>.
22.51
30.77
<CLASS>
24.01
30.95
27.04
CLIP [30]
a photo <CLASS>.
26.71
31.37
28.85
photo of <CLASS>.
25.03
31.97
28.08
of a <CLASS>
24.55
30.41
27.17
pholo <CLASS> photo of a <CLASS>.
26.59
33.11
29.49
27.01
33.23
29.80
Token 2, 3.4
27.77
27.60
27.68
None
25.81
31.43
28.34
Token
22.27
26.69
24.28
Token
13.21
19.38
15.71
Token
22.45
26.81
24.44
Token
23.41
28.31
25.63
CoOP [8]
Token 1,
12.91
12.90
Token 3,
27.61
33.11
Token 1.
23.65
30.17
Token 1, 2,
17.95
18.72
Token 1, 2,
18.49
20.82
19.59
Token 1, 3,
27.01
35.27
30.59
Token 2,
27.67
28.73
28.19
Token 2, 3,
29.77
31.23
30.28
None
25.81
31.43
28.34
Token
25.93
33.17
29.11
Token
26.59
32.33
29.18
Token
25.75
30.65
27.99
CoCoOP [11]
Token
29.23
33.17
31.08
Token 1,
28.45
32.87
30.50
Token 3,
30.43
33.65
31.96
Token 1,4
31.33
35.21
33.16
Token 1, 2,
28.75
31.49
30.06
Token 1, 2,
31.33
35.81
33.42
Token 1, 3,
30.19
35.51
32.63
Token 3.
30.13
35.45
32.57
Capture comprehensive range of well-lit, high-resolution images of an <CLASS> from various angles. meticulously showcasing its specific design features with perfect clarity and precision for unparalleled accuracy in aircraft.
31.43
36.32
33.70
<CLASS>.
25.81
31.43
28.34
Capture an <CLASS>>
26.53
31.79
28.92
Range of images of an <CLASS>>
28.81
32.93
30.73
Well-lit images of <CLASS>>
26.53
32.75
29.31
High-resolution images of <CLASS>.
28.99
33.11
30.91
Images of <CLASS> from various angles.
28.21
34.13
30.89
IPO
Comprehensive range of images of an <CLASS>,
28.21
32.75
30.31
Meticulously showcase the <CLASS>'s design features.
25.03
30.83
27.63
Specific design features of <CLASS>. Perfect clarity precision the <CLASS>. Capture well-lit images Capture high resolution images <CLASS>. Capture images an <CLASS> from various angles.
28.27
34.01
30.88
23.77
31.31
27.02
<CLASS>.
27.31
28.91
30.07
31.38
27.79
32.63
30.02
Showcase the <CLASS> with clarity and precision.
25.27
22.26
23.67
Unparalleled accuracy of <CLASS> aircraft.
28.69
33.05
30.72
Table 25: Comparison of various prompts with occlusion sensitivity analysis across different models on the FGVCAireraft dataset.
32

---

Models
Prompts
Base
Novel
photo of <CLASSs.
69.36
75.35
72.23
<CLASS>.
65.91
72.74
69.16
<CLASS>>
68.75
66.65
67.68
photo <CLASS>.
62.88
70.42
66.44
of <CLASS>,
65.27
69.95
67.53
CLIP [30]
photo <CLASS>.
65.01
75.73
69.96
photo of <CLASS>.
70.47
72.75
71.59
of a <CLASS>. photo of <CLASS>. photo a <CLASS>.
69.19
75.47
72.19
69.76
75.55
72.54
69.52
70.62
70.07
Token 1, 2, 3,
71.47
72.47
71.97
None
65.91
72.74
69.16
Token
61.39
65.98
63.60
Token 2
61.14
66.43
63.68
Token
69.87
71.54
70.70
Token 4
63.72
66.22
64.95
CoOP [8]
Token 1,
64.76
68.34
66.50
Token 3,
69.51
71.57
70.52
Token 1,
65.23
68.66
66.90
Token 2,
69.98
72.66
71.29
Token 2,
66.27
72.07
69.05
Token 1,
69.64
71.45
70.53
Token 2, 3,4
69.62
72.22
70.90
Token 2, 4
73.67
75.50
74.57
None
65.91
72.74
69.16
Token Token
70.85
75.63
73.16
70.52
75.25
72.81
Token
69.64
75.48
72.44
CoCoOP [11]
Token 4
71.64
75.05
73.31
Token 1,
72.55
72.65
72.60
Token 3,
73.56
72.61
73.08
Token 1,
73.18
72.72
72.95
Token 1, 2,
72.53
69.01
70.73
Token 2,
73.26
68.87
71.00
Token 1,
73.46
68.75
71.03
Token 3,4
73.24
68.02
70.53
A photo a <CLASS>, a type of large-scule scale scene.
72.25
77.53
74.80
<CLASS>.
65.91
72.74
69.16
A photo <CLASS>.
69.36
75.35
72.23
Photo of <CLASS>.
69.52
70.62
70.07
A <CLASS>, a type of large-scale scene.
71.51
77.02
74.16
IPO
Large-scale scene of <CLASS>.
71.03
76.63
73.72
Type of large-scale scene: <CLASS>.
69.97
76.08
72.90
A photo the <CLASS>. The <CLASS>, large-scale scene.
71.74
76.71
74.14
69.83
74.83
72.24
A type of large-scale scene: <CLASS>.
70.92
76.64
73.67
Scene of <CLASS>.
70.55
74.95
72.68
Table 26: Comparison various prompts with occlusion sensitivity analysis across different models on the SUN397 dataset.
33

---

Models
Prompts
Base
Novel
photo of <CLASSs.
54.63
59.18
56.81
<CLASS>.
53.70
59.18
56.31
<CLASS>>
53.70
60.27
56,80
photo <CLASS>.
43.63
44.93
44.27
of <CLASS>,
55.79
58.82
57.26
CLIP [30]
photo <CLASS>.
43.17
41.43
42.28
photo of <CLASS>.
53.47
52.05
52.75
of a <CLASS>. photo of <CLASS>. photo a <CLASS>.
55.21
60.75
57.85
53.36
51.69
52.51
54.63
59.78
57.09
Token 1, 2, 3,
60.80
47.53
53.35
None
53.70
59.18
56.31
Token
50.93
52.66
51.78
Token 2
49.88
48.07
48.96
Token 3
59.03
44.93
51.02
Token 4
54.05
55.43
54.73
CoOP [8]
Token 1,
46.53
42.03
44.17
Token 3,
55.90
46.74
50.91
Token 1,
54.17
57.00
55.55
Token 1, 2,
54.51
48.67
51.42
Token 2,
51.62
44.69
47.91
Token 1,3,
55.67
46.74
50.82
Token 2, 3,4
55.32
46.38
50.46
Token 2,
58.70
52.70
55.54
None
53.70
59.18
56.31
Token
57.41
51.45
54.27
Token
57.64
50.60
53.89
Token
59.03
53.38
56.06
CoCoOP [11]
Token 4
57.87
52.66
55.14
Token 1,
55.44
52.05
53.69
Token 3,
57.06
54.83
55.92
Token 1,
56.37
53.62
54.96
Token 2,
56.02
49.28
52.43
Token 1,
55.44
52.29
53.82
Token 1,
57.29
52.90
55.01
Token 2, 3,
57.18
52.78
54.89
Classify the intricate <CLASS> texture.
55.45
62.47
58.75
<CLASS>.
53.70
59.18
56.31
Classify <CLASS>.
53.01
57.97
55.38
The intricate <CLASS>.
54.17
58.70
56.34
Intricate <CLASS> texture.
52.66
55.43
54.01
Classify the <CLASS>, Classify the texture of the <CLASS>.
55.56
59.66
57.54
IPO
53.70
62.44
57.74
<CLASS> texture.
53.12
60.51
56.57
Texture of the <CLASS>.
52.43
63.77
57.55
The <CLASS>'s intricate texture. Classify intricate <CLASS> texture.
52.31
60.63
56.16
54.40
58.70
56.47
Table 27: Comparison various prompts with occlusion sensitivity analysis across different models on the DTD dataset.
34

---

Models
Prompts
Base
Novel
photo of <CLASS>
50.26
69.90
58.47
<CLASS>.
47.19
66.49
55.20
<CLASS>
53.57
70.64
60.93
photo <CLASS>. of <CLASS>,
46.26
53.77
49.73
54.14
74.23
62.61
CLIP [30]
photo <CLASS>
48.14
62.95
54.56
photo of <CLASS>>
48.02
67.92
56.26
of a <CLASS>.
62.43
71.31
66.58
photo of <CLASS>. photo of <CLASS>.
49.62
67.97
57.36
48.43
67.38
56.35
Token 2, 3,4
69.13
50.33
58.25
None
47.19
66.49
55.20
Token
47.95
67.85
56.19
Token
58.83
66.03
62.22
Token
74.88
46.00
69.55
Token 4
$7.30
71.59
63.65
CoOP [8]
Token 1,
$7.36
67.23
61.90
Token 4
76.57
46.23
57.65
Token 1,
54.76
68.77
60.97
Token 1, 2,
67.14
40.36
50.41
Token 2.
61.67
65.51
63.53
Token 3.
75.40
45.31
56.60
Token 3.4
79.14
44.77
57.19
Token 2. 3.4
71.13
62.87
66.75
None
47.19
66.49
55.20
Token
74.62
51.38
60.86
Token
72.83
50.15
59.40
Token
76.05
48.85
59.49
Token
71.21
50.10
58.82
CoCoOP [11]
Token 2
72.10
47.95
57.60
Token
72.55
58.23
64.61
Token 1, 4
76.26
58.92
66.48
Token 1, 2,
74.67
40.21
52.27
Token 1, 2,
75.36
51.28
61.03
Token 3,
70.40
51.08
59.20
Token 2, 3,
66.83
47.54
55.56
Analyze the <CLASS> vehicles in the satellite image with state-of-the-art algorithms for precise classification and optimal efficiency.
64.97
82.13
72.54
<CLASS>.
47.19
66.49
55.20
Analyze <CLASS> vehicles.
48.95
74.46
59.07
Analyze the <CLASS>.
$1.60
75.64
61.35
<CLASS> vehicles in the satellite image.
51.55
80.15
62.74
IPO
Vehicles in the satellite image: <CLASS>.
58.26
69.67
63.46
Sfate-of-the-as the-art algorithms for <CLASS> classification. Precise classification of <CLASS> vehicles.
$4.60
74.85
63.14
61.71
76.59
68.35
Optimal efficiency in <CLASS> analysis. Analyze the <CLASS> vehicles in the satellite image.
56.12
79.26
65.71
58.17
78.82
66.94
Use state-of of-teo---h-arr algorithms for <CLASS> classification.
58.17
75.54
65.73
Classification and efficiency of <CLASS> vehicles.
63.79
76.46
69.55
Table 28: Comparison various prompts with occlusion sensitivity analysis across different models the
Eu dataset.
35

---

Models
Prompts
Novel
photo of <CLASS>.
68.1 15
75.07
71.44
<CLASS>.
67.68
72.47
67.00
<CLASS>
68.25
68.82
photo <CLASS>. of <CLASS>.
65.05
68.74
66.84
68.87
72.31
70.55
CLIP [30]
photo <CLASS>.
64.74
71.98
68.17
photo of <CLASS>
69.70
77.23
73.27
<f
68.20
ahhoto <CLASS>.
68.61
76.10
72.16
of <CLASS>.
68.10
75.01
71.39
Token 3,4
72.50
63.57
67.74
None
67.68
72.47
67.00
Token
65.21
71.55
68.23
Token
64.32
70.90
67.45
Token
74.20
71.39
72.77
Taken
65.15
67.78
CoOP [8]
Token 1,
64.63
73.12
68.61
Token 3,
72.75
69.17
70,91
Token 1.4
66.91
70,96
68.88
Token 1,
74.61
73.12
73.86
Tuken 1,
Token 1.3.4
73.11
69.33
71.17
Token 3.
74.20
69.01
71.51
Token 2.
74.73
72.80
73.75
None
67.68
72.47
67.00
Tuken
72.60
72.53
72.56
Token
71.92
71.82
71.87
Token
71.20
70.96
71.08
Token
71.77
70.52
71.14
CoCoOP [11]
Token 1,
73.73
72.42
73.07
Token 3,
74.72
71.66
73.16
Token 1.4
75.54
72.58
74.03
Token 1,
74.61
72.20
73.39
Token 1,
75.08
71.12
73.05
Token 3.
76.27
72.58
74.38
Token 3.
76.01
72.28
74.10
Capture high-quality, well-lit image person flawlessly demonstrating the <CLASS> action with impeceable visual representation to achieve unmatched.
72.43
79.35
75.73
<CLASS>.
67.68
72.47
67.00
Capture <CLASS>.
67.32
71.23
69.22
High-quality image of <CLASS>.
71.41
76.96
74.08
Well-lit image <CLASS>,
77.56
73.30
Image of <CLASSS>
70.17
75.5
72.74
IPO
Person demonstrating the <CLASS>. Demonstrating the <CLASS> action.
72.60
76.31
74.41
71.41
74.96
73.14
Impeccable visual representation Capture person demonstrating the <CLASS>.
<CLASSS>
70.63
76.58
73.48
70.53
76.47
Flawlessly demonstrating the <CLASS>. Visual representation of the <CLASS> Achieve unmatched representation of <CLASS>
70.48
78.26
70.84
75.72
73.20
69.03
71.55
70.27
Table 29: Comparison of various prompts with occlusion sensitivity analysis different models the
UCF101 datasct.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of this paper claimed in the abstract. Detailed information can be found in the fourth paragraph of the introduction section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide "limitation" subsection in the conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
37

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We provide all experimental details in the experiment section. The detailed input and output of Prompt Optimization Prompt of our method can be found in Appendix and supplemental materials.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
38

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all experimental details the experiment section. The detailed Prompt Optimization Prompt and Python codes of our method can be found in Appendix and supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ Bww.dee/Coeeeuumnnineeme
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
ctaiss.ccpubllcc// geticc/ggiees.eee CooeeSoomiiseeeeeee wwwssnoliccyyiiioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We follow the standard experimental setup in IPO, where the data splits, is set as the same as previous works CoOP [8] and CoCoOP [11]. Detailed information can be found in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following the standard experimental setup, we repeat each experiment over random seeds and report the mean of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
39

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computing resources experiments 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /ttps://neurrs ctppiippp.ccppppapa wttpccccpppcccccccccccccccccccct http:p/bli.tttiiii.....eeeeeeeee
Answer:
[Yes]
Justification:
We reviewed and followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the potential broader impacts in the conclusion section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
40

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The data and models pose no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers that produced the code package and datasets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. paperswithcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
41

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of the datasetsscodee/modll are provided in the supplemental materials. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
42


Reviews:
[{"review_id": "7X7ogOcVil", "reviewer": "Reviewer_U899", "summary": "The paper introduces a method named IPO for VLMs which uses LLMs to dynamically generate and refine text prompts. The method is to improve the accuracy and interpretability of prompts. Experiments show that it can address issues like overfitting and lack of human comprehension in traditional gradient descent-based methods.", "questions": "Will this method be generalized to other vision related tasks?", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The prompts generated using IPO enhance the performance of vision-language models across various vision task related datasets.\nThe process of generating prompts are interpretable which brings better transparency and controllability.", "weaknesses": "LMM seems contribute little to th final performance. Is the reason that used LMM is not strong enough to generate high-quality image caption? Also for different LLMs, the performance gap is not obvious and the author did not test on GPT4 or stronger models. \nThe overall performance is not much better than optimized methods like CLIP.  \nCould the authors provide more examples to prove the iterpretability advantage of this method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Impact of LMM:**\n\nWe thank Reviewer U899 for the insightful suggestion. Indeed, when we replaced the 2.8B parameter MiniCPM-V-2 LMM with the higher capacity GPT-4o (estimated 500B~1T parameters), we observed a performance improvement for 10 out of 11 datasets. On average, the performance across 11 datasets improved: Base: +1.66%, Novel: +1.89%, and H: +1.77%.\n\n| LMM          | Params          | Base   | Novel  | H      | \n|--------------|-----------------|--------|--------|--------|\n| CLIP         | -               | 69.34  | 74.22  | 71.70  |   \n| w/o LMM      | -               | 71.12  | 76.03  | 73.49  |  \n| w/ MiniCPM-V-2 | 2.8B          | 71.76  | 77.00  | 74.29  |  \n| w/ GPT-4o      | 500B~1T        | 72.78  | 77.92  | 75.26  |  \n\n**(2) Impact of LLM:**\n\nWe observed a similar positive impact when upgrading the LLM capacity. To demonstrate that using a stronger LLM, such as GPT-4, can generate more effective prompts for our model, we conducted further experiments utilizing both GPT-4 and GPT-4o. Specifically, when upgrading the LLM to GPT-4o and pairing it with the GPT-4o LMM, the overall H-score increased by 1.77% compared to the original results with GPT-3.5-turbo and MiniCPM-V-2. This improvement highlights the benefit of using larger models for enhancing task generalization. \n\n\n\n| LLM            | Params         | LMM        | Params         | Base   | Novel  | H      | \n|----------------|----------------|------------|----------------|--------|--------|--------|\n| GPT-3.5-turbo  | 175B           | MiniCPM-V-2 | 2.8B          | 71.76  | 77.00  | 74.29  |  \n| GPT-4          | 500B~1T        | MiniCPM-V-2 | 2.8B          | 72.67  | 77.62  | 75.06  |\n| GPT-4o         | 500B~1T        | MiniCPM-V-2 | 2.8B          | 72.91  | 78.13  | 75.42  |\n| GPT-3.5-turbo  | 175B           | GPT-4o      | 500B~1T        | 72.78  | 77.92  | 75.26  |  \n| GPT-4          | 500B~1T        | GPT-4o      | 500B~1T        | 72.93  | 78.01  | 75.38  |\n| GPT-4o         | 500B~1T        | GPT-4o      | 500B~1T        | 73.41  | 78.93  | 76.06  |\n\n\n\nThese improvements are reflected in the quality of prompts generated by the upgraded models.  For example, on the DTD dataset, the prompt generated by GPT-3.5 turbo changes from \"Classify the intricate <CLASS> texture\" to \"Analyze and classify the detailed <CLASS> texture in this image, considering its unique patterns and variations.\" Similarly, on the ImageNet dataset, the prompt generated by GPT-3.5 turbo changes from \"Take a high-quality photo of a <CLASS>\" to \"Capture a sharp, high-resolution photo of a <CLASS> with clear details and vibrant colors.\"\n\n\n**(3) Other Related Tasks:**\n\nIn other prompt-based vision tasks, such as segmentation and detection, the design of the text prompt is crucial. Our method, being task-agnostic, can be easily embedded into any vision task to optimize the text prompt. For instance, in our experiments, we incorporated IPO into pre-trained semantic segmentation models [a] [b], where the original text prompt was \"a photo of a [CLASS].\" Using GPT-4o as the LLM and LMM, we crafted more effective text prompts specifically suited to the open-vocabulary semantic segmentation task, leading to enhanced performance and demonstrating the value of IPO in optimizing text prompts for this application. We intend to further investigate the use of IPO in other vision tasks in future work.\n\n\n| Methods         | pAcc | mIoU(S) | mIoU(U) | hIoU |\n|-----------------|------|---------|---------|------|\n| SPNet      | -    | 78.0    | 15.6    | 26.1 |\n| ZS3       | -    | 77.3    | 17.7    | 28.7 |\n| CaGNet      | 80.7 | 78.4    | 26.6    | 39.7 |\n| SIGN       | -    | 75.4    | 28.9    | 41.7 |\n| Joint        | -    | 77.7    | 32.5    | 45.9 |\n| ZegFormer  | -    | 86.4    | 63.6    | 73.3 |\n| zsseg [a]   | 90.0 | 83.5    | 72.5    | 77.5 |\n|ZegCLIP [b] | 94.6 | 91.9    | 77.8    | 84.3 |\n| **zsseg + IPO**    | 91.2 |84.7    | 73.2    | 78.6|\n|**ZegCLIP + IPO** | 95.3  | 92.7    | 78.7   | 85.1|   \n\n\n\n[a] Xu, et al. \"A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model.\" ECCV 22\n\n[b] Zhou, et al. \"Zegclip: Towards adapting clip for zero-shot semantic segmentation.\" CVPR 23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Aeu4g2a9Hs", "reviewer": "Reviewer_VMZQ", "summary": "The paper addresses the challenge of optimizing text prompts for vision-language models, specifically focusing on the interpretability of these prompts. Traditional methods for prompt optimization rely on gradient descent, which often results in overfitting and produces prompts that are not human-readable. This paper introduces a new approach called Interpretable Prompt Optimizer (IPO), which leverages large language models (LLMs) to generate and optimize prompts in a dynamic and interpretable manner. The paper details the design of the IPO framework and provides extensive experimental results across 11 datasets. The findings demonstrate that IPO not only improves the accuracy of vision-language models compared to traditional gradient-based methods but also significantly enhances the interpretability of the generated prompts.", "questions": "Authors explor using LLM as an interpretable optimizer for prompt learning, I think this is a contribution worth a weekly accept score, i.e., 6 scores in NeurIPS rating. However, this score should be build upon all below concers are addressed:\n1)  I can tolerate that the current performance are not that remarkable, but this tolerance should be built upon the experiments are fair. To make a fair comparison in the framework, authors should make comparison with some of the knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. So that we can exculde  the effectivness of the memory retrieval mechanism of IPO.\n\n2) Why Table 5 only contains the 1-shot comparive experiments ? Detailed 16-shots or training with more samples setting are needed. \nI will confirm the effectivness of the your IPO if you draw a remarkable performance. If not, just take a breath and provide enough evidence that LLM optimizer is effectiveness, such as sound losses, sound improvement in the training set, sound improvement in the test sets in different settings. I will agrre with the effectivness of the your IPO if the evidence are sound.\n\n\n[1] Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149. \n\n[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663. \n\n[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.", "limitations": "Experiments now are not sound and the authors do not provide enough evidnce to prove the effectiveness of IPO.", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces IPO, a novel method that uses LLMs to optimize prompts in a way that maintains human readability and interpretability. This approach contrasts with traditional gradient-based methods that often produce opaque prompts.\n\n2. The POP system stores past prompts and their performance metrics, allowing LLMs to generate more effective prompts through iterative refinement. This system enhances the contextual understanding and effectiveness of the generated prompts.\n\n3. The IPO method incorporates LMMs to generate image descriptions, which improves the synergy between textual and visual data. This leads to more accurate and contextually relevant prompts.\n\n4. The paper validates the effectiveness of IPO across 11 different datasets, demonstrating its superiority over traditional methods in terms of both accuracy and interpretability.", "weaknesses": "The main weaknesses of this paper lies in the experiment can not support the effectiveness of the LLM optimizer, which is the most important contribution of the authods. IPO incorporates the Large Language Model (LLM) as an optimizer to learn an interpretable prompt pool, and everytime a new instance comes, prompts will be extracted from this prompt. This mainstream is similari to knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. Therefore, the experiments should contains:\n1) **A fair comparitive with some of the knowledge bank based prompt learning methods  such as L2P [1], AttriCLIP [2], DualPrompt [3]. To exclude the effectivness of the memory retrieval mechanism of IPO.**\n\n2) **The current comparitive experiment merely involves the 1-shot setting in Table 5. Please provides a more comprehensive experiment settings, such as 16-shots and full-sized tuning setting. Besides, the IPO methods do not draw an obvious performance improvement in benmark datasets in the case that IPO has an obviously longer prompt that the learnable method.** \n\n\n3) What's more, the IPO does not perform well in the large-scale generic datasets such as ImageNet. This is not intuitive, because the LLM should show a better performance in tghe generic datasets. Lastly, I am not agree with your average metric, because the data size of these 11 benchmarks have significant difference.\n\n4) Table 6 are not referred in the author's paper, and the detailed benchmark is also not elaborated.\n\n[1]  Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149.\n[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663.\n[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Comparison with Knowledge Bank-Based Prompt Learning Methods:**\n\nWe sincerely thank the reviewer for pointing out these three interesting works. First, we would like to clarify the differences between our IPO and these methods. The works mentioned (L2P, AttriCLIP, DualPrompt) are based on visual prompt learning for continual learning, where they train a prompt bank during the training phase, and during testing, the appropriate prompt is retrieved from this prompt bank using the test sample. In contrast, our IPO primarily focuses on few-shot VLMs, where during training, LLMs are utilized as our prompt bank to generate dataset-specific prompts based on contextual information. During testing, predictions are made directly using the learned prompts without the need for additional retrieval.\n\nSince our IPO optimizes text prompts using LLMs and cannot be directly applied to these three methods, we conducted a fair comparison by applying L2P within the Visual Prompt Tuning (VPT) [c]  framework, which also learns prompts in the visual space and applies them to few-shot VLM tasks. VPT + L2P trains a prompt bank during training, and during testing, the test samples query the prompt bank to find the appropriate prompt. The table below shows the comparison between VPT + L2P and our method across 11 datasets in the 16-shot setting. We found that while L2P does improve VPT's performance, demonstrating L2P's effectiveness in VLM, our IPO still outperforms VPT + L2P. We will include this comparison in the revised manuscript to demonstrate that the effectiveness of IPO is not solely due to a memory retrieval mechanism.\n\n| Model         | Base   | Novel  | H      | \n|---------------|--------|--------|--------|\n| VPT [c]          | 72.53  | 72.34  | 72.43  |   \n| VPT + L2P     | 74.15  | 74.93  | 74.54  |  \n| IPO           | 79.92  | 80.51  | 80.21  |\n\n[c] Jia, et al. \"Visual prompt tuning.\" European Conference on Computer Vision 2022.\n\n\n**(2) Comparison with Other Prompt Learning Methods:**\n\nIn Table 6, we compare our IPO with other methods in the 16-shot setting. To clarify, prompt learning includes both textual prompt tuning and prefix tuning. Prompt tuning methods like CoOP and CoCoOp treat the textual prompt as learnable parameters optimized using few-shot samples, while prefix tuning involves adding learnable tokens to the text encoder, vision encoder, or both. Examples include MaPLe, PromptSRC, and CoPrompt. Our method falls under prompt tuning, so we mainly compare against other prompt tuning methods. From Table 6, our IPO shows a 4.38% improvement in average performance compared to CoCoOp, while providing an interpretable prompt as well. We will provide a complete comparison table of the 16-shot performance across 11 datasets in the appendix of the revised manuscript.\n\n**(3) Performance on Large-Scale Generic Datasets:**\n\nIPO with GPT-3.5 Turbo, indeed, does not show an improvement on the large-scale ImageNet. This is because ImageNet has a large number of classes and samples, which results in longer LLM input when generating descriptions for each sample. GPT-3.5 Turbo has limited performance in handling long-text inputs. The table below shows the results on ImageNet when IPO uses GPT-4o, which has superior long-text understanding compared to GPT-3.5 Turbo. We found that IPO using GPT-4o leads to better performance improvements over other methods as well as a considerable improvement over IPO with GPT-3.5 Turbo. \n\n| Model            | Base   | Novel  | H      | \n|------------------|--------|--------|--------|\n| CLIP             | 72.43  | 68.14  | 70.22  |   \n| CoOp             | 73.20  | 67.43  | 70.20  |  \n| CoCoOp           | 73.90  | 69.07  | 71.40  |\n| MaPLe            | 74.03  | 68.73  | 71.28  |\n| CoPrompt         | 73.97  | 70.87  | 72.39  |\n| IPO w/ GPT-3.5   | 74.09  | 69.17  | 71.54  |\n| IPO w/ GPT-4o    | 76.14  | 72.13  | 74.09  |\n\nAdditionally, using the harmonic mean as the average metric is a common strategy in prompt learning for VLMs, as seen in works like CoOp, CoCoOp, MaPLe, and CoPrompt. We will clarify this in the revised manuscript.\n\n**(4) Clarification of Table 6:**\n\nTable 6 uses the same benchmarks as Table 5, with the 16-shot setting across 11 datasets. We will reference Table 6 in Line 283 of the revised manuscript and clarify the benchmarks used. Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CRF7wgLKF3", "reviewer": "Reviewer_4MWY", "summary": "This paper proposes an interpretable prompt optimizer (IPO) which uses an LLM to iteratively optimize prompt templates that lead to improved zero-shot visual classification performance on CLIP.", "questions": "1) In the base-to-novel evaluation, 1-shot refers to the setting where one sample of each category in the base classes is provided during training while no sample of the novel classes is used in training? \n2) Line 165, “all four models failed to understand our Prompt Optimization Prompt”, here it should be four models or six models (as the experiments are conducted on six models)?", "limitations": "The proposed method has improved performance on novel classes, but suboptimal performance on base classes where the prompts are optimized.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) The proposed method outperforms baselines on the novel classes in the evaluation on base-to-novel generalization benchmark", "weaknesses": "1) Line 91, “However,to the best of our knowledge, no existing studies have investigated how LLMs could be used to optimize text prompts within vision-language models”. There are actually related work that employed LLMs for optimizing prompts for visual classification in VLMs which are not covered in this paper, e.g.  \n[a] Liu et al. Language Models as Black-Box Optimizers for Vision-Language Models. CVPR’24  \n[b] Mirza et al. Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs. ECCV’24  \nAnalysis and proper comparison to these works should be conducted.\nHere [a] also uses an LLM (ChatGPT) to iteratively update the prompt templates for visual classification on CLIP with the good prompts and bad prompts passed as in-context examples. This weakens the novelty of the proposed method. \n2) In Table 5, why is the performance of the proposed method consistently worse than baselines on base classes? How does the method perform in a usual one-shot classification setting (instead of base-to-novel setting)? \n3) In table 1,  the metric “H” should be specified", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Missing Important References:**  \nWe thank Reviewer 4MWY for bringing the CVPR 2024 paper by Liu et al. and the forthcoming ECCV 2024 paper by Mirza et al. to our attention. Both works are indeed relevant and will be discussed in the related work and experimental sections of our revised manuscript. While these papers contribute valuable insights to the field, they do not compromise our novelty claim.\n\nLiu et al. propose a method that utilizes LLMs as black-box optimizers for vision-language models, iteratively refining prompts based on in-context examples. Their approach focuses on leveraging ChatGPT to improve prompt templates for visual classification tasks. In contrast, our IPO method differs in that we incorporate past prompt history as episodic memory within the design process. This allows our model to generate better prompts by considering the contextual information of previous successes and failures. Additionally, we use LMMs to generate image descriptions during the prompt design process, enabling the creation of more accurate, dataset-specific prompts. Our experimental results demonstrate that IPO outperforms Liu et al.'s method, with a 13.19% improvement in 1-shot base-to-novel generalization (74.29% vs. 61.1%).\n\nMirza et al. explore a different aspect of prompt optimization by focusing on zero-shot vision-language models. Their method does not utilize a specific scoring mechanism for prompt optimization, whereas our IPO method employs the training sample's loss and accuracy as a scoring mechanism, optimizing prompts for better performance in few-shot scenarios. While Mirza et al. make important contributions, our approach is distinct in its focus on few-shot learning and the use of LMMs to enhance prompt accuracy.\n\n\n\n\n\n**(2) Performance on Base Classes and One-Shot Classification (Table 5):**\n\nIn Table 5, the performance of our proposed IPO method on base classes is indeed lower compared to some baselines. This is because IPO is designed to optimize prompts with a focus on generalization across both base and novel classes. While this approach enhances performance on novel classes, it can result in slightly reduced performance on base classes where the model might not fully exploit specific base class features to avoid overfitting.\n\nHowever, this difference in performance tends to diminish when relying on higher capacity models, which are better equipped to handle the complexities of both base and novel classes. To further clarify, we evaluated IPO in a usual one-shot classification setting, where both base and novel classes were treated as base classes during training. The results, presented in the table below, show that IPO still outperforms previous methods in this setting, demonstrating its ability to generate more generalized and effective prompts across diverse datasets.\n\n| Models   |  Acc     | \n|----------|----------|\n| CoOp     | 64.13%   | \n| CoCoOp   | 67.24%   |\n| IPO      | 68.71%   |\n\n\n\n\n\n**(3) Clarification of \"H\":**  \nFollowing the common convention in the prompt learning literature CoOp [50], CoCoOp [51], and MaPLe [18], \"H\" refers to the harmonic mean. We will clarify this in the revised manuscript.\n\n**(4) 1-shot in Training:**  \nYes, the reviewer's understanding is correct. In the 1-shot setting, one sample per category in the base classes is provided during training, with no samples from novel classes used during training. This approach is consistent with the traditional base-to-novel setting [50, 51, 18].\n\n**(5) Typo on Line 165:**  \nThis is a typo; it should refer to six models instead of four. We will correct this in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hHbbMqdCYA", "reviewer": "Reviewer_xRmh", "summary": "This paper presents an Interpretable Prompt Optimizer (IPO) designed to improve the performance and interpretability of pre-trained visual language models such as CLIP. By dynamically generating textual cues using a large language model (LLM) and combining it with a large multimodal model (LMM) to generate image descriptions, IPO demonstrates better accuracy and interpretability than traditional gradient descent-based cue learning methods on multiple datasets.", "questions": "1）Will this parameterless training time be shorter for IPO compared to the gradient descent approach?\n\n2）How does the length of the prompt history affect the stability of training convergence？", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1）The paper proposes a new prompt optimization framework that combines the advantages of LLM and LMM.\n\n2）Extensive testing on 11 datasets demonstrated the effectiveness of the method.\n\n3）The paper is clearly structured with diagrams and charts to aid illustration.\n\n4）Improved interpretability of visual language models is important for achieving better human-computer collaboration.", "weaknesses": "1）Line 150 :(2) -> (3)\n\n2）There are no citations throughout the paper Table 6.\n\n3）Lack of cross-dataset experimental evaluation to validate the generalizability of IPO.\n\n4）The paper mentions the use of large language models such as GPT-3.5 Turbo, but does not discuss in detail the computational cost and efficiency of these models during training and inference. For resource-constrained environments, this may be an important consideration.\n\n5）The importance of each component of the IPO is described in Section 5.2, but it lacks in-depth analysis, e.g., it could be mined for some more intrinsic reasons for the rise in points from a regularization perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Line 150:**  \nFixed. Thank you.\n\n**(2) Table 6:**  \nWe will add citations for each method in Table 6 in the revised manuscript.\n\n**(3) Cross-Dataset Experimental Evaluation:**  \nWe conducted a cross-dataset experimental evaluation, following the traditional setting, and found that our IPO outperforms previous gradient-based prompt learning methods. The results are shown in the table below, and we will include this experiment in the revised manuscript.\n\n| Source     | ImageNet | Caltech101 | OxfordPets | StanfordCars | Flowers102 | Food101 | Aircraft | SUN397 | DTD   | EuroSAT | UCF101 | Average |\n|------------|----------|------------|------------|--------------|------------|---------|----------|--------|--------|---------|--------|---------|\n| **CoOp**   | **71.51** | 93.70      | 89.14      | 64.51        | 68.71      | 85.30   | 18.47    | 64.15  | 41.92  | 46.39   | 66.55  | 63.88   |\n| **Co-CoOp**| 71.02     | **94.43**  | 90.14      | 65.32        | 71.88      | 86.06   | 22.94    | 67.36  | 45.73  | 45.37   | 68.21  | 65.74   |\n| **IPO**    | 72.15     | 94.34      | **90.96**  | **66.10**    | **72.75**  | **86.75**| **25.14**| **67.97**| **47.01**| **48.56**| **69.23**| **67.36**|\n\n**(4) Computational Cost and Efficiency:**  \nSince our model uses LLMs to optimize prompts, it doesn't involve gradient calculations and only requires API calls to the LLM and LMM for prompt optimization. Therefore, compared to directly using CLIP, our method does not incur additional memory overhead. Regarding training time, we use GPT-3.5 Turbo as our default optimizer, iterating 100 steps for each dataset to derive the final prompt (as noted in Line 199). The training speed is primarily dependent on the response time of GPT-3.5 Turbo in generating prompts. During testing, the generated text prompts are directly fed into the text encoder, resulting in no additional computational cost, and inference efficiency remains consistent with CLIP.\n\n\n\n**(5) In-Depth Analysis of IPO Components:**  \nFrom a regularization perspective, our IPO leverages episodic memory to prevent overfitting by considering the performance of past prompts, ensuring the generation of more generalizable prompts. In **(7) Impact of Prompt History Length**, we found that the length of Prompt History affects the prompts generated by IPO, further proving that episodic memory effectively addresses overfitting. Additionally, the incorporation of image descriptions ties the prompts closely to the data, reducing variance in predictions, as demonstrated in Table 4 of our paper. Furthermore, using LLMs for non-gradient-based prompt generation introduces variability, further mitigating the risk of overfitting. We will include this analysis in the revised manuscript.\n\n**(6) Shorter Training Time:**  \nYes, compared to gradient descent approaches, IPO’s parameterless training process is shorter because it does not require gradient calculations or parameter updates.\n\n**(7) Impact of Prompt History Length:**  \nWe have evaluated the impact of different prompt history lengths on the final performance, as shown in the table below.\n\n| History Length | Base   | Novel  | H      | \n|----------------|--------|--------|--------|\n| n = 0          | 69.15  | 75.20  | 72.04  |   \n| n = 1          | 70.25  | 75.43  | 72.74  |  \n| n = 5          | 70.95  | 76.21  | 73.49  |  \n| n = 10         | 71.23  | 76.41  | 73.72  |    \n| n = 20         | 71.76  | 77.00  | 74.29  |  \n| n = 50         | 71.81  | 76.81  | 74.23  |  \n| n = 100        | 72.02  | 76.81  | 74.33  |  \n\nWe found that without prompt history, model performance decreases due to the lack of contextual information for the LLM, making it difficult to converge. As the history length increases, performance gradually improves, reaching convergence at n=20. Although n=100 yields the best average performance, a longer history increases the length of LLM input, leading to higher API costs. Therefore, we selected n=20 for our IPO. We will include this experiment in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WN0SvBL35R", "reviewer": "Reviewer_XUaZ", "summary": "This paper introduces a method called Interpretable Prompt Optimization (IPO) for vision-language models.\nThe goal of IPO is to improve the performance and interpretability of vision-language models by dynamically generating and optimizing text prompts. The paper addresses the limitations of current prompt optimization methods, which often lead to overfitting and produce prompts that are not understandable by humans. IPO leverages large language models (LLMs) to generate effective and human-readable prompts (which is the main advantage of their method). It incorporates a Prompt Optimization Prompt that guides the LLMs in creating prompts and stores past prompts with their performance metrics, providing rich in-context information. Additionally, IPO integrates a large multimodal model (LMM) to generate image descriptions, enhancing the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance while maintaining human comprehension. The paper validates IPO across 11 datasets and demonstrates that it improves the accuracy of existing prompt learning methods and enhances the interpretability of the generated prompts.\n\nOverall, IPO ensures that the prompts remain human-understandable, facilitating better transparency and oversight for vision-language models.", "questions": "Overall, the paper provides a fresh view on prompt-tuning by making the process interpretable. In a way, the paper automates prompt-engineering for a domain by using a LLM as an optimizer. I am going with Borderline Accept, but happy to revisit my scores if the Weaknesses are addressed in the rebuttal.", "limitations": "See Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles one of the most important problems in prompt tuning. Existing prompt tuning methods are not interpretable at all (as they optimize a sequence of vectors with a model specific loss function). This paper mitigates this issue by using a LLM as an optimizer -- which inherently generates text.\n\n- The prompt optimization template though is simple, can be scalable towards multiple tasks (as shown in the paper).\n\n- Thorough ablations on the language model and the multimodal language model has been provided in the paper.", "weaknesses": "- [Major]. While the paper obtains interpretable prompts, the method primarily obtains improved performances for the novel classes, but not for the base classes.  In fact for a large number of the datasets, the base classes performance is significantly low when compared to other methods. I believe that there should be a balance in the performance between the novel and base classes. To improve the paper, the authors are suggested to provide strong justifications explaining this.\n\n- [Minor]. This area has a multitude of papers so it might be tedious to compare with all the methods. However, I would suggest the authors to add a separate section having a discussion on the comparison of their method with more recent prompt-tuning methods such as LFA, PLOT, DFA etc. \n\n- [Minor]. I would also suggest the authors to extend the discussion of the limitations of the method. Currently, as I see, the method is suitable for few-shot scenarios, but what if one has access to a large number of domain samples? How can this method be made scalable?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(1) Performance on Base vs. Novel Classes:**\n\nPrevious prompt learning methods like CoOP and CoCoOP, which rely on gradient-based optimization, tend to overfit to base classes, resulting in performance loss on novel classes. Our IPO, however, uses LLMs to optimize prompts with a focus on learning more generalizable prompts across datasets. This results in better performance on novel classes compared to gradient-based methods. \nIn response to Reviewers U899 and 4MWY, we have conducted additional experiments demonstrating that our method can achieve a balance in performance between novel and base classes by increasing the capacity of the LLM and LMM, which allows for better handling of longer context information.\n\n\n**(2) Comparison with Recent Methods:**  \nWe appreciate the reviewer’s suggestion. We have provided comparisons with LFA and PLOT in the tables below, using the same experimental setting. Our IPO method consistently outperforms both LFA and PLOT. Regarding DFA, we were unable to locate the corresponding paper. If possible, could the reviewer please provide the title or more details on DFA? We will include these comparisons with recent prompt-tuning methods (LFA, PLOT, DFA, etc.) in the revised manuscript.\n\n**Comparison with LFA across 11 datasets in 16-shot scenarios:**\n\n| Model | Base  | Novel | H     | \n|-------|-------|-------|-------|\n| LFA   | 83.62 | 74.56 | 78.83 |  \n| IPO   | 79.92 | 80.51 | 80.21 |\n\n**Comparison  with PLOT on average accuracy across 11 datasets in 1-shot and 16-shot scenarios:**\n\n| Model | 1-shot | 16-shot | \n|-------|--------|---------|\n| PLOT  | 65.45  | 76.20   | \n| IPO   | 74.29  | 80.21   | \n\n\n\n\n\n\n\n**(3) Scalability for Large Datasets:**\n\nOur IPO method is primarily designed for few-shot scenarios. However, when dealing with large domain-specific datasets, the need to generate extensive image descriptions, which can lead to substantial computational costs due to the large text inputs required for LLMs.\nCurrently, our model uses an input length of approximately 5,000 tokens. When scaled to larger datasets, the input length may increase to around 50,000 tokens. Using GPT-4 with an 8k context length, the cost for our current input size (5,000 tokens) is approximately \n 0.15 dollars per input (0.03 dollars per 1,000 tokens). For the expanded input size of 50,000 tokens, the cost would rise to approximately 3.00 dollars per input.\nIf we were to use GPT-4 with a 32k context length, the cost for the 50,000-token input would be approximately 3.00 dollars for the first 32,000 tokens and an additional 1.08 dollars for the remaining 18,000 tokens, totaling approximately 4.08 dollars per input. Since our IPO method requires 100 iterations during training, the costs would multiply accordingly when scaled to large inputs.\n\n A potential solution is to fine-tune an LMM to input training images directly, thus eliminating the need for additional description generation. We will clarify this limitation and potential solution in the revised manuscript.  We believe the cost is justified, especially for domains like health, legal, and finance, where human interpretation of prompts is key.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to extend our sincere thanks to all the reviewers for their valuable feedback and suggestions. Your insights have been instrumental in refining our work, and we have addressed your concerns in the revised manuscript. Below, we highlight the most significant updates and improvements based on your feedback:

1. **Improved Model Performance with Higher Capacity LLMs/LMMs**: 
   We have conducted additional experiments demonstrating that by increasing the capacity of the LLM and LMM to GPT-4o, our IPO method achieves better performance across both base and novel classes. 

2. **Comparative Analysis with Recent Methods**:
   - We conducted a fair comparison between IPO and knowledge bank-based prompt learning methods, such as L2P, within the VPT framework.
   -  We have also included evaluations against recent prompt-tuning methods like LFA and PLOT across multiple datasets.  

3. **Scalability Considerations**:
   We have detailed the computational costs associated with scaling our method to larger datasets, particularly when using GPT-4 with varying context lengths.  

4. **Clarifications and Additional Experiments**:
   - We corrected typos and added necessary references in the revised manuscript.
   - We have also provided additional tables and cross-dataset evaluations to further substantiate our findings.

These updates address the core concerns raised by the reviewers, and we believe they strengthen the manuscript. Thank you again for your constructive feedback, and we look forward to any further suggestions you may have.

Meta-review: The paper introduces an interpretable prompt optimization method (IPO) for vision-language models. IPO uses large language models (LLMs) to dynamically generate textual prompts, and helps improve both performance and interpretability, which are key advantages over traditional gradient-based methods. The method’s scalability across multiple datasets and improved accuracy for novel classes are notable strengths.

Reviewers appreciated the novel approach of using LLMs for prompt optimization and the clear presentation of the method. However, concerns were raised about the performance drop on base classes, scalability to larger LLMs, evaluation on additional tasks, missing comparisons with some related methods, cross-dataset evaluation, ablation on prompt history length and the computational efficiency of IPO in resource-constrained environments. The authors provided a detailed rebuttal addressing these concerns, including new experimental results and clarifications on limitations. All reviewers appreciated the comprehensive feedback and decided to vote for acceptance in the final recommendation. 

Taking into account the reviews, rebuttal, and discussions, the area chair believes that IPO presents a valuable contribution to interpretable prompt learning for VL models. The paper’s strengths outweigh its weaknesses, and thus, an accept decision is recommended. Authors are reminded to include all suggested changes in the final version.
Decision: Accept (poster)