IPO: Interpretable Prompt Optimization for Vision-Language Models

Authors: yingjun  du
wenfang  sun
g m cees
  snoek
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 20:26:56

Abstract: Pre-trained vision-language models like CLIP have remarkably adapted to various downstream tasks. Nonetheless, their performance heavily depends on the specificity of the input text prompts, which requires skillful prompt template engineering. Instead, current approaches to prompt optimization learn the prompts through gradient descent, where the prompts are treated as adjustable parameters. However, these methods tend to lead to overfitting of the base classes seen during training and produce prompts that are no longer understandable by humans. This paper introduces a simple but interpretable prompt optimizer (IPO), that utilizes large language models (LLMs) to generate textual prompts dynamically. We introduce a Prompt Optimization Prompt that not only guides LLMs in creating effective prompts but also stores past prompts with their performance metrics, providing rich in-context information. Additionally, we incorporate a large multimodal model (LMM) to condition on visual content by generating image descriptions, which enhance the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance, while maintaining human comprehension. Extensive testing across 11 datasets reveals that IPO not only improves the accuracy of existing gradient-descent-based prompt learning methods but also considerably enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, our approach ensures that the prompts remain human-understandable, thereby facilitating better transparency and oversight for vision-language models.

Main paper: 
 introduction: 
 Vision-language models, trained on a diverse array of image-text pairs encapsulating a broad vocabulary of real-world concepts [1, 2, 3] , have demonstrated notable adaptability across various downstream tasks [4, 5, 6, 7] . These models perform zero-shot image classification by filling in a predefined prompt template (e.g., "a photo of a [CLASS]") with specific class names for the text encoder. Despite their effective generalization to new tasks, the performance can be influenced by minor changes in the wording of prompt templates [8] . Instead of manually creating handcrafted prompts, recent developments in natural language processing [9, 10] and computer vision [8, 11, 12, 13] have proposed methods to learn a set of soft prompts with minimal labeled data. Despite the strides made in learning prompts, the current state of the art remains limited by its lack of interpretability and the overfitting problems on the base classes, which can be prohibitive in diverse and dynamic application environments. These limitations underscore the need for a more adaptable and user-friendly approach to prompt optimization in vision-language models. Drawing from recent advancements in using large language models (LLMs) as optimization tools [14] , our paper, for the first time, incorporates these capabilities into vision-language modeling. Unlike gradient descent-based methods [8, 11, 13] , which often fail to provide explanations for the generated [8, 11] treat the text prompt as learnable parameters V . By minimizing the loss through gradient descent on the training set, an optimized prompt V is obtained after I iterations, which is not interpretable by humans. In contrast, our interpretable prompt optimization leverages an LLM as optimizer to optimize the loss and accuracy. After I iterations, the resulting optimized top prompt is effective and human-readable.prompts and tend to overfit on base classes, natural language-based methods enable LLMs to develop and refine solutions through continuous feedback iteratively. This approach improves interpretability in complex tasks like prompt optimization for vision-language models, making it easier for humans to understand the generated prompts. However, existing research on these methods [14, 15, 16] primarily addresses language tasks and has not yet explored their potential for integrating prompt optimization with an LLM in vision-language models.To address these challenges, this paper proposes an interpretable prompt optimizer (IPO) for visionlanguage models that leverages the capabilities of LLMs to generate and refine text prompts dynamically. First, we design a Prompt Optimization Prompt to prompt LLMs to generate more effective prompts that improve the accuracy of CLIP and reduce the loss in base classes. Our Prompt Optimization Prompt also stores past prompts along with their corresponding accuracy and loss as episodic memory, thereby providing richer in-context information to enable LLMs to generate more effective prompts. Second, to incorporate image information within the Prompt Optimization Prompt, we propose using a large multimodal model (LMM) to generate descriptions of images in base classes that can be added to the Prompt Optimization Prompt. This integration facilitates a more intuitive interaction between the textual and visual modalities, which allows the Prompt Optimization Prompt to utilize image information, thereby generating datasetspecific prompts to enhance the generalization performance of CLIP. The framework of our IPO, illustrated in Figure 1 , showcases the comparison between traditional gradient-based prompt optimization and the proposed interpretable prompt optimization. Third, the prompts generated by our optimizer are human-interpretable. For example, on the Food101 dataset [17] , the initial prompt evolves from ["a photo of a [CLASS]"] to ["Categorize the image depicting a delicious and appetizing <CLASS> with remarkable visual qualities."]. Our generated prompts perform 10.29% better in novel classes than the gradient-based method CoOP [8] , reducing overfitting while maintaining interpretability.We validated our IPO across 11 different datasets, demonstrating that it surpasses traditional gradientbased state-of-the-art methods in accuracy and excels in interpretability. Our approach generates human-comprehensible prompts that can be seamlessly integrated into existing vision-language models to enhance performance. We conducted rigorous comparative experiments to quantify the interpretability between gradient-based prompt learning and our method. We demonstrated the importance of specific keywords in our generated prompts and revealed that not all tokens learned through traditional prompt learning methods are essential. 
related work: 
 Prompt learning in vision-language models. Prompt learning, originally introduced in the natural language processing community [18, 19, 20] , involves applying a fixed function to input tokens to provide task instructions to the model. In the computer vision community, prompt learning has been explored in various forms, including textual prompt tuning [8, 11, 21, 22, 23, 24] , and prefix tuning [13, 12, 25, 26, 27, 28, 29] . 1) Prompt tuning mainly involves treating text prompts as learnable parameters, using a small amount of data to fine-tune these parameters. As pioneered by CoOp [8] and CoCoOp [11] , which both fine-tune a CLIP vision-language model [30] for few-shot transfer by optimizing a continuous set of prompt vectors within its language branch. Bayesian prompt learning [21] formulated prompt learning as a variational inference problem and demonstrated its ability for unseen class generalization. 2) Prefix tuning primarily involves adding learnable tokens to the text encoder [31] , vision encoder [12, 25] , or both encoders [13, 27, 28, 32] . These tokens are fine-tuned using a small amount of data. Note that these methods do not optimize the initial text prompts. Instead, they focus on enhancing the model's understanding capabilities by integrating these additional, trainable tokens. Our method belongs to prompt tuning, but unlike previous approaches that use gradient descent to optimize prompts, we propose using LLMs to optimize prompts. Our method leverages the natural language capabilities of LLMs to iteratively refine feedback-based prompts, aiming to enhance both the effectiveness and the explainability of the prompts.LLMs as prompt optimizers. Several recent works explore the role of LLMs as prompt optimizers for NLP tasks. Some use LLMs to directly optimize the task instruction for in-context learning [33, 34, 14] . Other studies use LLMs to mutate prompts for evolutionary algorithms [35, 36] . However, to the best of our knowledge, no existing studies have investigated how LLMs could be used to optimize text prompts within vision-language models. This approach could potentially open up new avenues for integrating and enhancing the capabilities of vision-language models through more effective and contextually appropriate text prompts.Meta-prompting. Suzgun and Kalai [37] introduce meta-prompting to transform a single LLM into a versatile "conductor" capable of managing and integrating multiple independent LLM queries.By using high-level instructions, meta-prompting guides the LLM in decomposing complex tasks into smaller subtasks. The core of OPRO [14] involves designing a meta-prompt for LLMs to optimize prompts for each task. This meta-prompt includes two key pieces of information: previously generated prompts with their corresponding training accuracies and a description of the optimization problem. Self-select [38] leverages meta-prompting to optimize instruction selection. It considers a set of provided templates and chooses the most suitable template. Meta-prompting is related to instruction tuning [39] as both techniques provide high-level guidance to improve the performance and adaptability of LLMs. However, while instruction tuning focuses on fine-tuning models with a variety of tasks to improve generalization, meta-prompting offers the advantage of dynamically guiding the model to decompose and manage complex tasks in real-time. Liu et al. [40] proposes a method that utilizes LLMs as black-box optimizers for vision-language models, iteratively refining prompts based on in-context examples. Their approach focuses on leveraging ChatGPT to improve prompt templates for visual classification tasks. Mirza et al. [41] explores a different aspect of prompt optimization by focusing on zero-shot vision-language models. Our Prompt Optimization Prompt is akin to meta-prompting, it stores past prompts along with their corresponding accuracy and loss, thereby providing richer in-context information to enable LLMs to generate more effective prompts. Different from prior meta-prompting, our Prompt Optimization Prompt generates prompts beyond LLMs for vision-language models. 
preliminaries: 
 Contrastive Language-Image Pre-Training (CLIP). The goal of CLIP [30] is to develop an image encoder f I and a text encoder g T via contrastive pre-training with a large collection of paired images and captions. This process aims to map image-text pairs to a common semantic space. After the pre-training phase, CLIP is able to perform zero-shot visual recognition by treating classification as a task of matching images to text. Specifically, the placeholder term "[CLASS]" is used within a prompt template (e.g., "a photo of a [CLASS]") for the text encoder g T . Here, g T (T i ) denotes the text features adapted for class i, and the probability of classifying class i from an image I is:EQUATIONwhere ⟨g T (T i ), f I (I)⟩ represents the cosine similarity between the image feature f I (I) and the text feature g T (T i ) specific to the i-th class, K is the total number of classes, and τ is the temperature parameter that is tuned during training.Prompt learning improves the adaptability of the CLIP model by eliminating the need for manual prompt engineering. It facilitates the automatic generation of prompts using a limited number of examples from a downstream task. CoOp [8] presents a method where a set of M continu-ous context vectors V ={v 1 , v 2 , . . . , v M } serve as the learnable prompt. The constructed prompt T i ={v 1 , v 2 , . . . , v M , c i } merges these learnable context vectors V with the class-specific token embedding c i , which is then processed by the text encoder g T (•). In CoOp, the optimization of these static context vectors V aims to minimize the negative log-likelihood for the correct class token:EQUATIONhere, y i represents the one-hot encoded ground-truth label for class i. In downstream applications, the pre-trained model parameters are kept unchanged, which allows the learnable prompt vectors V to be optimized efficiently using only a small number of samples through the minimization of the cross-entropy loss. 
methods: 
 Figure 1 depicts the comprehensive structure of our interpretable prompt optimizer. At each step of the optimization, the LLM generates candidate prompts for the vision-language task by considering both the description of the optimization problem and the feedback from previously evaluated prompts stored in the Prompt Optimization Prompt. These new prompts are then assessed and incorporated into the Prompt Optimization Prompt for future optimization cycles. The optimization process concludes either when the LLM can no longer generate prompts that improve the optimization scores, or when a predefined maximum number of optimization steps is reached. Next, we will detail the design of the Prompt Optimization Prompt and explain how image information is integrated into the Prompt Optimization Prompt.Prompt Optimization Prompt design At the core of our optimizer is the design of the Prompt Optimization Prompt, which enhances the performance of the vision-language model by optimizing the prompts through the prompt LLM. Figure 2 shows an example of our Prompt Optimization Prompt. Our Prompt Optimization Prompt consists of the following components: (1) Instructions: These guide the LLM by clearly defining its task to optimize the prompt for achieving better performance in classification tasks. (2) Textual descriptions of training images: These descriptions provide the LLM with detailed information about the images, enabling it to generate dataset-specific prompts.(3) Previously generated prompts and corresponding scores: This component supplies in-context information, including past prompts and their performance metrics, allowing the LLM to refine its prompt generation more accurately. By incorporating these elements, our approach leverages the iterative refinement capabilities of LLMs to dynamically generate and optimize text prompts. The instructions ensure that the LLM understands the optimization goal, the textual descriptions offer rich image-related context, and the historical data aids in producing more effective and precise prompts.For the textual descriptions of training images, we utilize a large multimodal model (LMM) to generate text descriptions for each training image. Specifically, we employ MiniCPM-V-2.0 [43] to generate descriptions of the content of images from base classes.In the appendix, we provide content descriptions for some images from each dataset generated using MiniCPM-V-2.0. We denote the extracted image textual features as f M (•).Additionally, we have attempted to directly optimize prompts using the LMM with the Prompt Optimization Prompt. Specifically, we input images from the base classes and the Prompt Optimization Prompt into the LMM, aiming for the LMM to generate better prompts. We experimented with six different LMMs: BLIP-2 [44] , Qwen-VL-Chat-9.6B [45] , FUYU-8B [46] , MiniCPM-V-2.0 [43] , and llava-llama3-8B [47] . Unfortunately, all six models failed to understand our Prompt Optimization Prompt and generated new prompts that were merely descriptions of the images, not the universal prompts we desired. This failure might be due to the fact that the training of these LMMs did not consider such a task. Note that image descriptions are not mandatory. In our 16-shot experiments, we omitted the image descriptions in the Prompt Optimization Prompt due to the limited text input length that the LLM can handle.We utilize an episodic memory mechanism to retrieve past prompts and their corresponding scores, which include metrics such as loss and accuracy. Here, we denote the memory as M. During each iteration, we retrieve the top-20 prompts R(M), based on their accuracy from M and use them as the current memory, denoted as m. Moreover, we consistently include the prompt "a photo of <CLASS>" in our history at every step, as this is a frequently used Below is the image description for each image:Image 0: The image showcases a close-up view of the texture on an object with distinct stripes, buttons and stitching details. Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0, while the accuracy varies from 0 to 100.Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above.Below are the prompts created according to your guidelines: and effective prompt within the CLIP framework [30] . Therefore, our optimization loss is defined as:EQUATIONwhere I indicates the our designed instruction for LLM, T represents the new human-interpretable text prompt optimized by the LLM. Note that our optimizer is parameter-free, which differentiates it from traditional gradient-based prompt learning methods. Instead, we leverage the LLM to optimize the prompt, reducing L CE iteratively until convergence. The input and output example in Figure 2 shows the structured information fed into the LLM, while output demonstrates the optimized prompts generated by the LLM. For more detailed examples of Prompt Optimization Prompt input and output, please refer to the appendix. 
experimental setup: 
 We validate the effectiveness of our approach on the base-to-new generalization benchmark for evaluating prompt learning in vision-language models [8, 11] . Across all experiments, we benchmark the models' performance in a 1-shot and commonly used 16-shot setting. To ensure consistency, all results from learning-based methods are averaged over three random seeds. We use the harmonic mean (H) as the average metric, which is a common approach in prompt learning for vision-language models.Eleven Datasets. We follow CLIP [30] and CoOp [8] to use 11 image classification datasets, i.e., ImageNet [48] and Caltech101 [49] for generic object classification, OxfordPets [50] , Stanford-Cars [51] , Flowers102 [52] , Food101 [17] and FGVCAircraft [53] for fine-grained image recognition, EuroSAT [54] for satellite image classification, UCF101 [55] for action classification, DTD [42] for texture classification, and SUN397 [56] for scene recognition.Six Baselines. To conduct a comparative evaluation, we utilize a number of established baselines including CLIP [30] , Coop [8] , CoCoOp [11] , MaPLe [13] , PromptSRC [28] , and CoPrompt [32] . Note that all methods do not present 1-shot results in their publications, so we perform 1-shot experiments using their available code.Training details. We use GPT-3.5 Turbo as our default optimizer, iterating 100 steps for each dataset to derive the final prompt. At each step, we generate five prompts and compare their accuracy with past prompts, storing the top-20 prompts in our history. Ultimately, we select the prompt with the highest accuracy as the final prompt. For generating image descriptions, we employ MiniCPM-V-2.0 [43] as the default LMM, using the prompt: "[Please generate the description in detail, do not provide the class name in the description.]". We added image descriptions to the 1-shot Prompt Optimization Prompt but not to the 16-shot version due to the character input limitations of GPT-3.5 Turbo, which prevent adding detailed information for each class's images. All experiments were conducted on a GeForce RTX 3090. Our code is available at https: //github.com/lmsdss/IPO.Interpretable prompt analysis. To analyze the importance of specific words or phrases in text prompts, we display a comparative experiment on the Flower102 dataset. Specifically, in Table 1a , we use "a photo of a <CLASS>" as the prompt, which is the most commonly utilized format. By employing occlusion sensitivity analysis, we individually remove each word to test the importance of the remaining words. We find that using just "<CLASS>" as the prompt performs only 0.89% lower on the novel classes than "a photo of a <CLASS>", indicating that the CLIP model can generate more discriminative features using just the category name. Additionally, the prompt "a photo <CLASS>." achieves the best performance. By comparing "<CLASS>" with "photo <CLASS>", we determine that the word photo is particularly significant on the novel classes.In Table 1b , we present the results of the CoOP and CoCoOP models when some learned prompt tokens are removed to analyze which token is most crucial. Surprisingly, for both models, performance improves in novel classes when some or all tokens are removed. This indicates severe overfitting in base classes by these models, where the learned tokens are only applicable to base classes. Removing all tokens allows the models to retain some of the original performance of CLIP on novel classes. Additionally, these methods, which only learn tokens, make it challenging for humans to understand the specific meanings of each token, complicating interpretation.In Table 1c , we also demonstrate the final prompt produced by our model: "Identify the unique visual features of the <CLASS> flower accurately," which achieves a performance of 79.6%, surpassing the original CLIP by 2.9% on novel classes. When comparing this optimal prompt with others, removing any words from it results in worse performance than the original prompt. Specifically, comparing "Identify <CLASS>" with "<CLASS>" reveals that including "Identify" boosts performance by 1.7% on novel classes. This highlights the importance of the word "Identify" in datasets like Flower102. We have included additional analyses on various datasets in the appendix.Overall, the comparative experiments demonstrate that our prompt can be more easily interpreted and understood by humans, while also providing insights into the significance of certain key words Prompts CLIP [30] Base 1 : Comparison of various prompts with occlusion sensitivity analysis across different models on the Flower102 dataset [52] . The shaded areas in the table indicate the original performance of each method. Bold blue refers to the result of the best prompt for each model. The original CLIP model shows particular sensitivity to the word photo. In contrast, the tokens learned by CoOP and CoCoOP affect especially base class performance, while removing these learned tokens improves novel class performance. By contrast, with our interpretable prompt optimization, every word makes a meaningful contribution to both base and new classes. We provide results for more datasets in the appendix.Base 2 : Effect of LLM and LMM choice. We obtain best results with GPT-3.5-turbo as the LLM optimizer, and MiniCPM-V-2.0 for generating image descriptions for the Prompt Optimization Prompt. in the prompt. This understanding can guide us in identifying crucial words that enhance prompt effectiveness.In Table 2 , we analyze the effect of the choice of the LLM optimizer and LMM for generating image descriptions as Prompt Optimization Prompt inputs. We evaluate various models and report their average performance across 11 datasets. Using GPT-3.5turbo as the optimizer results in better prompt generation, especially improving performance on novel classes. The training speed of our IPO is heavily influenced by the computational efficiency of LLMs used. Since PaLM and GPT-3.5 are not open-source, we rely on their respective APIs to generate prompts. Consequently, our training speed depends on the API call latency and the computational complexity of these models. Alternatively, Phi2 and Phi3 are open-source, allowing us to generate prompts directly using their weights. Therefore, for researchers and practitioners seeking faster and more cost-effective training, we recommend utilizing open-source large models for prompt generation. When comparing different LMMs, we found that LLaVA-Llama-3-8B and MiniCPM-V-2.0 performTake a high-quality photo of a <CLASS>.Categorize the <CLASS> shown in the image.Take a well-composed photo of a <CLASS> with optimal lighting, focus, and minimal distractions. Capture the pet's unique characteristics, including expression and posture, to ensure a clear and distinct image.Describe the distinguishing characteristics of the <CLASS> in the image.Identify the unique visual features of the <CLASS> flower accurately.Identify the primary ingredient in the <CLASS> and describe its texture, color, and presentation.Capture a comprehensive range of well-lit, high-resolution images of an <CLASS> from various angles, meticulously showcasing its specific design features with perfect clarity and precision for unparalleled accuracy in aircraft. SUN397A photo of a <CLASS>, a type of large-scale scene. DTD Classify the intricate <CLASS> texture.Analyze the <CLASS> vehicles in the satellite image with state-of-theart algorithms for precise classification and optimal efficiency.Capture a high-quality, well-lit image of a person flawlessly demonstrating the <CLASS> action with impeccable visual representation to achieve unmatched. almost identically on average. However, MiniCPM-V-2.0 shows better performance on novel classes. Before using LLMs, we first use LMMs to generate image descriptions, which are then used as inputs for the LLMs. In terms of computation cost, MiniCPM-V-2.0, with its smaller parameter size, generates descriptions more quickly. Therefore, we recommend this lightweight LMM as the image description generator for more efficient processing. In summary, the text comprehension capabilities of LLMs are crucial for determining the quality of the optimized prompts.Interpretable prompts generated per dataset. Table 3 showcases the diverse prompts generated by our optimizer for each dataset. Our method enhances model accuracy by concentrating on the most relevant attributes, such as those in Flowers102 [52] and Food101 [17] . Consequently, it delivers high-quality text prompts that improve vision-language models. We encourage future researchers to leverage these interpretable prompts in their own downstream tasks.Benefit of image description. In Table 4 descriptions enhances the model's performance. This improvement suggests that IPO, when generating new prompts, can effectively integrate information from the images themselves. As a result, the optimizer is able to produce prompts that are more specific to the data, thereby increasing the relevance and accuracy of the generated content. This highlights the importance of multimodal inputs in optimizing the prompting abilities of vision-language models.Comparison with knowledge bank-based prompt learning methods. Our IPO leverages LLMs to optimize prompts, which conceptually aligns with previous bank-based prompt learning approaches.To assess its relative effectiveness, we compared IPO to traditional bank-based methods, specifically using L2P [57] , which learns to dynamically prompt (L2P) a pre-trained model to learn tasks sequentially under different task transitions. We apply L2P within the visual prompt tuning (VPT) [12] framework, which also learns prompts in the visual space and applies them to few-shot vision- L2P's efficacy in the VLM domain, our IPO method still demonstrates superior results compared to VPT + L2P. This comparison, which we will include in the revised manuscript, reinforces that IPO 's effectiveness is not merely due to a memory retrieval mechanism but also benefits from prompt optimization through LLMs.Comparison with state-of-the-art. surpassing the second-best performer, the original CLIP [30] , by 2.78% in average performance. Other methods do not perform as well as CLIP in novel classes, indicating overfitting to base classes. For instance, CoCoOP [11] performs better in base classes but falls behind the original CLIP by 2.05% in novel classes. In particular, on the most challenging FGVCAircraft [53] , and EuroSAT [54] dataset, the current state-of-the-art model, CoPrompt [32] , performs poorly. This is because methods based on prefixtuning, like CoPrompt [32] , PromptSRC [28] , require substantial amounts of data for training to achieve adequate generalization. Consequently, on more challenging datasets, when data is scarce, it becomes difficult to fine-tune these prefixes effectively. In contrast, our model outperforms other methods, exceeding the second-best by 1.78% in harmonic mean. Additionally, in Table 7 , we compared our method's performance with other traditional gradient-based prompt learning methods on a 16-shot setting across all datasets, where our approach consistently performs well in novel classes. Demonstrating that our approach can mitigate overfitting and generalize better to novel classes. 
conclusion: 
 In this paper, we presented a novel approach to prompt optimization for vision-language models, addressing the limitations of existing gradient-descent-based methods. By integrating large language models for dynamic text prompt generation and optimization, we introduced the IPO system. This system guides LLMs in crafting effective prompts while maintaining a record of past prompts and their performance metrics, offering valuable in-context information. Additionally, we incorporated large multimodal models to generate image descriptions, enhancing the synergy between textual and visual modalities. Our comprehensive evaluation across 11 datasets demonstrated that our method improves the initial accuracy of vision-language models compared to traditional gradient-descentbased prompt learning methods. Most notably, our approach significantly enhances the interpretability of the generated prompts. By leveraging the strengths of LLMs, IPO ensures that the prompts remain human-understandable, thereby facilitating better transparency and oversight for vision-language models. This improvement in interpretability is crucial, as it allows for more effective and trustworthy human-AI collaboration, making vision-language systems more reliable and accessible.Limitation. Our IPO method is primarily designed for few-shot scenarios. However, when dealing with large domain-specific datasets, the need to generate extensive image descriptions, which can lead to substantial computational costs due to the large text inputs required for LLMs. Currently, our model uses an input length of approximately 5,000 tokens. When scaled to larger datasets, the input length may increase to around 50,000 tokens. Using GPT-4 with an 8k context length, the cost for our current input size (5,000 tokens) is approximately 0.15 dollars per input (0.03 dollars per 1,000 tokens). For the expanded input size of 50,000 tokens, the cost would rise to approximately 3.00 dollars per input. If we were to use GPT-4 with a 32k context length, the cost for the 50,000-token input would be approximately 3.00 dollars for the first 32,000 tokens and an additional 1.08 dollars for the remaining 18,000 tokens, totaling approximately 4.08 dollars per input. Since our IPO method requires 100 iterations during training, the costs would multiply accordingly when scaled to large inputs. In future work, we aim to investigate methods for LMM fine-tuning to enable the direct input of both images and text, thereby generating even more sample-specific prompts.Broader Impact. This paper explores the use of an LLM as an optimizer for refining text prompts in vision-language models. We introduce a straightforward yet interpretable approach to prompt optimization, which holds potential for societal impact, particularly in vision-language tasks.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The data and models pose no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original papers that produced the code package and datasets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We report the 16-shot performance of our IPO method across 11 datasets, providing detailed results for the Base, Novel, and H metrics in Table 8 . Our IPO method consistently outperforms all other approaches on the novel classes and the H metric, highlighting its effectiveness in reducing overfitting.We conducted a comprehensive cross-dataset experimental evaluation using the standard 16-shot setting to assess the performance of our IPO method. The results, presented in As demonstrated in Table 10 , upgrading the LLM capacity yielded a similarly positive impact on performance. To explore how a more advanced LLM, like GPT-4, could generate more effective prompts for our model, we conducted additional experiments with both GPT-4 and GPT-4o. Specifically, when we enhanced the LLM to GPT-4o and paired it with the GPT-4o LMM, we observed a significant overall increase in the H-score by 1.77% compared to the initial setup using GPT-3.5-turbo alongside MiniCPM-V-2. This improvement underscores the advantages of employing larger, more capable models, as they facilitate greater task generalization and more robust performance. The findings suggest that scaling up model capacity in both the LMM and LLM components can lead to substantial gains in prompt quality and adaptability across various tasks, indicating a promising direction for further enhancing our model's versatility and effectiveness.In prompt-based vision tasks like segmentation and detection, the design of the text prompt plays a pivotal role. Our task-agnostic method, IPO, can be seamlessly integrated into various vision tasks to optimize text prompts. For example, as shown in Table 13 , we applied IPO to pre-trained semantic segmentation models [58, 59] , where the original prompt used was 'a photo of a [CLASS].' By utilizing GPT-4o as both the LLM and LMM, we generated more effective, context-specific text prompts tailored to the open-vocabulary semantic segmentation task, leading to significant performance improvements. These results highlight the potential of IPO to enhance text prompt design in segmentation tasks, showcasing its adaptability and value. We plan to extend our exploration of IPO to other vision tasks in future studies, aiming to further validate its effectiveness in optimizing prompt construction across a broader range of applications.IPO with GPT-3.5 Turbo, indeed, does not show an improvement on the large-scale ImageNet. This is because ImageNet has a large number of classes and samples, which results in longer LLM input when generating descriptions for each sample. GPT-3.5 Turbo has limited performance in handling long-text inputs. The table 12 shows the results on ImageNet when IPO uses GPT-4o, which has superior long-text understanding compared to GPT-3.5 Turbo. We found that IPO using GPT-4o leads to better performance improvements over other methods as well as a considerable improvement over IPO with GPT-3.5 Turbo. the text prompt. For instance, as shown in Table 13 , we incorporated IPO into pre-trained semantic segmentation models [58, 59] , where the original text prompt was "a photo of a [CLASS] ." Using GPT-4o as the LLM and LMM, we crafted more effective text prompts specifically suited to the open-vocabulary semantic segmentation task, leading to enhanced performance and demonstrating the value of IPO in optimizing text prompts for this application. We intend to further investigate the use of IPO in other vision tasks in future work.The Table 14 compares the performance of GPT-3.5 turbo and GPT-4o across different batch sizes. We observed that as the batch size increases to 128, GPT-3.5 turbo's performance begins to decline due to its limited capacity for handling longer input texts effectively. In contrast, GPT-4o maintains strong performance even at larger batch sizes. However, using extremely large batch sizes with GPT-4o becomes cost-prohibitive. even larger batch sizes could potentially improve performance further, the cost considerations become a critical factor.We evaluated the effect of varying prompt history lengths on model performance, as shown in Table 15 . Our findings indicate that without prompt history, performance declines due to the absence of contextual information, making it challenging for the LLM to converge. As the history length increases, performance progressively improves, with convergence observed at n=20. Although using n=100 yields the highest average performance, the extended input length significantly raises API costs. As a result, we selected n=20 for our IPO, balancing performance gains with cost efficiency.Base Novel H LFA [66] We conducted a comparative evaluation with LFA [66] and PLOT [67] under the same experimental conditions, as shown in Tables 16. Our IPO method consistently outperforms both LFA and PLOT across the benchmarks.In Figure 3 , 4, 5 and 6, 7, 8, we show detailed inputs and outputs of different training steps in our Prompt Optimization Prompt. We observed that each optimized prompt is unique, and both loss and accuracy exhibit a downward trend.To demonstrate that IPO can indeed serve as optimizer for prompt learning in vision-language models, we present the optimization process's loss and accuracy on ImageNet in Figure 9 . As the training steps increase, the loss consistently decreases, and the accuracy gradually improves, proving the optimization capability of IPO. Additionally, IPO not only allows for interpretable prompt generation but also reduces the risk of overfitting during training. Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0, while the accuracy varies from 0 to 100.Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above.Figure 3 : An example of our Prompt Optimization Prompt with input with initial instruction on the Food101 [17] dataset.In Table 17 , we provide detailed prompts for each dataset in 16-shot format. We encourage future researchers to utilize these interpretable prompts in their own downstream tasks.In Table 18 , we provide descriptions of some training samples generated using Mini-CPM-V-2.0 on each dataset, which serve as input for image information in our Prompt Optimization Prompt. Note that we did not use this image information in the 16-shot setup due to the context length limitations of the language model.Our Prompt Optimization Prompt is a crucial component of our optimizer, serving to enhance the performance of the vision-language model by optimizing the prompts through the prompt LLM. Figure 2 displays an example of our Prompt Optimization Prompt. Initially, the instruction in the first segment of the Prompt Optimization Prompt defines the role of the LLM. It introduces two essential tokens, <INS> and <CLASS>, which represent the prompt and category, respectively. The primary function of this instruction is to inform the LLM of its role and the contents of the Prompt Optimization Prompt, enabling a more effective understanding of the Prompt Optimization Prompt. Note that in this section, our model does not involve image information, hence the Prompt Optimization Prompt here lacks statements like "Here is a description of some features of the flowers Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0, while the accuracy varies from 0 to 100.Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above. Below are some previous prompts with their scores. The scores consist of loss and accuracy. The loss value is equal to or greater than 0, while the accuracy varies from 0 to 100.Generate a prompt different from all the prompts <INS> above, with a lower loss and higher accuracy than all the prompts <INS> above. The prompt should begin with <INS> and end with </INS>. The prompt should be concise, effective, and generally applicable to all problems above.STEP 100text prompt:Train a state-of-the-art deep learning model for advanced <CLASS> image classification with exceptional accuracy and minimal loss. in the image" and the subsequent description of the image. In the next section, we will discuss how to integrate image information into the Prompt Optimization Prompt.The subsequent instruction concerns the history of the prompts and their associated scores, which include metrics such as loss and accuracy. Our task follows CoOP [8] , focusing on base classes with few samples. We calculate the loss and accuracy for these base classes using the generated prompts. Initially, we inform the LLM that the following details are past prompts along with their scores, and we clarify the range of values for loss and accuracy. What follows are the historical prompts and scores. This history functions similarly to episodic memory, to prevent the generation of suboptimal prompts while providing in-context information that enhances the creation of better prompts. Additionally, as the history of past prompts grows, we only retain the top 20 prompts based on accuracy to avoid information overload. The selection of these top 20 prompts is determined by their accuracy scores. Moreover, we consistently include the prompt "a photo of <CLASS>" in our history at every step, as this is a frequently used and effective prompt within the CLIP framework [30] .TLet's address problem, a photo of a <CLASS>.Take an awe-inspiring photograph of <CLASS> that beautifully captures its essence with exceptional clarity, vibrant colors, impeccable composition, and mesmerizing details.Capture a well-lit, high-resolution photo of the <CLASS> with optimal focus and minimal distractions. Ensure proper composition and framing to highlight its unique characteristics. Emphasize the pet's distinct traits by capturing its expression and posture accurately.Use a higher resolution camera to capture the vehicle photo <CLASS>..Capture a high-resolution image of a <CLASS> in perfect lighting conditions, ensuring precise focus and a clutter-free background for maximum classification accuracy and to highlight the flower's distinctive features.Categorize the image depicting a delicious and appetizing <CLASS> with remarkable visual qualities.Capture a comprehensive set of high-resolution images of an <CLASS> from various angles, ensuring optimal lighting conditions and precise focus for unparalleled accuracy in aircraft model recognition. SUN397A photo of a <CLASS>, a type of large-scale scene. DTD Capture an image depicting the distinct pattern of <CLASS>. The final paragraph of the instruction specifies the ultimate goal of the task: to generate improved prompts based on the aforementioned instructions and the history of past prompts. The aim is to achieve lower loss and higher accuracy in this vision-language task, and it also outlines the format for the final generated prompt. These three components constitute the entire content of the Prompt Optimization Prompt, each playing a critical role for the LLM. It is important to note that in this Prompt Optimization Prompt, we do not consider the specific content of the images; we merely use images to calculate scores, and the details of the images are overlooked. In the following section, we will utilize a LMM to generate content from images and then incorporate it into the Prompt Optimization Prompt.In Tables 19 20 21 22 23 24 25 26 27 28 29 , we present various prompts analyzed using occlusion sensitivity analysis across different models and datasets. We found that CLIP is particularly sensitive to photos. However, CoOP and CoCoOP exhibit severe overfitting on base classes, leading to poor performance on base classes when some tokens are removed, but improved performance on novel classes. In contrast, our optimized prompts show performance degradation to varying degrees when certain words or phrases are removed, indicating that the key words or phrases in our generated prompts are essential.The cheetah's fur has spots and stripes, its eyes are black with white pupils. It is looking straight into the camera while standing against a clear blue sky background.The image prominently features a large white and red airplane with the Turkish flag logo, flying against a backdrop of blue sky dotted with clouds.The image features a black Audi TT parked outside the dealership, with its silver rims and elegant design.The car in the image is a 2016 Acura TSX, distinguished by its sleek design and distinctive tail lights.The flowers have a combination of pink and white colors with distinct leaves, creating an attractive appearance.The flowers in the image are large, have a yellow center with dark spots and petals that vary from pink to red. They appear vibrant due to their color contrast against natural backgrounds like leaves or rocks on trees behind them.The cat in the image is a light-colored, possibly an orange tabby with big yellow eyes and white whiskers.The image features a small white puppy with brown spots, which is likely to be an American Pit Bull Terrier breed.The food in the image is a delicious pastry with meat and cheese filling, covered by caramelized topping.The image features a colorful and diverse salad with various ingredients such as onions, corn kernels, lettuce leaves (cabbage), peanuts or pumpkin seeds.The aircraft in the image is a large commercial airplane, likely used for passenger transportation. It has an orange and white color scheme with prominent windows along its body to provide natural light inside during flights.The aircraft in the image is a Western Airlines Boeing 737, identified by its distinctive tail fin with "Western" written on it. It has various markings including numbers and letters that are part of their registration or identification system used for aviation purposes.The image showcases a commercial airplane cabin with rows of blue and white chairs, each equipped with cup holders for passengers' convenience.The image captures the grandeur of a monument with an elaborate arch, surrounded by fountains and illuminated buildings at night.The image showcases a close-up view of the texture on an object with distinct stripes, buttons and stitching details.The leaf has a yellowish background with darker spots and irregular black marks, indicating possible disease or damage.The fish, characterized by its light yellow body and distinct blue fins with a black spot on the tail fin's edge.The image features a white chicken with red comb and orange legs, standing on wooden planks surrounded by straw or grass.The image features a black car with four-wheeled design, possibly indicating that it is an automobile.The image captures various cars, each with distinct features such as their color and shape. For instance, one car is red in color while another has a unique design that stands out from the rest of them.A baby in a yellow shirt and blue jeans is actively crawling on the tiled floor, while an adult wearing red pants stands nearby.A female gymnast is skillfully performing on a balance beam, showcasing her athletic abilities as she navigates the challenging obstacle. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper introduces an interpretable prompt optimization method (IPO) for vision-language models. IPO uses large language models (LLMs) to dynamically generate textual prompts, and helps improve both performance and interpretability, which are key advantages over traditional gradient-based methods. The method’s scalability across multiple datasets and improved accuracy for novel classes are notable strengths.

Reviewers appreciated the novel approach of using LLMs for prompt optimization and the clear presentation of the method. However, concerns were raised about the performance drop on base classes, scalability to larger LLMs, evaluation on additional tasks, missing comparisons with some related methods, cross-dataset evaluation, ablation on prompt history length and the computational efficiency of IPO in resource-constrained environments. The authors provided a detailed rebuttal addressing these concerns, including new experimental results and clarifications on limitations. All reviewers appreciated the comprehensive feedback and decided to vote for acceptance in the final recommendation. 

Taking into account the reviews, rebuttal, and discussions, the area chair believes that IPO presents a valuable contribution to interpretable prompt learning for VL models. The paper’s strengths outweigh its weaknesses, and thus, an accept decision is recommended. Authors are reminded to include all suggested changes in the final version.

Reviews: review: paper_summary: paper_summary: The paper introduces a method named IPO for VLMs which uses LLMs to dynamically generate and refine text prompts. The method is to improve the accuracy and interpretability of prompts. Experiments show that it can address issues like overfitting and lack of human comprehension in traditional gradient descent-based methods.
strength_weakness: strength_weakness: LMM seems contribute little to th final performance. Is the reason that used LMM is not strong enough to generate high-quality image caption? Also for different LLMs, the performance gap is not obvious and the author did not test on GPT4 or stronger models. 
The overall performance is not much better than optimized methods like CLIP.  
Could the authors provide more examples to prove the iterpretability advantage of this method?
questions: questions: Will this method be generalized to other vision related tasks?
limitations: limitations: n/a
paper_summary: paper_summary: The paper addresses the challenge of optimizing text prompts for vision-language models, specifically focusing on the interpretability of these prompts. Traditional methods for prompt optimization rely on gradient descent, which often results in overfitting and produces prompts that are not human-readable. This paper introduces a new approach called Interpretable Prompt Optimizer (IPO), which leverages large language models (LLMs) to generate and optimize prompts in a dynamic and interpretable manner. The paper details the design of the IPO framework and provides extensive experimental results across 11 datasets. The findings demonstrate that IPO not only improves the accuracy of vision-language models compared to traditional gradient-based methods but also significantly enhances the interpretability of the generated prompts.
strength_weakness: strength_weakness: The main weaknesses of this paper lies in the experiment can not support the effectiveness of the LLM optimizer, which is the most important contribution of the authods. IPO incorporates the Large Language Model (LLM) as an optimizer to learn an interpretable prompt pool, and everytime a new instance comes, prompts will be extracted from this prompt. This mainstream is similari to knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. Therefore, the experiments should contains:
1) **A fair comparitive with some of the knowledge bank based prompt learning methods  such as L2P [1], AttriCLIP [2], DualPrompt [3]. To exclude the effectivness of the memory retrieval mechanism of IPO.**

2) **The current comparitive experiment merely involves the 1-shot setting in Table 5. Please provides a more comprehensive experiment settings, such as 16-shots and full-sized tuning setting. Besides, the IPO methods do not draw an obvious performance improvement in benmark datasets in the case that IPO has an obviously longer prompt that the learnable method.** 


3) What's more, the IPO does not perform well in the large-scale generic datasets such as ImageNet. This is not intuitive, because the LLM should show a better performance in tghe generic datasets. Lastly, I am not agree with your average metric, because the data size of these 11 benchmarks have significant difference.

4) Table 6 are not referred in the author's paper, and the detailed benchmark is also not elaborated.

[1]  Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149.
[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663.
[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.
questions: questions: Authors explor using LLM as an interpretable optimizer for prompt learning, I think this is a contribution worth a weekly accept score, i.e., 6 scores in NeurIPS rating. However, this score should be build upon all below concers are addressed:
1)  I can tolerate that the current performance are not that remarkable, but this tolerance should be built upon the experiments are fair. To make a fair comparison in the framework, authors should make comparison with some of the knowledge bank based prompt learning methods such as L2P [1], AttriCLIP [2], DualPrompt [3]. So that we can exculde  the effectivness of the memory retrieval mechanism of IPO.

2) Why Table 5 only contains the 1-shot comparive experiments ? Detailed 16-shots or training with more samples setting are needed. 
I will confirm the effectivness of the your IPO if you draw a remarkable performance. If not, just take a breath and provide enough evidence that LLM optimizer is effectiveness, such as sound losses, sound improvement in the training set, sound improvement in the test sets in different settings. I will agrre with the effectivness of the your IPO if the evidence are sound.


[1] Wang Z, Zhang Z, Lee C Y, et al. Learning to prompt for continual learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 139-149. 

[2] Wang R, Duan X, Kang G, et al. Attriclip: A non-incremental learner for incremental knowledge learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3654-3663. 

[3] Wang Z, Zhang Z, Ebrahimi S, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 631-648.
limitations: limitations: Experiments now are not sound and the authors do not provide enough evidnce to prove the effectiveness of IPO.
paper_summary: paper_summary: This paper proposes an interpretable prompt optimizer (IPO) which uses an LLM to iteratively optimize prompt templates that lead to improved zero-shot visual classification performance on CLIP.
strength_weakness: strength_weakness: 1) Line 91, “However,to the best of our knowledge, no existing studies have investigated how LLMs could be used to optimize text prompts within vision-language models”. There are actually related work that employed LLMs for optimizing prompts for visual classification in VLMs which are not covered in this paper, e.g.  
[a] Liu et al. Language Models as Black-Box Optimizers for Vision-Language Models. CVPR’24  
[b] Mirza et al. Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs. ECCV’24  
Analysis and proper comparison to these works should be conducted.
Here [a] also uses an LLM (ChatGPT) to iteratively update the prompt templates for visual classification on CLIP with the good prompts and bad prompts passed as in-context examples. This weakens the novelty of the proposed method. 
2) In Table 5, why is the performance of the proposed method consistently worse than baselines on base classes? How does the method perform in a usual one-shot classification setting (instead of base-to-novel setting)? 
3) In table 1,  the metric “H” should be specified
questions: questions: 1) In the base-to-novel evaluation, 1-shot refers to the setting where one sample of each category in the base classes is provided during training while no sample of the novel classes is used in training? 
2) Line 165, “all four models failed to understand our Prompt Optimization Prompt”, here it should be four models or six models (as the experiments are conducted on six models)?
limitations: limitations: The proposed method has improved performance on novel classes, but suboptimal performance on base classes where the prompts are optimized.
paper_summary: paper_summary: This paper presents an Interpretable Prompt Optimizer (IPO) designed to improve the performance and interpretability of pre-trained visual language models such as CLIP. By dynamically generating textual cues using a large language model (LLM) and combining it with a large multimodal model (LMM) to generate image descriptions, IPO demonstrates better accuracy and interpretability than traditional gradient descent-based cue learning methods on multiple datasets.
strength_weakness: strength_weakness: 1）Line 150 :(2) -> (3)

2）There are no citations throughout the paper Table 6.

3）Lack of cross-dataset experimental evaluation to validate the generalizability of IPO.

4）The paper mentions the use of large language models such as GPT-3.5 Turbo, but does not discuss in detail the computational cost and efficiency of these models during training and inference. For resource-constrained environments, this may be an important consideration.

5）The importance of each component of the IPO is described in Section 5.2, but it lacks in-depth analysis, e.g., it could be mined for some more intrinsic reasons for the rise in points from a regularization perspective.
questions: questions: 1）Will this parameterless training time be shorter for IPO compared to the gradient descent approach?

2）How does the length of the prompt history affect the stability of training convergence？
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a method called Interpretable Prompt Optimization (IPO) for vision-language models.
The goal of IPO is to improve the performance and interpretability of vision-language models by dynamically generating and optimizing text prompts. The paper addresses the limitations of current prompt optimization methods, which often lead to overfitting and produce prompts that are not understandable by humans. IPO leverages large language models (LLMs) to generate effective and human-readable prompts (which is the main advantage of their method). It incorporates a Prompt Optimization Prompt that guides the LLMs in creating prompts and stores past prompts with their performance metrics, providing rich in-context information. Additionally, IPO integrates a large multimodal model (LMM) to generate image descriptions, enhancing the interaction between textual and visual modalities. This allows for the creation of dataset-specific prompts that improve generalization performance while maintaining human comprehension. The paper validates IPO across 11 datasets and demonstrates that it improves the accuracy of existing prompt learning methods and enhances the interpretability of the generated prompts.

Overall, IPO ensures that the prompts remain human-understandable, facilitating better transparency and oversight for vision-language models.
strength_weakness: strength_weakness: - [Major]. While the paper obtains interpretable prompts, the method primarily obtains improved performances for the novel classes, but not for the base classes.  In fact for a large number of the datasets, the base classes performance is significantly low when compared to other methods. I believe that there should be a balance in the performance between the novel and base classes. To improve the paper, the authors are suggested to provide strong justifications explaining this.

- [Minor]. This area has a multitude of papers so it might be tedious to compare with all the methods. However, I would suggest the authors to add a separate section having a discussion on the comparison of their method with more recent prompt-tuning methods such as LFA, PLOT, DFA etc. 

- [Minor]. I would also suggest the authors to extend the discussion of the limitations of the method. Currently, as I see, the method is suitable for few-shot scenarios, but what if one has access to a large number of domain samples? How can this method be made scalable?
questions: questions: Overall, the paper provides a fresh view on prompt-tuning by making the process interpretable. In a way, the paper automates prompt-engineering for a domain by using a LLM as an optimizer. I am going with Borderline Accept, but happy to revisit my scores if the Weaknesses are addressed in the rebuttal.
limitations: limitations: See Weaknesses.

Comments: title: Response to Authors
Thank you for the response.
Thank you for the response.
Thank you for the response.
comment: Thanks for the efforts in the response! My concerns are addressed and I increase my rating accordingly.
Thanks for the experimental results. The weaknesses have been addressed and as promised, I increase my score.
We appreciate your thorough response. The authors have resolved most of my concerns, and the paper can be improved to be 6 rates
We are very grateful to hear that most of our issues have been resolved. “IPO w/GPT-4o, batch size=256” with a performance of (76.81/72.73/74.41) for ImageNet refers to the **1-shot** setting, using **GPT-4o** as our LLM. In contrast, the performance reported in Table 6 (77.83/72.45/75.04) is for the **16-shot** setting, using **GPT-3.5 turbo**. We will clarify these detailed settings in the revised manuscript. If you have any further questions, please feel free to let us know.
OK, most concerns have been addressed. But why does the highest performance of "IPO w/GPT-40, batchsize=256" is (76.81/72.73/74.41) for ImageNet, while in Table 6, the reported performance is (77.83/72.45/75.04). If any different experiment setting is used, please discuss them in detail.
We sincerely appreciate the reviewer's encouragement and suggestions. We will ensure that the revised manuscript includes all the experiments and typos mentioned in our response.
Thank you very much for your encouragement.

1. We will clarify this point in the related work section of our revised manuscript, including a discussion on knowledge bank-based prompt learning methods. Additionally, we will incorporate the comparison experiment with VPT + L2P.

2. We present the 16-shot performance of our IPO method across 11 datasets, showing detailed results for Base/Novel/H metrics in the table below. Our IPO outperforms all other methods on the Novel classes and the H metric, demonstrating its ability to mitigate overfitting. However, due to the word limit in the rebuttal, we cannot include comparisons with all methods here. In the revised manuscript, we will include detailed comparisons with CLIP, CoOP, CoCoOp, MaPLe, PromptSRC, CoPrompt, LFA, PLOT, and VPT + L2P.



| Model    | ImageNet | Caltech101 | OxfordPets | StanfordCars | Flowers102 | Food101 |
|------------|----------|------------|------------|--------------|------------|---------|
| CoOp  | 76.47/67.88/71.92 | 98.00/89.81/93.73     | 93.67/95.29/94.47    | 78.12/60.40/68.13    |97.60/59.67/74.06     | 88.33/82.26/85.19 |
| CoCoOp  | 75.98/70.43/73.10 | 97.96/93.81/95.84      |  95.20/97.69/96.43    | 70.49/73.59/72.01        |94.87/71.75/81.71     |90.70/91.29/90.99   |
| MaPLe | 76.66/70.54/73.47  | 97.74/94.36/96.02     | **95.43**/97.76/96.58    | 72.94/74.00/73.47       |95.92/72.46/82.56     | 90.71/92.05/91.38 |
| PromptSRC  | 77.60/70.73/74.01 | **98.10**/94.03/96.02     | 95.33/97.30/96.30   | **78.27**/74.97/**76.58**  |**98.07**/76.50/85.95     | 90.67/91.53/91.10 |
| IPO   | **77.83**/**72.45**/**75.04**     |  97.32/**95.23**/**96.26**     | 95.21/**98.23**/**96.70**  | 73.42/**75.71**/74.55  | 96.78/**78.32**/**86.58** | **90.92**/**93.08**/**91.99**     | 

| Model    | Aircraft | SUN397 | DTD   | EuroSAT | UCF101 | Average |
|------------|----------|------------|------------|--------------|------------|---------|
| CoOp  | 40.44/22.30/28.75   | 80.60/65.89/72.51  |79.44/41.18/54.24 | 92.19/54.74/68.69  | 84.69/56.05/67.46  |82.69/63.22/71.66 |
| CoCoOp  | 33.41/23.71/27.74   | 79.74/76.86/78.27 | 77.01/56.00/64.85 | 87.49/60.04/71.21 | 82.33/73.45/77.64  |80.47/71.69/75.83  |
| MaPLe  | 37.44/35.61/36.50 | 80.82/78.70/79.75     | 80.36/59.18/68.16    | 94.07/73.23/82.35     | 83.00/78.66/80.77    | 82.28/75.14/78.55 |
| PromptSRC  | **42.73**/37.87/40.15 | **82.67**/78.47/80.52     | **83.37**/62.97/71.75  | 92.90/73.90/82.32       | **87.10**/78.80/82.74     | **84.26**/76.10/79.97  |
| IPO  |  41.21/**41.42**/**41.31** | 81.25/**80.92**/**81.08** | 82.14/**66.81**/**73.69** | **94.25**/**80.11**/**86.61**| 85.32/**80.92**/**83.06**|79.92/**80.51**/**80.21**|






3. You are correct in your understanding. When we use a larger LLM like GPT-4o, its ability to handle longer text inputs allows us to increase the batch size, leading to higher-quality prompt generation. Even when using the same batch size, GPT-4o's text understanding capability is better than GPT-3.5 turbo, resulting in good performance.

For ImageNet, batch training is necessary due to its large number of classes and samples, which require processing multiple instances simultaneously to generate effective prompts. However, this need for batch processing does not apply to other datasets with fewer classes, where single-instance processing suffices.

The table below compares the performance of GPT-3.5 turbo and GPT-4o at different batch sizes. We observed that when the batch size increases to 128, the GPT-3.5 turbo's performance starts to decline due to its limited ability to process longer input texts effectively. However, GPT-4o maintains strong performance even at larger batch sizes. That said, using very large batch sizes with GPT-4o becomes cost-prohibitive, so we selected a batch size of 128 for our experiments. We found that even larger batch sizes could further improve performance, but the cost becomes a key factor. We will include this experimental comparison in the revised manuscript.

| Model            | Batch size | Base   | Novel  | H      | 
|------------------|------------|--------|--------|--------|
| IPO w/ GPT-3.5   | 4          | 73.11  | 68.08  | 70.51  |
| IPO w/ GPT-4o    | 4           | 74.32  | 67.98  | 70.55  |
| IPO w/ GPT-3.5   | 16           | 73.42  | 68.43  | 70.82  |
| IPO w/ GPT-4o    | 16          | 74.94  | 70.75  | 72.78  |
| IPO w/ GPT-3.5   | 32           | 73.79  | 68.72  | 71.16  |
| IPO w/ GPT-4o    | 32          | 75.01  | 70.93  | 72.91  |
| IPO w/ GPT-3.5   | 64          | *74.09*  | *69.17* | *71.54*  |
| IPO w/ GPT-4o    | 64          | 75.34  | 71.23  | 73.45  |
| IPO w/ GPT-3.5   | 128         | 73.67  | 68.07  | 70.75  |
| IPO w/ GPT-4o    | 128         | 76.14  | 72.13  | 74.09  |
| IPO w/ GPT-3.5   | 256         | 73.11  | 67.81  | 70.36  |
| IPO w/ GPT-4o    | 256         | **76.81**  | **72.73**  | **74.71**  |
We appreciate your thorough response. The authors have resolved most of the issues highlighted in the initial review, and we wish to retain the current score.
I am happy with your response. However, some concerns are still not solved.

1. Please add these clarifications to the revised manuscript to make the manuscript clearer:
 In contrast, our IPO primarily focuses on few-shot VLMs, where during training, LLMs are utilized as our prompt bank to generate dataset-specific prompts based on contextual information. During testing, predictions are made directly using the learned prompts without the need for additional retrieval.

2. **Please extend Table 6 to be as detailed as Table 5 because this setting is very important. (If you have space limits, please provide the result of large-scale datasets such as ImageNet)** 

3.  I can not understand why ImageNet results in a longer LLM input when generating image descriptions for its large samples and categories. Is this because of a larger batch size? In my view, it is not necessary to input all the dataset categories when you processing one training sample. Therefore, I am not satisfied with this answer. Please make more clarification.

You are almost close to your 6 rates. Please hold on.