AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 03:22:08

Abstract: Visual question answering aims to provide responses to natural language questions given visual input. Recently, visual programmatic models (VPMs), which generate executable programs to answer questions through large language models (LLMs), have attracted research interest. However, they often require long input prompts to provide the LLM with sufficient API usage details to generate relevant code. To address this limitation, we propose AdaCoder, an adaptive prompt compression framework for VPMs. AdaCoder operates in two phases: a compression phase and an inference phase. In the compression phase, given a preprompt that describes all API definitions in the Python language with example snippets of code, a set of compressed preprompts is generated, each depending on a specific question type. In the inference phase, given an input question, AdaCoder predicts the question type and chooses the appropriate corresponding compressed preprompt to generate code to answer the question. Notably, AdaCoder employs a single frozen LLM and pre-defined prompts, negating the necessity of additional training and maintaining adaptability across different powerful black-box LLMs such as GPT and Claude.
In experiments, we apply AdaCoder to ViperGPT and demonstrate that it reduces token length by 71.1%, while maintaining or even improving the performance of visual question answering.

Main paper: 
 related work 2.1 visual question answering: 
 End-to-end models. In the early phase of VQA research history, a number of neural network architectures designed to process multimodal inputs were introduced. These include a combination of a convolutional neural network (CNN) for visual feature extraction and a recurrent neural network (RNN) for textual feature extraction [13, 24, 41] . Recent models often include attention modules to enhance individual feature extraction for each modality and to combine features of multiple modalities effectively [1, 28, 36, 38] . Large-scale pre-training has become a critical component in improving the performance of these models, enabling them to answer complex questions by implicitly associating words with specific regions in images [20] [21] [22] . More recently, LLMs have been incorporated into VQA frameworks with prompt tuning techniques such as self-prompt tuning [40] . However, these models do not explicitly distinguish between visual processing and reasoning, limiting their interpretability. Some recent studies have focused on techniques to improve interpretability such as causal inference [5] , reasoning path [23] , reasoning prompts [19] and gradient-based explainability method [34] .Visual programmatic models. To improve interpretability and generalizability, VPMs that generate programs to answer questions based on visual input have been gaining research attention. This is a novel approach that leads to more manageable and traceable inferences because the generated programs contain logical sequences that are understandable to humans and articulate a step-by-step methodology for reaching conclusions. Examples of VPMs include ViperGPT [31] , VisProg [12] , and CodeVQA [27] . All of these generate Python programs utilizing image processing APIs, such as object detection, through a frozen LLM. However, generating programs to answer complex and compositional questions requires many APIs and example codes for them. As a result, the length of the input prompt becomes long. To the best of our knowledge, this work is the first to propose adaptive prompt compression for VPMs.Code generation. Extensive research and development in the field of natural language processing (NLP) has led to the creation of LLMs that excel at a variety of NLP tasks. Among these, a distinct group of LLMs is specifically designed for programming code generation, having been trained on large amounts of programs and documents related to programming. For example, Codex [7] , a variant of the GPT-3 lineup, demonstrates its proficiency in multiple programming languages. CodeLlama [27] , which is built on Llama2 [33] and has an expanded code dataset, shows improved performance in handling larger contexts in programming.Most recently, black-box LLMs such as GPT-3.5/4 [4] , Claude1 and Gemini [32] integrate extensive knowledge from a broad spectrum of domains, including programming, allowing them not only to generate code but also to understand and execute complex instructions given by humans. Since their zero-shot performance on programming tasks is remarkably high, they are expected to automate many aspects of coding in future that were previously manual and time-consuming, and are also useful for integration into VPMs for visual question answering. Reasoning and interpretability. Interpretability is an important consideration when integrating LLMs into real-world systems, especially in contexts that require high reliability and accountability. Various prompting techniques have significantly improved the interpretability of LLMs. For example, chain-of-thought prompting [16] , which provides an LLM with a series of contextual examples, enables intermediate reasoning to reach final conclusions. Tree-ofthought prompting [39] constructs a tree structure of thoughts, enriching the decision-making process by branching out various reasoning pathways. VPMs can also be viewed as an extended prompting method that improves interpretability because they show a sequence of logical steps leading to a conclusion by understandable programs. However, these methods also increase the complexity of input prompts because the instructions for LLMs need to be detailed, thus increasing computational costs. Prompt compression. Several strategies have been developed to compress prompts, notably by creating specialized tokens through prompt-based fine-tuning of LLMs [8, 9, 25, 35] , with the goal of minimizing the number of tokens processed during inference. However, fine-tuning of LLMs often limits their generalizability and is not always applicable to black-box LLMs. Other efforts have focused on token reduction. These include token pruning during inference [10, 17, 18] and token merging [3] . However, these methods are generally proposed for small models such as BERT and ViT, and rely on fine-tuning or intermediate inference results. Most recently, Jiang et al. [15] have introduced LLMLingua, which compresses prompts with a small model and feeds the compressed prompts to an LLM. This method can be applied to black-box LLMs because it does not require comprehensive fine-tuning of LLMs.In contrast to these previous studies, this work aims to define and formulate all procedures of prompt compression and inference for code generation with a single frozen LLM to fully leverage the advantages of powerful black-box LLMs. 
adacoder framework: 
 This section introduces AdaCoder, a framework for adaptive prompt compression for VPMs. Figure 2 shows an overview of the AdaCoder framework, which consists of two phases: the compression phase and the inference phase. The compression phase is run only once to prepare compressed prompts, each of which is specialized for a specific question type. The inference phase classifies question type and adaptively selects a compressed preprompt to generate code for visual question answering. Below, we begin with a preliminary formulation of a VPM. We then present each phase of AdaCoder.Notation and settings. We follow the notation used in previous work on VPMs [29, 31] . Let 𝑥 ∈ 𝑋 be an input image and 𝑞 ∈ 𝑄 be an input question about the image, where 𝑋 is a set of images and 𝑄 is a set of questions. VPMs aim to generate a code 𝑧 ∈ 𝑍 that returns the answer 𝑎 ∈ 𝐴 to the question, where 𝑍 is a set of executable codes and 𝐴 is a set of answers.The process of answering questions is divided into two steps: the code generation step and the execution step. The former generates a code asEQUATIONwhere Π : 𝑄 → 𝑍 is a code generation module. The latter executes the code with an input image byEQUATIONwhere Λ : 𝑋 × 𝑍 → 𝐴 is the execution engine. This work utilizes the Python execution engine for Λ. Large language model. To implement the code generation module, a single frozen LLM 𝜋 : 𝑇 → 𝑇 is often used, where 𝑇 is a set of texts 2 . For example, the code generation module Π can be defined byEQUATIONwhere 𝑝 pre ∈ 𝑇 is a preprompt that gives instructions to generate code using image and text processing APIs, 𝑞 ∈ 𝑄 is an input question, and + indicates textual concatenation. Here, APIs include both low-level functions, such as image cropping, and high-level functions, such as object detection.Preprompt definition. In order to provide the LLM with detailed instructions on how to use the APIs, the preprompt 𝑝 pre typically includes API definitions, coding instructions, and example snippets of code. We define a preprompt 𝑝 pre byEQUATIONwhere 𝑝 def is a text of API definitions, 𝑐 ∈ 𝑍 is textually concatenated example snippets of Python code, 𝑝 inst ∈ 𝑇 is a coding instruction written in a natural language, and Ψ is a structural aggregation function to insert code snippets to immediately after function definitions as comments. For example, a code snippet for comparing the positions of objects is inserted immediately after the definition of the object detection function. Below, we review the preprompt of ViperGPT [31] , which we use in Section 4. method at least once. For example, the following code snippet is given for the find method that detects objects in images.# Return the foo def execute_command(image) -> List[ImagePatch]: image_patch = ImagePatch(image) foo_patches = image_patch.find("foo") return foo_patches3) Coding instruction. The coding instruction provides short instructions describing how to write code, specifying a programming language and how APIs should be used. Specifically, 𝑝 inst is the following text:Write a function using Python and the ImagePatch class (above) that could be executed to provide an answer to the query.Consider the following guidelines:-Use base Python (comparison, sorting) for basic logical operations, left/right/up/down, math, etc. -Use the llm_query function to access external information and answer informational questions not concerning the image.Token length. One of major limitations of previous VPMs is that the input token length is long, resulting in a large computational load. This work addresses this limitation by introducing an adaptive prompt compression method. More specifically, we define the input token length of the code generation module in Eq. (3) asEQUATIONwhere |𝑡 | is the number of tokens of a text 𝑡 ∈ 𝑇 . Our goal is to reduce this length.As shown in Figure 2a , the compression phase creates a set of compressed prompts C = { pdef } ∪ { ĉ𝑡 : 𝑡 ∈ 𝑌 },where 𝑝 pre is the original preprompt in Eq. ( 4), 𝜋 is a frozen LLM, and 𝑟 pre is the instruction to rewrite API definitions. Figure 3a shows the definition of 𝑟 pre .Compressing code snippets. This step compresses the code snippets for each question type as follows:EQUATIONwhere 𝑝 pre is the original preprompt, 𝑟 code is the instruction to write code snippets, 𝑟 sp is an additional instruction to write code specialized for a specific question type with a placeholder to insert the definition of question type 𝑑 𝑡 , and 𝑡 ∈ 𝑌 is a question type. Figure 3b and 3c show the definitions of 𝑟 code and 𝑟 sp , respectively.Here, we assumed that a pre-defined set of question types 𝑌 is given. For example, with the GQA dataset [14] , five question types shown in Table 1 are provided with their definitions. Consider the following guidelines:-Make the API reference shortly.-Must include all class, methods, fields, and functions above.-Must include some explanations to make clear the usage of each class, methods, fields, and functions above.(b) Instruction to write code snippets 𝑟 codeWrite some examples of a question and Python code using the API reference above that provides the answer to the question.-Must use all class, methods, fields, and functions above at least one time. -Must refer to code example above.-Must write at least three example. {type_definition[i]} is a placeholder to which a question type definition 𝑑 𝑡 . See Table 1 for example type definitions.The inference phase generates a code to answer the input question by utilizing a compressed preprompt adaptively selected based on the question type as shown in Figure 2b . More specifically, this phase consists of four steps: question classification, preprompt generation, code generation, and execution. Question classification. Given an input question 𝑞 ∈ 𝑄, this step predicts the question type. We define classification prompt 𝑝 cls and use the LLM 𝜋 for question classification as follows:EQUATION)where t is the predicted question type. The classification prompt consists of a short instruction for classification and a list of definitions of question types. The definition of classification prompt is shown in Figure 4 .Preprompt generation. This step generates a compressed preprompt given the question type as follows:EQUATIONwhere pdef ∈ C is the compressed API definitions in Eq. 6, 𝑝 inst is the coding instruction, ĉ t ∈ C is the snippets of code for the question type t, and Ψ is the structural aggregation function in Eq. ( 4). Note that the computational cost of this step is almost negligible because both compressed prompts, pdef and ĉ t , have already been computed in the compression phase. We do not compress the coding instruction 𝑝 inst , because it is already short.Code generation. This step generates a Python code 𝑧 as follows:EQUATION)where ppre is the compressed preprompt.Execution. Finally, the predicted answer 𝑎 to the question is obtained by executing the code as follows:EQUATION)where 𝑥 is an input image, and Λ is the Python execution engine. attr question asking about the attributes or position of an object. (e.g. "What is the color of bar?", "On which of image is the foo?") rel question derived from an affirmative sentence and asking about its subject or object (e.g. "What is the foo next to the baz wearing?", "Is the qux holding a quux?").global question asking about the entire situation of the scene, such as weather or facility (e.g. "Is it foo?").AdaCoder formulation. By substituting Eqs. ( 8) and ( 9) into Eq. ( 10), we can finally define the code generation module Π Ada of AdaCoder as follows:EQUATIONby which a code is generated as 𝑧 = Π Ada (𝑞). The total token length is given byEQUATIONBelow, we discuss the computational cost and adaptiveness. Computational cost. Although, in the first sight, AdaCoder seems computationally more expensive than the conventional code generation module in Eq. (3) because the LLM 𝜋 is called twice in Eq. ( 12); indeed, AdaCoder improves the computational efficiency in practice when a black-box LLM such as GPT or Claude is used for 𝜋 with state-of-the-art VPMs such as ViperGPT, because the input token length is significantly shortened. Compared to previous prompt compression methods such as LLMLingua, our approach is more efficient and effective because it can reduce the token length while preserving the structure of code. We will experimentally demonstrate this in Section 4.2.1. Adaptiveness. A major strength of AdaCoder is that it does not require additional training to adaptively compress the preprompt. Since recent black-box LLMs exhibit quite high zero-shot performance on text processing tasks such as text classification and summarization, AdaCoder leverages these capabilities to enhance efficiency and reduce the computational costs of VPMs. 
experiments 4.1 experimental settings: 
 Datasets. We use three VQA datasets for evaluation: GQA [14] , VQAv2 [11] , and NLVR2 [30] . visual content to generate answers. The NLVR2 dataset is designed to test a model's ability to understand complex natural language statements and their correspondence to a given pair of images.From each of these two dataset, we randomly choose 2,000 QAs3 . Evaluation metrics. We use the exact match accuracy (%) for caseinsensitive answers as a QA performance evaluation metric. The reduction rate (%) of the input token length is used to evaluate the compression performance.Baselines. The baselines are ViperGPT [31] and LLMLingua [15] applied to it. They are state-of-the-art VPM and prompt compression method, respectively. Implementation details. We implement AdaCoder on top of the official implementation of ViperGPT4 . The API set consists of basic image and text processing functions. Specifically, it consists of the ImagePatch class and a set of auxiliary functions. The ImagePatch class is a class to store a image region and has the following nine methods.1) crop, 2) overlaps_with, 3) find, 4) exists, 5) best_text_match, 6) verify_property, 7) simple_query, 8) llm_query, 9) compute_depth.The auxiliary function set consists of the following four functions.1) distance, 2) best_image_match, 3) bool_to_yesno, 4) coerce_to_numeric.Each method or function is provided with its definition in Python and example code snippets. See the Appendix for more details.LLMs. We use GPT and Claude for both ViperGPT and AdaCoder.For GPT, we use gpt-3.5-turbo, released as version 1106, which is trained on data up to September 2021 and is provided by the OpenAI platform. For Claude, we use claude-3-haiku, released as version 20240307. This is a model trained using large amounts of feedback on long document tasks. Note that the original ViperGPT used code-davinci-002 (the GPT-3 Codex [7] ), which was fine-tuned for code generation tasks and is no longer accessible.QA accuracy. Table 2 shows QA accuracy in comparison to the ViperGPT baseline. We see that AdaCoder reduces the input token length by 71.1%, while improving QA accuracy on all of the three datasets. This shows the effectiveness and efficiency of the proposed prompt compression method.The simple compression setup omits the instruction prompt 𝑟 sp to compress for specific question type (i.e., question type classification is omitted). We see that the QA accuracy is significant degraded by this omission, which confirms the effectiveness of our adaptive approach.With LLMLingua, we observed that it cannot maintain the structure of code snippets in the preprompt after compression at a reduction rate of 71.1% (the same rate as ours), resulting in a QA accuracy of 0%. Therefore, the results in Table 2 are given at a lower reduction rate ≃ 25% by adjusting the compression ratio parameter accordingly. With this setting, LLMs can generate executable code with a probability of 98%; however, the QA accuracy is degraded by 2.2 points on GQA. This shows that prompt compression for VPMs is challenging, and that our approach specialized for preserving code structure is effective. Question type classification. Figure 5 shows the confusion matrix of question type classification for the GQA dataset. We observe that two question types, "attr" and "global", achieve accuracies greater than 75%. The types "rel" and "obj" are often misclassified as "attr" and "cat", respectively. This is because the questions are often short, making it difficult to distinguish between them.To investigate how these classification errors affect the final QA accuracy, Table 3 compares AdaCoder using 1) predicted question types, 2) ground-truth question types, 3) random question types, and 4) without using question type based compression. We observe three key findings. First, the best performance is achieved by using ground truth question types. This highlights the importance of classifying question types to improve overall accuracy. Second, the performance drop due to classification errors is less than 1.0 points. This suggests that AdaCoder effectively classified the critical question types necessary for code generation, even though the accuracy for question classification is not very high. Third, the method using random question types, which compresses prompts for each question type and randomly choose one of them in inference, is better than the method without question type based compression. This is because the instruction prompt 𝑟 sp in Eq. ( 7) for specializing code snippets to each question type makes it more likely to provide code snippets that are related to each other, thereby increasing the probability of completing the program. When this instruction is omitted and compression is performed regardless of the question type, code snippets that are effective for any question type tend to be retained after compression. However, this approach results in the loss of some specific snippets that are necessary to complete the program, thereby reducing QA accuracy. These results suggest that the instruction 𝑟 sp is important for compressing code snippets. Compressed prompts. Table 4 summarizes the token length and compression performance for each component of the input preprompt. We see that both API definitions and code snippets are significantly compressed. A comparison of the original and compressed API definitions is shown in Figure 6 . We see that descriptions of methods unnecessary for coding, such as those for the initialization method, are omitted, and the remaining sections are condensed into shorter sentences. This is an effective compression achieved by the language understanding and summarization capabilities of black-box LLMs.Computational time. Since the model weights and details of the black-box LLMs are not publicly available, and API response times can be affected by server congestion, a detailed analysis of computation times is not possible. However, the total time for experiments on the GQA dataset was reduced by 55%.Ablation study. Table 5 presents the results of an ablation study. We see that both compression of API definitions and code snippets contribute to each other for both reducing the input token length and improving QA accuracy. Table 6 summarizes the QA accuracy obtained by using a single compressed prompt. We see that even with one prompt of either "attr" or "rel", our method achieves comparable or slightly better performance than the ViperGPT baseline (41.3%). However, using one prompt of either "obj" or "global", the QA accuracy is significantly degraded. These results demonstrate that our adaptation approach is essential for improving QA accuracy while compressing input prompts. The detailed QA accuracy by question type is analyzed in Table 7 . We see that the four compressed prompt specialized for "obj", "cat", "attr", and "rel" performed the best for corresponding questions. For the "global" questions, the prompt for "attr" was the best. This is because "global" questions are highly varied and not easily categorized. Defining fine-grained QA types would be interesting as a next step in future research.Error analysis. Table 8 shows an error analysis, where we manually counted the occurrence of four types of errors. "Coding error" indicates that the generated program is not executable or returns nothing. "Cannot answer to simple query" indicates that the program is correct but the simple_query method returned a response such as "I cannot answer". "No object detected" indicates that no object is detected by the find method. "Wrong answer" indicates that the returned answer was wrong. We have two observations. First, the predominant type of error was wrong answers, and AdaCoder reduced their frequency. Second, despite AdaCoder's improvement in coding quality, there is still a 7.8% incidence of coding errors. This suggests that there is still room for improvement in instructing LLMs about API usage. Several minor errors were also observed. "Correct but with unnecessary details" refers to responses that were marked incorrect because they provided additional, unnecessary information, such as the response "Yes, there is an apple on the table" where the ground truth is "Yes". "Correct except with articles" refers to cases where the instruction to respond with a single word was ignored and an article was added, resulting in responses such as "a car" instead of "car". "Correct by paraphrasing" refers to errors resulting from the use of interchangeable terms that do not change the meaning, such as using "lady" instead of "woman". Qualitative examples. Figure 7 presents qualitative examples of the generated programs. As shown, few example code snippets related to the input question are adaptively selected. These examples help LLM to generate a program to answer the question. When the program is executed, the object patches are detected and then the relative position or colors are compared to derive a correct answer. 
conclusion: 
 We introduced AdaCoder, a framework for adaptive prompt compression for visual programmatic models. AdaCoder efficiently generated programs for visual question answering by compressing and selecting prompts depending on the question type. A single black-box LLM is effectively employed to perform question type classification, textual compression and code generation, eliminating the need for additional training. In experiments, we demonstrated the effectiveness and efficiency of AdaCoder in comparison to ViperGPT and LLMLingua. Finally, we discuss limitations and future work. Limitations. As this work relies on black-box LLMs, analysis from the perspective of neural network architecture is limited. Alternative choices to LLMs for code generation may include open-source white-box models, such as CodeLlama and StarCoder. However, since AdaCoder requires high-quality text classification and summarization, these models were not suitable for prompt compression. New research directions leveraging the combination of white-box and black-box LLMs need to be further explored. Future work. To advance multimodal automated programming, future research directions that focus on pushing the boundaries beyond the traditional scope of VQA would be interesting. This includes developing methods for interactive code modification to enable a more dynamic and responsive programming environment. Additionally, we plan to explore the automatic extension of APIs to facilitate their evolution in becoming more efficient and effective in addressing the complex requirements of multimodal interactions. We believe that the present work contributes to fostering new ideas for such novel research directions for the multimedia community.  
: 
 and natural language processing [2, 6, 11, 14, 30, 37] . Thanks to the advantages of deep learning techniques, significant progress has been made in VQA over the past decade with end-to-end learning models such as GLIP [22] . However, these models do not explicitly distinguish between visual processing and reasoning, which limits their generalizability and interpretability.To overcome this limitation, several pioneering studies have introduced visual programmatic models (VPMs), models that generate executable programs specifically designed to answer questions, providing a more manageable and transparent inference process [12, 29, 31] . VPMs typically consist of a large language model (LLM) for code generation and a set of APIs for image processing. Given an input question, the LLM analyzes the text to understand the intent and the required computational steps. It then generates a program that, when executed, can manipulate and analyze images by using APIs to produce the desired answer, where the APIs include both low-level modules (e.g., image cropping) and high-level modules (e.g., object detection). VPMs have proven effective and are gaining traction; however, they also face challenges in terms of computational complexity, as long prompts are required to enable the LLM to understand API usage for generating appropriate programs.To reduce computational costs, the development of efficient neural network architectures has been extensively studied. However, these approaches require retraining or additional learning, which is not feasible for application to LLMs trained with huge data, such as GPT and Claude, which we refer to as black-box LLMs. Recently, research has begun to focus on prompt compression [8, 15, 25] , which involves optimizing input prompts to achieve high performance with shorter inputs. For example, LLMLingua [15, 26] compresses prompts using smaller models before using black-box LLMs.Inspired by these studies, we introduce AdaCoder, a framework of adaptive prompt compression for VPMs. More specifically, Ada-Coder operates in two phases: a compression phase and an inference phase. The compression phase generates a set of compressed preprompts, each depending on a specific question type, given a preprompt that describes all API definitions in the Python language with example snippets of code. The inference phase adaptively selects a compressed prompt by classifying the question type and generates a Python program to answer the input question, as shown in Figure 1 . Notably, we implement all of the modules of AdaCoder with a single frozen LLM, which allows implementation with blackbox LLMs. Our contributions are as follows:1) We propose AdaCoder, a novel prompt compression framework for VPMs. It adaptively selects a short instruction for code generation based on question type. 2) We define and formulate all procedures of AdaCoder with a single frozen LLM. This avoids additional training and enables implementation with black-box LLMs. 3) We demonstrate the effectiveness of AdaCoder over the state-ofthe-art ViperGPT [31] model on three VQA datasets with GPT and Claude. We show that the token length of input prompts is reduced by 71.1%, while maintaining or even improving question answering performance. We also show that AdaCoder outperforms LLMLingua [15] in our evaluations. 
 
 Appendix: https://claude.aiThis work assumes that questions, answers, and codes are in text form, i.e., 𝑄, 𝐴, and 𝑍 are subsets of 𝑇 .This is due to the usage limits of Claude and GPT. The list of sampled QA IDs will be provided along with our code.https://github.com/cvlab-columbia/viper 


Decision: The paper presents AdaCoder, an adaptive prompt compression framework for visual programmatic models, which adaptively selects concise instructions for code generation based on the type of query. AdaCoder effectively reduces token length during inference on the experimented VQA tasks.

Reasons to accept: 
The motivation of prompt compression in LLMs and multimodal tasks is valid and clear. The paper is a pioneer work in exploring the compression perspective in Programmatic VQA.

Major concerns:
1. The long-term significance of prompt compression remains unclear. (Reviewer f2r6, FB2n - Q1)

AC agrees that the long-term importance of prompt compression remains unclear given the increasing LLM context window. However, AC thinks we should not penalize novel explorations due to such uncertainty, the discussions in rebuttal L15 provides a valid explanation on why such technique could be useful.

2. The influence of question type (question classification accuracy) on the final performance (Reviewer FB2n - Q2)

AC agrees with the concerns raised in Reviewer FB2n’s final rating justification, that the current method heavily relies on the QA classification, instead of already being a robust compression approach across all problems.
Instead of dodging the problem or overclaiming the effectiveness, authors should comprehensively discuss and analyze such limitations to inspire future works in closing the gap. This may include the oracle performance with best QA results on different tasks to understand the cause of failures, and how different types of problems are affected by the categories.

3. The effectiveness on general VL tasks, beyond VQA (Reviewer MnMj - Q2).

AC thanks authors for adding Table 1 in rebuttal for results on visual reasoning, image editing, visual grounding, counting, relative depth estimation, and object tagging. Authors are encouraged to organize the results into the revision, with the analysis similar to the ones in the main paper and in earlier questions. 

The paper receives the initial rating of weak reject, borderline reject, and weak accept. After the rebuttal, reviewers decided to keep their rating. After reading the paper, reviews, and discussion, AC agrees that the paper has merit and recommends accept, **with the condition that authors address the concerns listed above**, and incorporate corresponding content into the final version.

Reviews: review: paper_summary: paper_summary: The paper titled "AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering" introduces a novel approach to enhancing visual programmatic models (VPMs) by employing an adaptive prompt compression framework. This method, named AdaCoder, operates in two phases: a compression phase and an inference phase. During the compression phase, the system generates compressed preprompts tailored to specific question types. In the inference phase, AdaCoder selects the appropriate preprompt based on the input question, enabling efficient code generation for visual question answering.

Key contributions of AdaCoder include a significant reduction in token length—by 71.1%—without compromising performance. The framework uses a single frozen large language model (LLM) to handle all phases, thus avoiding the need for additional training and maintaining adaptability across different LLMs. Experimental results demonstrate that AdaCoder outperforms existing methods like ViperGPT and LLMLingua in terms of both efficiency and accuracy.

Overall, AdaCoder presents a robust solution for reducing computational load and improving the performance of VPMs, making it a valuable contribution to the fields of computer vision and natural language processing.
strength_weakness: strength_weakness: One of the strengths of this paper is its unique motivation. In the context of the vast number of papers on in-context learning for VQA using large language models (LLMs), this paper stands out with its innovative approach. The authors identify the significant challenge of lengthy input prompts required for LLMs in visual programmatic models and address it with a novel adaptive prompt compression framework. This fresh perspective not only contributes to the existing body of knowledge but also opens new avenues for research in optimizing prompt efficiency and effectiveness in VQA tasks.

Additionally, the experimental validation is thorough and robust. The authors have conducted extensive experiments across multiple datasets, demonstrating that AdaCoder effectively reduces token length by 71.1% while maintaining or even improving performance. The comparative analysis with state-of-the-art methods like ViperGPT and LLMLingua provides compelling evidence of the framework's superiority in terms of both efficiency and accuracy. This comprehensive experimental evaluation reinforces the practical applicability and potential impact of AdaCoder in the field of visual question answering.
limitations: limitations: One limitation of the paper, ironically, also lies in its unique motivation. While the approach of reducing input token length through adaptive prompt compression is innovative and well-executed, its long-term significance is uncertain. As large language models continue to evolve, there is a clear trend toward supporting larger context windows. This raises the question of whether efforts to shorten input prompts will remain relevant. In the future, as LLMs are likely to handle increasingly larger context windows, the necessity and impact of reducing token length may diminish. This potential shift in the development trajectory of LLMs casts some doubt on the lasting value of the proposed solution.
paper_summary: paper_summary: This paper proposes AdaCoder, an adaptive prompt compression framework for visual programmatic models. AdaCoder operates in two phases: a compression phase that generates compressed prompts for different question types, and an inference phase that predicts the question type and uses the corresponding compressed prompt to generate code to answer the question.  By employing compressed prompts, AdaCoder reduces token length while maintaining or improving performance.  Experiments on ViperGPT demonstrate that AdaCoder highly reduces token length while maintaining the performance.
strength_weakness: strength_weakness: 1.	It works well in reducing token length and can be adapted to multiple LLMs.
2.	The motivation is straightforward and clear.
limitations: limitations: 1.	The challenge of long prompts tokens is not clear. Why does this need to be emphasized, and does it cause issues beyond computational complexity?
2.	In the experiments, we found that direct token compression (w/o considering Q types) still results in significant performance degradation. This is not consistent with the authors' claimed conclusion, not that compressing the token still improves performance, but that the prompts to the Q types mitigate the performance degradation.
3.	The authors need to add this experimental setup: add the prompts to the Q types on top of the other existing methods to prove the effect of token compression.
4.	The experiment found considerable improvement in adding prompts for question types, but the focus of the paper is not on the question types. Besides, it seems that the authors' design did not sufficiently distinguish the correct question types (Fig. 5).
paper_summary: paper_summary: This work addresses the issue of excessively long prompts in Visual Programmatic Models (VPMs) caused by APIs and example code. It proposes a solution in the form of AdaCoder, a novel prompt compression framework designed specifically for VPMs.
strength_weakness: strength_weakness: 1. AdaCoder adaptively selects concise instructions for code generation based on the type of query.
2. This approach eliminates the need for additional training and allows for implementation with black-box large language models (LLMs).
limitations: limitations: 1. The novelty of this methodology is limited, as it merely involves compressing and invoking predefined prompt sets using existing models.
2. My major concern about this paper is that focusing this VPM method solely on Visual Question Answering (VQA) tasks limits its contribution significantly. Previous methods, such as Visprog [1], ViperGPT [2], and GENOME [3], have demonstrated their improvements across various multimodal tasks, as the strength of VPM methods lies in their generality and ability to iterate and update [3]. Whether the compression method proposed in the paper can be applied to different multimodal tasks is worth further consideration.
3. In Section 3.2, Do the compressed prompt words have a dependency on the model? It is unclear whether compressing prompts with different models impacts inference. For instance, does compressing with Claude and inferring with GPT affect the experimental results?

[1] Gupta T, Kembhavi A. Visual programming: Compositional visual reasoning without training[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14953-14962.

[2] Surís D, Menon S, Vondrick C. Vipergpt: Visual inference via python execution for reasoning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11888-11898.

[3] Chen Z, Sun R, Liu W, et al. GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs[J]. arXiv preprint arXiv:2311.04901, 2023.