De-fine: Decomposing and Refining Visual Programs with Auto-Feedback

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 04:48:14

Abstract: Visual programming, a modular and generalizable paradigm, integrates different modules and Python operators to solve various vision-language tasks. Unlike end-to-end models that need task-specific data, it advances in performing visual processing and reasoning in an unsupervised manner. Current visual programming methods generate programs in a single pass for each task where the ability to evaluate and optimize based on feedback, unfortunately, is lacking, which consequentially limits their effectiveness for complex, multi-step problems. Drawing inspiration from benders decomposition, we introduce De-fine, a training-free framework that automatically decomposes complex tasks into simpler subtasks and refines programs through auto-feedback. This model-agnostic approach can improve logical reasoning performance by integrating the strengths of multiple models. Our experiments across various visual tasks show that De-fine creates more accurate and robust programs, setting new benchmarks in the field. The anonymous project is available at https://anonymous.4open.science/r/De-fine_Program-FE15

Main paper: 
 introduction: 
 Large visual models [2, 7, 21, 26, 36] based on transformers, excel in various visual tasks, such as object detection [20] , visual question answering [4] , video description [37] , and visual reasoning [35] , by using large-scale unsupervised pre-training and supervised multi-task training. However, these end-to-end models do not essentially reveal internal logic and need fine-tuning for new tasks [27] , which might be costly and challenging for complex and long-tailed tasks. In pursuit of task inference without additional training, interpretable programming-based approaches have been developed, allowing the expression of logic and reasoning for visual tasks through the assembled code modules. Previous works like Visual Programming [8] and ViperGPT [32] , use code-generation models to compose vision-language models (VLMs) [39, 41] into subroutines and assemble a program for visual tasks. ViperGPT, for instance, uses a provided API to access modules and generate executable code. It requires no further training and leverages the expressive power of programming languages, making it effective for solving complex visual tasks.Despite their advantages, current programming-based methods tend to generate lines of atomic code sequentially in a single pass, without properly decomposing the task into smaller manageable . . . subtasks and generating corresponding program blocks separately. Such a manner leads to two main issues: 1) Insufficient hierarchical task decomposition: Failing to plan the program structure in a hierarchical manner, previous works [8, 32] largely struggle to handle complex tasks efficiently, especially for handling logically intricate tasks. This could potentially lead to sub-optimal or hard-tomaintain code and contradict the original design intention for the compositional task. Further, without a clear decomposition of the problem, identifying and fixing bugs becomes daunting as the error could be deeply embedded within intertwined program logic. Modular code, in contrast, enables easier isolation and resolution of issues.2) Ineffective intermediate feedback utilization: Since the program is generated in a single pass, programming-based methods fail to take advantage of intermediate results and system-returned values during code execution that can enhance code quality and facilitate debugging. This indicates a lack of adeptness in leveraging real-time feedback and adjusting code logic dynamically, which could otherwise lead to more refined and debuggable code outputs.Drawing an analogy from the process of human programmers [9, 24] , we consider four essential steps typically followed in program development: 1) Reference: search for the most relevant algorithm that aligns with the given task's logical structure to serve as a highlevel logical reference. 2) Decomposition: based on the reference's structure, decompose and frame the programming task into several subtask modules, later constructing an executable draft program.3) Feedback: obtain systematical feedback for program revision by comparing the expectation with program output, intermediate variables, and compiler return values. 4) Iteration: progressively refine the program based on the feedback until the desired correct outcome. We exemplify the idea in Figure 1 . For the query "How many muffins on the table?", one may first retrieve a similar query (e.g. "How many toys on the desk?") and parse its logic structure as a reference. Then we decompose the task into two subtasks: "find the muffins on the table" and "count the number of muffins". The programmer will further evaluate the output, intermediate results, and the program against the expectation, and finally refine the program iteratively until satisfied.Inspired by this process in software engineering, we propose De-fine, a training-free framework that decomposes intricate tasks into executable program blocks by modeling the logic structure of relevant tasks and automatically refines the program based on multifaceted feedbacks from the execution. Our framework advances in relieving humans from the tedious process of converting ideas into programs. Specifically, as shown in Figure 2 , De-fine generates an abstract logical prompt that reveals the internal task logic without redundancy. This prompt is selected based on semantic similarity to the task query and can imitate the logic of the program after masking. Then, prompted by abstract logical examples, a large language model (LLM) generates executable programs that decompose the tasks for the queries. After execution, De-fine automatically refines the program blocks by multifaceted feedback derived from the program results, intermediate variables, and compiler messages. These systematic feedbacks are summarized by categories through multiple targeted specific models.Importantly, the two core modules of De-fine collaborate well with each other. Mutually, the refinement part can extract applicable codes based on feedback and expand the codebase to a logically well-structured one which will be used as prompts for future tasks. This logical codebase even does not require manual annotation or any ground-truth programs for training. Reversely, the decomposition part also contributes to the refinement part by instructing the program to generate more detailed code that provides richer reference information for feedback. These two components are interdependent and synergistically reinforce each other.Our empirical evaluation on various benchmarks reveals that the proposed method is able to effectively decompose intricate multi-step problems and be adapted to various visual tasks such as grounding [14] , reasoning [35] , and image question answering [1, 12, 23] . By capitalizing on multi-faceted feedback and the capabilities of other multi-modal language models, we achieve stateof-the-art (SOTA) zero-shot results on five tasks without model fine-tuning or supervised training.Overall, our contributions are as follows:• We revisit visual programming as a task of modular programming and optimization through feedback, solving them by the software engineering principles. With De-fine, we break down tasks into executable program blocks and refine them automatically using multifaceted feedback.  
related work: 
 Program generation and self-optimization have seen renewed momentum owing to the incredible understanding and generation capabilities of LLMs. We now discuss previous program generation, recent work in using LLMs for vision, and advances in selfrefinement.Program Generation. Visual program generation is an active research domain that aims to synthesize programs performing vision tasks using neural symbols [28] or Python modules [19, 33] . This approach is based on the assumption [3, 13] that vision tasks are inherently compositional and can be decomposed into atomic perceptual units like lines of code. Yet, complex tasks pose a challenge for this approach, as the generated code is often sub-optimal due to the insufficient semantic understanding of LLMs [15] . In contrast, De-fine can generate well-performance programs with a hierarchical structure.Visual Programming with LLM. Programming-based methods [29, 38, 40] are scalable and interpretable for vision tasks, as they can incorporate any vision or language module with a predefined interface. Additionally, they enable fine-grained image processing and editing through code-level operations. The progress of the program generation model [19] enhances the synthesis of programs for visual tasks without task-specific training. Nevertheless, they still require multiple manually labeled codes as context-learning examples. While De-fine can automatically retrieve relevant examples to assist in program generation.Refinement with Auto-feedback. Even for human programmers, there is no guarantee that the code written on the first try is always accurate. Therefore, we hope the model can provide multifaceted feedback and refine its previously generated program. Previous work like Self-debug [5] uses the error message of the program as feedback to modify the code generated. Self-refine [22] optimizes the output through feedback and refinement iteration. However, this feedback comes from a single modality of text and is only generated from the final result. In contrast, De-fine can provide feedback on variables during code execution and generate feedback types for different variable types, enabling it to handle visual, textual, and error messages.... Figure 3 : The pipeline of abstract logical prompt generation. Initially, we generate sub-steps to address the given query. Subsequently, we retrieve the most relevant code based on the semantic relevance of code comments and substeps. Then, we mask any irrelevant or redundant information in the retrieved code. This AL is finally provided as a prompt to the code generation model. 
methodology: 
 To address the limitations of current programming-based approach in insufficient decomposing and ineffective utilizing feedback, we propose De-fine, a training-free framework for decomposition and refinement, which eliminates the necessity to fine-tune any existing pre-trained models. Our method is shown in Figure 2 refines the program according to the systematic feedback to produce a well-performing code (Section 3.3). Moreover, these self-improved high-quality programs enrich the codebase for future use (Section 3.4). The whole process does not require any additional input and relies solely on the self-optimization of the model.In the context of visual programming, an effective decomposition necessitates fulfilling two criteria: logical problem decomposition and hierarchical code organization. Absent these, the code generated without explicit intention guidance may result in ambiguity or even compilation errors. To address this challenge, we initially dissect the query into natural language sub-steps, providing an explicit logical step. Subsequently, we extract relevant code from a database to refer to its logical structure, forming an implicit abstract code. These steps culminate in the creation of the abstract logical prompt. The prompt significantly improves code synthesis by eliminating irrelevant variables and enhancing the visibility of the internal logic, offering a substantial advantage in the clarity and effectiveness of program generation.Step Generation. For a given query 𝑞, we leverage the zero-shot reasoning capabilities of the LLM to decompose it into sub-steps 𝑄 = {𝑞 1 , 𝑞 2 , . . . , 𝑞 𝑁 } where 𝑞 𝑖 represents the 𝑖-th step in addressing the original problem. This logical step will be embedded into the generated code as comments in the future to provide readability and explicit intentional guidance for the code generation engine.Abstract Code Extraction. Given a textual query 𝑞, we first retrieve 𝐾 code snippets 𝑍 = {𝑧 1 , 𝑧 2 , . . . , 𝑧 𝐾 } where 𝑧 𝑖 represents the 𝑖-th code whose description is similar to 𝑞 from a codebase 𝐵 (for details, please refer to Section 3.4). After getting candidates, we use placeholders (<pad>) to discard the irrelevant parts (e.g. variable names, specific conditions in if statements) and obtain the abstract code Ẑ = 𝑆 (𝑍 ), where Ẑ = { ẑ1 , ẑ2 , . . . , ẑ 𝐾 }. This abstract code does not require meticulous design. By masking irrelevance that is not pertinent to the current task, it guides the model to focus on learning universal solutions or strategies and directs the code generation engine to decompose the tasks hierarchically.For integration, we find the intuition from previous studies [10, 11] that codes with similar natural language descriptions share an analogous logical structure. We go one step further by sorting abstract code based on text similarity to the substeps in the logical step. We construct an abstract logical prompt 𝐴𝐿 = {𝑄, Ẑ } shown in Figure 3 , enabling program generation 𝑧 = 𝜋 (𝑞, 𝐴𝐿) via a generator 𝜋 (GPT-3.5-Turbo). During generation, the model is instructed to insert comments derived from the logical step, thereby enhancing the extraction and feedback of pertinent information. During program execution, De-fine takes advantage of intermediate results and system return values during code execution to enhance code quality and facilitate debugging. To achieve this, we systematically define several types of feedback: 1) Visual Feedback, 2) Textual Feedback, 3) Compile Feedback, and 4) Human Feedback (optional) that can dynamically adjust code logic based on the execution outcomes. These multifaceted feedbacks are generated by corresponding feedback generators, leading to more refined and debuggable code outputs.Specifically, after getting the program 𝑧, we apply an execution engine 𝜙 (𝑧, 𝑥) and a feedback generator 𝐺 (𝜙) to execute 𝑧 on the input image 𝑥. The generator 𝐺 extracts intermediate variables 𝑉 = {𝑣 1 , 𝑣 2 , . . . , 𝑣 𝑛 } (e.g. image patch, string, comment) and generates feedbacks 𝐹 = {𝐹 𝑣𝑖𝑠𝑢𝑎𝑙 , 𝐹 𝑡𝑒𝑥𝑡𝑢𝑎𝑙 , 𝐹 𝑐𝑜𝑚𝑝𝑖𝑙𝑒 , 𝐹 ℎ𝑢𝑚𝑎𝑛 } :Visual Feedback: The execution of the grounding and finding functions in the program 𝑧 return image patches with corresponding bounding boxes. We use a VLM (mPLUG-Owl [39] ) to process the intermediate image variable, generating feedback for dual objectives: 1) Image caption extraction: The VLM captions image 𝑥 and the image patches in 𝑉 with bounding boxes, converting visuals to text to clarify ambiguities in code generation queries. 2) Sub-step verification: It verifies whether the image patches in 𝑉 match the expected results of each substep in 𝑧. For example, if a substep is supposed to crop a face, the VLM checks whether the cropped image contains a face or not and generates Visual Feedback accordingly.Textual Feedback: We use the cognitive capability of the language model (LLaMA [34] ) to provide Textual Feedback for two purposes: 1) Logical question answering: We ask the model to answer logical questions about the text output, like how the intermediate variables 𝑉 deduce the final answer, and whether they match the substep reasoning process. 2) Text summarization: For atomic code, the program may produce verbose and repetitive output strings if there is a loop in the program. For the entire program, we also need to verify whether the reasoning between the steps in the comments is correct. We summarize these string outputs by the language model and generate Textual Feedback from a higher level.Compile Feedback: We first conduct a static analysis to identify potential hazards, including variable shadowing and naming conflicts. During compilation, the compiler identifies and reports any syntax or semantic errors, such as omitted semicolons, undeclared variables, and data type incompatibilities. These notifications serve as Compile Feedback for refining the compilation process and improving the precision of code execution.Human Feedback (optional): De-fine may also cooperate feedback directly from humans for optional. In programming, users can iteratively modify the program alignment with their intentions until get the anticipated outcome. This intention termed Human Feedback, tends to be markedly explicit and facilitates human-inthe-loop inference. De-fine is capable of leveraging human knowledge and expertise, alongside human creativity and heuristic reasoning, to provide high-quality feedback. We show an example of human feedback in Figure 4d .The feedback generated by De-fine above can consolidate the correct steps, clarify the ambiguous parts, and correct the wrong parts simultaneously. This enables the exploitation of the program itself and its intermediate variables, providing multifaceted feedback for the automatic refinement process of the model. We list the prompts required by the model in Appendix D. RefCOCO RefCOCO+ GLIP [18] 55.0 52.2 ReCLIP [30] 58.6 60.5 GENOME [6] 69.2 -ViperGPT [32] 73.1 67.9 De-fine 75.2 70.0In the previous step, we obtained feedback that can integrate information from various sources, such as patches and strings from intermediate variables, result output from the program, and returns from the compiler, into new programs. This feedback is then used to optimize the draft program with the help of De-fine's refiner, which is especially useful for code optimization, as it can enhance the performance and logic of the program. The automatic refiner reuses our previous code generation model 𝜋. Given a query 𝑞, an initial program 𝑧, and feedback 𝐹 , the refiner 𝜋 refine a new program 𝑧 * = 𝜋 (𝑞, 𝑧, 𝐹 ) that improves on 𝑧 in terms of accuracy, conciseness, and logic. The execution engine 𝜙 then takes an input image 𝑥 and the refined program 𝑧 * , then generates a result 𝑟 = 𝜙 (𝑥, 𝑧 * ) as the final output.Unlike rule-based or heuristic-based strategies, our refinement is feedback-oriented, facilitating adaptability across diverse queries and programs through the assimilation of execution feedback. Additionally, it adopts a holistic perspective, thereby optimizing the program in its entirety rather than optimizing isolatedly. Moreover, the process is interactive and iterative, rather than being constrained to a one-off or static execution, thereby excelling in the enhancement of the program via multiple iterations of feedback from the user or the execution.De-fine takes advantage of optimized code that has a consistent and hierarchical structure refined by feedback. This type of code can produce optimal results and facilitate code reuse. By adding the optimized code back to the codebase, we can enhance the quality and reliability of the programs in the codebase over iteration.We revisit the mention presented in Section 3.1 here. During the inference process, if there is no code corresponding to the current query in the codebase, the generated program will be added to it. Otherwise, it means that the updated code needs to be compared with the draft one. At this time, we execute them with auto-feedback as we mentioned in 3.2 and reuse the refiner as a selector and retain one of them.Initialization of the database, we randomly select 20% of all samples within the current testing task to fabricate query-code dyads for code retrieval and generate the program first (De-fine will still infer them later). It is worth noting that De-fine is a training-free framework. Consequently, we do not utilize any training data from any tasks, aiming to circumvent potential misunderstandings or accusations of unfairness that might arise from using training set data or labels. Our approach is meticulously designed to avoid any form of data leakage. Hence, we only used the questions from the test dataset (without labels) as part of our initial dataset to further this goal. We first infer a subset of the testing set, thereby excluding the utilization of training set data. This procedure will eschew annotations for the filtration, as well as abstain from leveraging data from its training dataset, ensuring no data leakage risk. Subsequently, the feedback generated during execution will be utilized to filter these pairs and preserve executable codes. In this way, the codebase can be expanded as samples are continuously inferred, providing more accurate retrieval and more relevant results in subsequent iterations. 
experiments: 
 The De-fine framework exhibits the capability to execute a multitude of visual tasks without training or access to ground-truth data.In this section, we present the experimental setup (Section 4.1) and undertake a comprehensive evaluation across three distinct visual tasks: (1) visual grounding (Section 4.2), (2) compositional visual question answering (Section 4.3), and (3) zero-shot reasoning on image pairs (Section 4.4). Additionally, we have conducted extensive ablation studies to assess the impact of various parameters and components within our framework (Section 4.5).Model Setup. For abstract code extraction, we can replace variables, judgments, and constant strings with placeholders (<pad>) in static code analysis. In practice, during the experiment, we use an end-to-end sketcher [17] to implement the mask operation. For the program generator, refiner, and filter, we adopt the GPT-3.5-Turbo Baselines. We use the following baselines: Visual Programming [8] , the procedure is given 24 contextual instances, run 5 times per execution for majority voting. For a fair comparison, all experiments utilize GPT-3.5-Turbo (1106) as the code generation engine within ViperGPT [32] . Furthermore, the analysis encompasses BLIP-2 [16] , Flamingo [2] , GLIP [18] , ReCLIP [30] , GENOME [6] as additional comparative frameworks.We compare different models on visual grounding tasks on Ref-COCO and RefCOCO+ [14] datasets. This task requires the model to locate the object in an image that corresponds to a given natural language description, as well as to demonstrate the ability of the model to reason about spatial relations and visual features.As the result shown in Table 1 , De-fine achieves a significant improvement over existing models under zero-shot settings. A potential reason for the inferior performance of end-to-end models (e.g. GLIP, ReCLIP) is their lack of an explicit representation of the internal reasoning structure and their inability to leverage modular tools. In contrast, ViperGPT can access modules via a predefined API, but De-fine surpasses it by automatically optimizing the program and refinement. This suggests that De-fine can validate and modify the generated program for better performance based on the feedback from intermediate variables at the refining stage. Furthermore, this result verifies the flexibility of De-fine, which can adapt to different queries and tasks by adjusting the program structure and parameters.De-fine is a novel method specifically designed for complex visual question answering tasks. It can intuitively show the process in which a complex problem is decomposed step by step and continuously improved program by its own feedback. In this section, we demonstrate the effectiveness of our model on three datasets: GQA [12] , OK-VQA [23] , and TallyQA [1] .As shown in Tables 2 and 3 , programming-based methods, despite employing large foundation models, are constrained by their single-pass approach. Different from Visual Programming which takes a majority voting strategy for maximum consensus predictions per query, De-fine decomposes the task into finer-grained subtasks without using any ground-truth data and incorporating feedback from multiple variables in iteration. Despite using the same abstract logical prompt as an example for ViperGPT, De-fine consistently outperforms existing models by a large margin in all VQA tasks. De-fine can also perform error correction on the generated programs. By using the Compile Feedback that we designed, the model can rapidly identify and fix the flaws in the program in the subsequent generation.We extend De-fine to accomplish reasoning on multiple images, not just one. Our model performs well on the NLVRv2 [31] benchmark which involves verifying statements about image pairs, the results are shown in Table 3 .Current visual models can process multiple images as input, yet they treat each image in isolation. The interrelation of different images relies on network fusion, which is affected by the sequence and quantity of images. De-fine synthesizes information from diverse modalities via feedback and offers a comprehensive correction proposal. This enables the model to improve its performance on multi-image tasks substantially.Qualitative Analysis. Figure 4 showcases how De-fine dynamically refines programs by systematical feedback across various modalities. We also display some error cases and generated feedback in Appendix E and F. A notable advantage of our approach is its facilitation of human-in-the-loop programming, enabling direct incorporation of human reasoning and knowledge through feedback. This process fosters a collaborative environment where users impart heuristic insights to the model, which in turn, validates these inputs via programmatic reasoning and outputs, enhancing human-computer interaction.Effectiveness of Individual Components. We perform an ablation study in four configurations of our model on the GQA and OK-VQA tasks (Table 4 ): 0) backbone: only single-pass program generation and execution, 1) backbone + in-context prompt: with decomposition module and the in-context prompt that uses the retrieved sample directly 2) backbone + abstract logical prompt: 
 
 Appendix:  


Decision: This work proposes a training-free framework that automatically decomposes complex tasks into simpler subtasks and refines programs through auto-feedback. The proposed model is evaluated on image question answering, visual reasoning, and grounding, achieving strong zero-shot performance. After rebuttal, all the reviewers are satisfied with author's response, and give high praise to this work due to the innovative methodology, insightful findings and well-written paper. Therefore, my recommendation is Accept with Oral Presentation, and we encourage the authors to incorporate the reviewers' feedbacks into the revised version.

Reviews: review: paper_summary: paper_summary: This paper introduces De-fine, a training-free framework that decomposes complex visual tasks into executable program blocks and refines them using multifaceted feedback. Inspired by software engineering principles, De-fine creates an abstract logical prompt to guide the generation of programs and employs four types of feedback to optimize program quality. The framework achieves SOTA zero-shot performance on various visual tasks without requiring fine-tuning or supervised training.
strength_weakness: strength_weakness: - A training-free framework capable of performing visual reasoning for complex visual tasks.
- Considerable improvements in comparison with baseline models.
- The experiments with visual grounding benchmarks and multiple VQA benchmarks show the effectiveness.
limitations: limitations: - The results from Table 5 show a limited improvement for the Compile-based Feedback Module. It would be better to add an explanation to this phenomenon.
 - At line 659, it is inappropriate to state that the proposed method outperforms existing models by a large margin. We can see the effectiveness of the proposed method from the evaluation result.

Questions
  - If the VLM returns a feedback that the cropped image does not match the query (e.g., a human face), how to update and improve a specific function in code?
  - For the evaluation of Visual Grounding task (i.e., the RefCOCOs benchmarks), what is the initial codebase used to retrieve the code snippets and then the abstract code extraction?
  - Why do we need a human-in-the-loop process as the feedback module when we expect a model to answer a given question automatically? Could this process help the development of the algorithm?


missing references for MLLMs:
  - Bai, Jinze, et al. "Qwen-vl: A frontier large vision-language model with versatile abilities." arXiv preprint arXiv:2308.12966 (2023).
  - Lu, Haoyu, et al. "DeepSeek-VL: towards real-world vision-language understanding." arXiv preprint arXiv:2403.05525 (2024).
  - Wang, Weihan, et al. "Cogvlm: Visual expert for pretrained language models." arXiv preprint arXiv:2311.03079 (2023).
  - Qi, Ji, et al. "CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations." arXiv preprint arXiv:2402.04236 (2024).
  - Lin, Ji, et al. "Vila: On pre-training for visual language models." arXiv preprint arXiv:2312.07533 (2023).
paper_summary: paper_summary: This paper introduces De-fine, a framework for visual programming that decomposes and refines visual programs using auto-feedback. The method leverages a large language model to generate programs based on an abstract logical prompt, and then refines the programs using feedback from program results, intermediate variables, and compiler messages. The proposed approach is model-agnostic and scalable, and can be applied to various visual tasks such as grounding, reasoning, and image question answering. Experimental results show that De-fine achieves state-of-the-art zero-shot performance on several benchmarks without any supervised training data.
strength_weakness: strength_weakness: 1、Innovative and practical methodology: The paper presents a novel approach to visual programming that combines program decomposition and refinement using a large language model. The method is simple yet effective, and can be applied to a variety of visual tasks.

2、Insightful empirical findings: The experimental results demonstrate the effectiveness of the proposed method, showing that it achieves state-of-the-art zero-shot performance on several benchmarks. The paper also includes ablation studies that provide insights into the contribution of each component of the framework.

3、Well-structured review of relevant literature: The paper includes a comprehensive review of related work in visual programming, end-to-end models, and interpretable models. This review helps to contextualize the proposed method and highlights its unique contributions.
limitations: limitations: 1、Inadequate implementation details for reproducing the study: While the paper provides a high-level overview of the proposed framework, it lacks detailed information about specific implementation choices. For example, it would be helpful to know more about how the abstract logical prompt is constructed and how the feedback is summarized by categories through multiple targeted specific models.

2、Limited evaluation and ablation studies for the proposed method: Although the paper includes some ablation studies, they are limited in scope and do not fully explore the effect of each component on the overall performance. More thorough evaluation and ablation studies would help to better understand the strengths and weaknesses of the proposed method.

3、Lack of clarity in exposition: Some parts of the paper could be made more clear and concise. For instance, the description of how De-fine generates an executable program based on an abstract logical prompt could be improved by providing a concrete example. Additionally, the explanation of how De-fine refines the program blocks using multifaceted feedback could be made more precise.
paper_summary: paper_summary: This paper focuses on the visual programming framework and proposes a method to improve the quality of code generation in visual programming by building a high-quality code base and feedback on executing procession, thereby improving the effectiveness of the visual programming framework on various visual tasks.
strength_weakness: strength_weakness: 1. This paper proposes a training-free improvement method for visual programming, a new and powerful multi-visual task framework, improving its performance by improving code quality.
2. The ablation experiment in this paper is very comprehensive, including a variety of visual programming frameworks and a variety of code generation engines.
3. Code error is a very common error in visual programming frameworks. It is good to improve the performance of visual programming by solving this problem.
4. The paper is well-written and reader-friendly.
limitations: limitations: 1. Feedback may not be correct. Will it lead to an accumulation of errors? As noted in Figure 6, modules of vision programming can also have errors. Likewise, feedback models such as LLaMa also have errors.
2. Can the diversity of reference codes be guaranteed? Visual programming itself is a framework for multiple different tasks. One of its great attractions is the creativity of the code, and then the use of VLM combination to achieve multiple different tasks. But is it reasonable for De-fine to retrieve reference code logic in a limited code base? I hope the author can give some analysis of the diversity of the code base.
3. When De-fine executes the program, it needs to go through multiple rounds of feedback to give the final result. Will this lead to a serious increase in time overhead? (The visual programming framework itself has a slower execution speed, including code generation and calls to different visual modules)
4. My biggest problem is the motivation of this work. Although improving the quality of generated code by referring to existing high-quality code is a method that is very consistent with software engineering (line 130), code error is also a very common type of error in visual programming frameworks (statistical results in Figure 6 ). But for such a method, will it affect the generalization of the visual programming framework? When this method encounters new tasks, will the reference in the code base have a negative impact? For example, the current code base is based on the VQA task and the Grounding task, will it have an impact on its performance when reasoning on image pairs is encountered? Therefore, although improving the quality of code generation is a good direction to improve the visual programming framework, whether this approach will have an impact on the generalization of the original method remains to be evaluated.

Others, but not limitations:

In Table 7, I suggest that the code generation engines used by Vipergpt are also indicated (in the Vipergpt author's official code, ChatGPT should be used, so that there can be a more intuitive comparison with line 3 in Table 7).

I may need more explanations for my questions. I may consider changing my score.