ExoViP: Step-by-step Verification and Exploration with Exoskeleton Modules for Compositional Visual Reasoning

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 10:07:09

Abstract: Compositional visual reasoning methods, which translate a complex query into a structured composition of feasible visual tasks, have exhibited a strong potential in complicated multi-modal tasks. Empowered by recent advances in large language models (LLMs), this multi-modal challenge has been brought to a new stage by treating LLMs as few-shot/zero-shot planners, i.e., vision-language (VL) programming.
Such methods, despite their numerous merits, suffer from challenges due to LLM planning mistakes or inaccuracy of visual execution modules, lagging behind the non-compositional models.
In this work, we devise a "plug-and-play" method, ExoViP, to correct errors in both the planning and execution stages through introspective verification. We employ verification modules as "exoskeletons" to enhance current VL programming schemes. Specifically, our proposed verification module utilizes a mixture of three sub-verifiers to validate predictions after each reasoning step, subsequently calibrating the visual module predictions and refining the reasoning trace planned by LLMs. 
Experimental results on two representative VL programming methods showcase consistent improvements on five compositional reasoning tasks on standard benchmarks. In light of this, we believe that ExoViP can foster better performance and generalization on open-domain multi-modal challenges.

Main paper: 
 introduction: 
 Compositional visual reasoning tasks, such as visual question answering or image editing following language instructions, are challenging multimodal tasks that require complex multi-step visual reasoning based on the language instruction. Compositional methods like neural modular networks (Andreas et al., 2015; Hu et al., 2017; Johnson et al., 2017; Hu et al., 2018; Le et al., 2022; Qian et al., 2022) , which translate the complex language instruction into feasible individual visual tasks, has been successful in this task. However, traditional compositional methods require well-designed neural modules for specific datasets, thus struggle in generalization to open domains. In addition, the intermedia embedding and attention among the neural modules can not be improved by introducing supervision signals or feedback, so the performance of these works is limited to the end-to-end training mechanism. Recently, empowered by the advances in large language models (LLMs) such as in-context learning and train-of-thought reasoning (Radford & Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020; OpenAI, 2023; Chowdhery et al., 2022) . recent methods like VISPROG (Gupta & Kembhavi, 2023) and ViperGPT (Dídac et al., 2023) apply LLMs as zero-shot/few-shot planners to solve visual reasoning tasks, i.e. visual language programming. These visual language programming methods leverage off-the-shelf pretrained vision models and compose them step by step according to the reasoning trace planned by LLMs, yielding interpretable intermediate results and highly generalizable reasoning ability.However, despite their merits, current visual programming methods still suffer from challenges due to the failure of the LLM planning or the visual modules, lagging behind the performance of non-compositional models. To analyze the drawbacks, we manually checked 100 randomly sampled failure cases of VISPROG (Gupta & Kembhavi, 2023) on the visual question answering GQA dataset (Hudson & Manning, 2019) . We find that most of the failures can be classified into two categories: (1) around 30% of the failures are due to planning errors: LLM can not parse the language query into a correct solvable program;(2) more than 40% of the failures are due to module error: the visual modules are not able to correctly execute the program. The others (less than 30%) are caused by synonyms (e.g. "woman" vs "lady") or ambiguity in the questions. More details, including statistics and examples of the failure cases, can be found in Appendix C.Motivated by these failure modes, in this work, we introduce EXOVIP, a "plug-and-play" method that uses "exoskeleton" verification modules to verify the reasoning results step by step, thus correcting the module errors and refining the LLM planning traces. In Fig. 1 , we demonstrate how EXOVIP helps correct the two types of errors. Specifically, the verification module contains a mixture of three sub-verifiers, including an image-text matching verifier, an image captioning verifier, and a visual question answering verifier. The verification module validates the correctness of the predictions of the vision modules step by step and calibrates them to correct the module errors. Furthermore, to refine the planning traces, we build a reasoning trace tree based on the verification scores as well as the self-correctness score from LLMs (Pan et al., 2023) , and search through the tree to find the best trace that has the highest score.To demonstrate the effectiveness of EXOVIP, we apply our method to two recent visual programming methods: self-defined programs, i.e., VISPROG (Gupta & Kembhavi, 2023 ) and Python code programs, i.e., ViperGPT (Dídac et al., 2023) . We run experiments on five compositional visual reasoning tasks: compositional image question answering on GQA Hudson & Manning (2019) ; referring expression understanding on RefCOCO and RefCOCO+ (Yu et al., 2016; Kazemzadeh et al., 2014) , natural language for visual reasoning on NLVR (Suhr et al., 2019) , visual abstract reasoning on KILOGRAM (Ji et al., 2022) , and language-guided image editing on MagicBrush (Zhang et al., 2023a) . Experiment results show consistent improvements with the two models on the five tasks.In light of this, we believe EXOVIP can foster better performance on open-world compositional reasoning tasks. To summarize, our main contributions are as follows:• We introduce the "exoskeleton" verification modules for compositional visual reasoning, which verifies the correctness of vision module predictions step by step. • We show how the verification modules are leveraged to correct the module errors by calibrating the module predictions, and to correct the planning errors by tree searching considering both verification scores and LLM self-correctness. • We apply our method on two models and show consistent improvements over five tasks, showing the effectiveness of EXOVIP. 
related work: 
 LLMs in multimodal tasks. LLMs brought great convenience to multimodal tasks with their generalizability and knowledgeability. Generally, there are three ways researchers use LLMs to solve multimodal tasks. Some researchers incorporate additional parameters to adjust LLMs for use in multimodal domains, then fine-tune the model with the LLMs either frozen (Tsimpoukelli et al., 2021; Alayrac et al., 2022; Li et al., 2023b; Gao et al., 2023; Li et al., 2023a; Dai et al., 2023; Zhang et al., 2023d) or unfrozen (Hao et al., 2022; Huang et al., 2023; Peng et al., 2023) . Others take language model as an expert, and mixture it with experts from other modalities, such as vision, speech to collaborate on various kinds of multimodal tasks (Zeng et al., 2023; Zhang et al., 2023c; Liu et al., 2023b) . In this work, we mainly focus on the third way which adopts LLM's planning ability in parsing complex queries. VISPROG (Gupta & Kembhavi, 2023) takes LLM to compose models for queries by generating programs. The strong zero-shot performance of VISPROG on a range of vision-language tasks demonstrates its potential in multimodal tasks involving complex reasoning.ViperGPT (Dídac et al., 2023) leverages LLMs to generate Python code, which composes a set of available modules. MM-REACT (Yang et al., 2023) builds a multi-round, dialogue-based system to call a set of vision experts by designing the prompt of LLMs. However, the performances of these works are hindered by both the parsed planning chain and the visual experts. Inspired by the excellent performance gain from the step-by-step verification (Lightman et al., 2023) , we improve this train of work with additional verification strategies.Compositional multimodal methods. Compositional methods have long been explored to improve neural models' interpretability and reasoning ability. At an early stage, neural module networks (NMN) (Andreas et al., 2015; Hu et al., 2017; Johnson et al., 2017; Hu et al., 2018; Le et al., 2022; Qian et al., 2022) compose neural models to end-to-end differentiable networks. However, the predefined neural modules have limited applications on open-domain challenges, and the intermedia embedding and attention makes it difficult to construct intermedia supervision signals. Recently, the presence of LLMs has made it possible to automatically compose various kinds of finetuned neural models (Zeng et al., 2023; Gupta & Kembhavi, 2023; Dídac et al., 2023; Yang et al., 2023; Liu et al., 2023b) or external tools (Parisi et al., 2022; Khot et al., 2023; Schick et al., 2023; Shen et al., 2023; Lu et al., 2023; Qin et al., 2023) . These works allow us to diagnose the intermedia rationales of the reasoning process. However, human annotation of these intermedia results can be rather timeconsuming. In this work, we make ways to correct errors in the intermedia results without any human intervention.Self-correctness in LLMs. Although LLMs achieve great success in various tasks, there are many errors in LLM-based natural systems (Pan et al., 2023) : hallucination (Li et al., 2023c; Zhang et al., 2023b ), unfaithful reasoning (Golovneva et al., 2022; Ribeiro et al., 2023; LYU et al., 2023) , toxic, biased, and harmful contents (Shaikh et al., 2022) , flawed code. One popular way to fix these errors is to use the LLMs themselves (Madaan et al., 2023; Shinn et al., 2023; Ye et al., 2023; Yan et al., 2023) to obtain feedback, which can be adopted to correct the errors. Motivated by the selfcorrection capability of LLMs in addressing mistakes from LLM-powered natural language systems, some researchers introduce the self-correcting strategy to reduce the reasoning chain in multimodal frameworks. IPVR (Chen et al., 2023) additionally utilizes LLMs to generate the rationale supporting the answer, checks the generated rationale with a cross-modality classifier, and makes sure that the rationale can consistently infer the predicted output. IdeaGPT (You et al., 2023) takes another LLM as a reasoner to get the final answer by summarizing the intermedia results from visual experts. Additionally, the reasoner helps to improve the results iteratively through self-consistency. However, it's intuitive that LLM's self-correction ability would be limited by the LLM itself. In our work, we combine the feedback from LLM and other visual experts to verify the intermedia results and the planned reasoning chain. 
preliminaries: 
 Task Definition. Our work focuses on a set of Visual Compositional Reasoning (VCR) tasks, such as visual question answering, referring expression understanding, visual reasoning using natural language, abstract reasoning, language-guided image editing. These VCR tasks require compositional reasoning about an image input I and a text input T , and predict the output, e.g. answer to a given question, edited images given a language instruction, etc.Visual-Language Programming (VISPROG). VISPROG (Gupta & Kembhavi, 2023 ) is a zero-shot model for the VCR tasks, utilizing LLMs and pretrained vision models. VISPROG first uses LLMs to decompose the complex text description into a sequence of individual operations, then executes each operation by calling various pretrained visual operation models, including object detectors, image captioners, VQA models, image generators, etc. In other words, different vision models are composed in a way that is specified by the LLM to get the prediction. Given the input text T , an LLM transforms it into an executable program P containing a sequence of operations: P = {o 1 , . . . , o n }, where n is the number of operations. Each operation o i can be executed by some symbolic operations (e.g., "crop", "and", "or"), or by calling some pretrained visual models (e.g. CLIP (Radford et al., 2021) , BLIP (Li et al., 2022b) ). The output of operation o i is denoted as a i . The final prediction is derived after we execute all the operations. However, this perspective highlights two key shortcomings of existing approaches: i) module error, the operation models can not predict the answer correctly; ii) planning error, the LLM might generate unfaithful reasoning. 
exovip: exoskeletons with verification and exploration: 
 To address the aforementioned shortcomings, we propose EXOVIP, a framework that adopts exoskeleton verification modules to calibrate the prediction of the execution modules and refine the reasoning path with tree searching. Fig. 1 depicts the overall framework.For each operation o i , we get a set of candidate answers {a i 1 , . . . , a i k }, with confidence scores {p i 1 , . . . , p i k }. Unlike VISPROG, which directly takes the top answer, we use additional verification modules to verify each candidate answer, thus producing verification scores {s i 1 , . . . , s i k }. Then the verification scores s are used to calibrate the original scores, so the errors made by the execution modules can be corrected. Additionally, we use the verification scores to search for a program with high verification scores, in order to refine the execution program P by tree-searching.In this section, we will first introduce the verification modules, and then describe how the verification results are applied to correct the results of execution modules, and to search for the reasoning trace.The verification modules aims to verify the candidate answers {a i 1 , . . . , a i k } given an operation o i . For example, the LOC(nightstand) operation returns a set of candidate bounding boxes containing a nightstand, then the verification module verifies whether each of the returned boxes contains a nightstand and produces verification scores.Our verification module is a mixture of three sub-verifiers, including an image-text matching verifier, an image captioning verifier, and a visual question answering verifier. Each verifier is a pretrained vision-and-language model that is taken off the shelf. The outputs of the three verifiers are combined as the final verification score. Note the verification model does not introduce additional pretrained models, as these verifiers are from the execution modules of VISPROG.Image-text matching verifier calculates the similarity between the whole images and all candidate sentences, which returns the semantic representation of the image-sentence pair. We construct the candidate sentences T ans by filling the template "a photo of" with candidate answers. In this work, we select CLIP (Radford et al., 2021) to calculate the similarity between images and sentences.EQUATIONImage captioning verifier leverages natural language to describe the visual details of the image. We first get the caption of the image C img by BLIP (Li et al., 2022b) . We then construct the descriptions of candidate answers C ans with the template "the image describe". Specifically, for candidate question-answer pairs, we initially transform the pair into a sentence before inserting it into the template. After that, we calculate the sentence semantic similarity (Reimers & Gurevych, 2019) between the captions and the constructed descriptions as the verification score.EQUATIONVisual question-answering (VQA) verifier is more flexible than others, which offers us more opportunities to evaluate the advanced relationships between image and language, such as entailment and factual consistency. Slightly different from the other two types of models, for VQA verifier, we design templates w.r.t. the neural modules. For example, we use "Is there any object in the image ?" for the object detection model, and use "Does this part looks like object ?" for the classification model used in the abstract reasoning task. We determine the verification score by BLIP (Li et al., 2022b) by calculating the difference in answer probabilities Q ans between "yes" and "no".EQUATIONVerification score Having the scores from each individual verification module, we compute their average to get the verification score for each given answer.s ans = avg(s itm ans , s cap ans , s vqa ans )Negative sampling. Empirically, we find that directly applying this verification score does not work well, because the score scales for different kinds of candidates are not well-calibrated. Motivated by recent works in truthfulness (Li et al., 2022a) , commonsense (Ye et al., 2022) , and bias (Ruggeri & Nozza, 2023) , we propose to take the difference of a candidate answer a j with its antonym n j as the final verification score. More specifically, the antonym n j is selected based on the text embeddings from CLIP Radford et al. (2021) , i.e. the word of lowest embedding similarity is selected. For example, the antonym of "nightstand" is "stocking". We then compute the difference of the verification scores of the candidate answer and its antonym, and get the final verification score. Mathematically, given a candidate answer a j , the final verification score isEQUATION)Calibration using verification scores After obtaining the verification scores of all candidate answers S = {s 1 , . . . , s k }, we normalize them as weights and calibrate the candidate predictions.EQUATIONwhere w j is the normalized verification score. More specifically, the verification score controlling the scaling factor (s min , s max are the minimum or maximum of all the candidate scores.s j is re-scaled to w j = sj smin smax smin • (⌧ 1 ⌧ ) + 1 ⌧ , where ⌧ is a hyper-parameterTo correct the second type of reasoning errors, i.e. planning errors, we further apply the verification scores to refine the reasoning trace predicted by LLMs. Motivated by the recent works showing that searching through a combinatorial problem space can greatly improve the performance of LLMs for complex tasks (Yao et al., 2023; Khalifa et al., 2023; Hao et al., 2023) , we introduce our dynamic reasoning trace searching procedure, which takes advantage of both the LLM self-correctness potential and our verification modules.:::::::::::::::::::::::::::::::::::::Tree-based reasoning trace searching (TRS) The reasoning trace searching procedure is represented as a tree structure, where each node of the tree is a reasoning operation. To get the best reasoning trace, we search from the tree using the beam search algorithm (Graves, 2012; Boulanger-Lewandowski et al., 2013; Sutskever et al., 2014) , which has long been proven to be effective in sequence-to-sequence problems. In each step of searching, we consider both the verification scores and the LLM self-correctness scores. More specifically, our trace searching procedure contains :: two three steps. First, in order to generate more diverse reasoning traces to search from, we randomly perturb the in-context examples (i.e. change the order or remove some samples of examples) in the prompt for LLM. Second, after we get the result of candidate neural modules, we sort them according to the ::::::::::: accumulative verification scores and select the top K candidate reasoning traces. Because the verification scores can be very close for the selected K traces, we further use the self-correctness ability of LLMs to reorder the K traces and select the top P from them (P < K). More details of the prompts used for LLM self-correction are included in Sec. E.2. If the verification score is zero at some step, we re-plan the search trace. 
experiments: 
 We set up experiments on the following five tasks. Refer to Appendix E for implementation details.We set up experiments on the following five tasks. Refer to Appendix E for implementation details.Compositional image question answering on GQA. GQA (Hudson & Manning, 2019 ) is a large-scale dataset containing complex reasoning questions about real-world images in MSCOCOstyle (Lin et al., 2014) . Considering the large size of the dataset, in order to balance the cost of LLM API and the diversity of evaluation dataset, we follow the setting of VISPROG Gupta & Kembhavi (2023) and sample a subset from GQA for evaluation. We randomly sample 5 samples from the balanced val set and 20 samples from testdev set of each question type. e.g. "weatherVerify" for judging the weather, "twoCmomon" for judging common attributions of two objects. In summary, there are 102 question types and 2327 questions in our test set.Referring expression understanding on RefCOCO and RefCOCO+. Given a natural language query describing a region in a given image, the referring expression understanding task requires identifying the bounding box of the object in the image being referred to. RefCOCO and RefCOCO+ (Yu et al., 2016; Kazemzadeh et al., 2014) are two standard datasets for this task. We randomly sample 2 samples per type from the test set from RefCOCO dataset and RefCOCO+ dataset. In summary, our test set includes 66 types, e.g. "bicycle", "backpack", and 261 queries.Natural language for visual reasoning on NLVR2. In NLVR2 (Suhr et al., 2019) , given a description of a collection of images, the model needs to justify whether the description is correct or not (binary classification). The task requires dealing with various kinds of linguistic phenomena, like numerical expressions, quantifiers, coreference, negation, etc. In this work, we use the NLVR2 balanced test set for evaluation, which includes 2316 questions and corresponding image pairs.Visual abstract reasoning on KILOGRAM. KILOGRAM (Ji et al., 2022) contains richly annotated tangram puzzles and requires the model to understand the abstract tangram shapes (e.g. dog, bird) and classify them. Specifically, given a textual description and a set of images, the task is to select the image corresponding to the description. This task evaluates the ability to generalize through abstraction, using visually ambiguous stimuli. We conduct experiments using the test set, where the textual descriptions solely contain the whole-shape description, and the images include parts with different colors. The test set contains 1,251 descriptions, with each one paired with 10 images.Language-guided image editing on MagicBrush. This task requires editing an image according to a natural language instruction, keeping the other area of the image unrelated to the instruction unchanged. The MagicBrush dataset (Zhang et al., 2023a) supports various editing scenarios including single-/multi-turn. Considering the accuracy of automatic evaluation metrics and the costs of human evaluation, in our experiments, we only choose the samples involving single-turn image editing to evaluate our method. In total, there are 100 examples in the test set. Following (Zhang et al., 2023a) , we select the CLIP-I and DINO, which measure the image quality with the cosine similarity between the generated image and reference ground truth image using their CLIP (Radford et al., 2021) and DINO (Caron et al., 2021) embeddings.We first apply EXOVIP to VISPROG and show results on the five tasks. Then we apply it to the python-code-based compositional reasoning method ViperGPT to demonstrate its generalizability.Baseline Model We set up the experiments following the settings in the official VISPROG implementation.1 Moreover, we select BLIP-flant5-xxl (Li et al., 2023b) and InstructBLIP-flan-t5-xl (Dai et al., 2023) as additional baselines, which are strong vision-language models incorporating LLMs and pretrained on large vision-language datasets. These baselines have shown strong zero-shot ability on various tasks like image caption and visual question answering.Table 1 : Results of compositional visual question answering on GQA. ::::::::::::Llava-1.5-13b* :: is ::::: tuned :: on ::::: GQA ::::::: training ::::::: corpora, :::and :::::::: evaluated :::: with ::::::::: additional :::::: prompt.Methods Accuracy BLIP2-xxl (Li et al., 2023b) 49.20 InstructBLIP-flant5-xl (Dai et al., 2023) 55.39 Llava-1.5-13b* (Liu et al., 2023a) 74.56 0 VISPROG (Gupta & Kembhavi, 2023) To verify the effectiveness of each component in our method, we run a series of analysis experiments on our method (also in Tab. 1). We have the following observations:a. Negative sampling is key to verification modules. Naively adding the verification modules (line-1) does not work, even making the performance worse. But when we introduce the negative sampling strategy using antonyms to the verification modules (line-2), the performance boost becomes significant. b. Exploration with reasoning trace matters. In line-3, "whole search" means we use LLMs to obtain a set of complete planning traces, then execute all the traces to get the final verification scores, and select the best trace with the highest verification score. The "beam search" strategy (line-4) means we select next step according to current verification scores. While "whole search" helps, "beam search" can further improve the accuracy to 60.57 from 59.17, which indicates the effectiveness of our tree-like step-by-step searching strategy. c. Self-correctness does help but is less significant than verification mechanism. In line-5, We only use LLM self-correctness during trace searching, without using the verification scores. While the result shows an accuracy gain of 2.75 over the original VISPROG, applying both leads to further better performance.Analysis on the sub-verifers. We evaluate the effects of different types of verification modules with the setting of the best demonstration setting. As is illustrated in Table 2 , Different verification modules share similar boost gain, but a mixture of these modules can benefit more.Analysis on the trace-searching strategy. We calculate the verification scores among different samples and plot the distribution of the verification scores in Tab. 2. We find two advances brought by the searching strategy. First, the average of the verification scores significantly improved after we applied our search strategy. Secondly, the variance gets larger after applying the search strategy, which indicates our method can potentially make use of the verification scores to prompt the effectiveness of the reasoning traces.Analysis on invalid programs. We calculate the percentage of failure cases that can not be correctly executed by the program interpreter. We are delighted to find out that our method reduces the error rate from 5.84% to 3.82%, which indicates our method can predict more executable plan routines compared to the baseline VISPROG.Table 3 : Results on RefCOCO and RefCOCO+.IoU Qwen-vl-chat-7b (Bai et al., 2023) 32.54 VISPROG (Gupta & Kembhavi, 2023) 27.28 EXOVIP 31.50 Table 4 : Visual reasoning on NLVR.Methods Accuracy OFA-large (Wang et al., 2022) 58.38 VISPROG (Gupta & Kembhavi, 2023) 67.66 EXOVIP 67.96Table 5 : Abstract reasoning on KILOGRAM.Methods Accuracy CLIP-large (Radford et al., 2021) 27.26 VISPROG (Gupta & Kembhavi, 2023) 24.46 EXOVIP 26.22Table 6 : Image editing on MagicBrush.Methods CLIP-I DINO InstructPix2Pix (Brooks et al., 2022) 84.19 69.60 VISPROG (Gupta & Kembhavi, 2023) 90.82 82.70 EXOVIP 91.27 83.40Baseline Model We adopt the Qwen-vl-chat-7b (Bai et al., 2023) as the baseline. Qwen-vl-chat-7b is a pre-trained large vision-language model that uses Qwen-7B with further training with aligned techniques. Qwen-VL outperforms current SOTA generalist models on multiple VL tasks and has a more comprehensive coverage in terms of capability range.Analysis As demonstrated in Table 3 , although our method can't achieve SOTA (Qwen-VL) on the RefCOCO dataset, it helps bridge the gap between VISPROG and the large vison-language model. While Qwen-VL is built on a LLM with 7 billion parameters, which is trained on trillions of tokens from the corpus, our method assembles a team of experts whose collective parameters total less than 1 billion. We believe our method can be improved with more advanced experts.Baseline Model We take the OFA-large (Wang et al., 2022) as baseline. OFA unifies a diverse set of cross-modal and unimodal tasks in a simple sequence-to-sequence learning framework.Analysis Table 4 shows the results. Although VISPROG exhibits strong complex reasoning ability over the end-to-end model, our method can hardly further improve its performance. We believe this is because we only take VQA modules to solve NLVR problems. The performance of decomposed VQA steps is hindered by the performance of VQA model, especially when there is error accumulation among a sequence of VQA steps.Baseline Model We use the CLIP-large (Radford et al., 2021) as a baseline to test its performance on the text-to-image retrieval task proposed by KILOGRAM.Analysis For our method, given an object, we adopt the LLM to get its possible semantic parts. At the same time, we segment the image into several visual parts. After that, we align the semantic parts with the visual parts to enhance the matching process. In Table 5 , we see the gap between VISPROG and CLIP. Although our method decreases the performance gap, the compositional method still can not achieve SOTA. Since part identification has already been demonstrated to play an important role in human abstraction Tversky & Hemenway (1984) . We believe our method can be enhanced by introducing a better scene segmentation model.Baseline Model We take InstructPix2Pix (Brooks et al., 2022) as a baseline. InstructPix2Pix is a conditional diffusion model trained on GPT3 augmented datasets.Analysis Table 6 and Fig. 4 show the results on MagicBrush. ::::: These :::::: results ::::::: illustrate ::: the :::::::: capability ::of ::: our :::::: method :: to ::::::: enhance ::: the :::::::: similarity ::::::: between ::: the ::::: edited ::::: image :::: and :: the ::::: target :::::: image, ::::::::: signifying :: the :::::::precision :: of ::: our :::::: image :::::: editing :::::::: technique. ::: For :: a :::: more ::::::::::::: comprehensive :::::::: evaluation :: of ::: the :::::: editing :::::: quality, ::we ::::: have ::::::::: conducted : a ::::: case ::::: study. Fig. 4 :::::: exhibits ::::: some :::::::: instances ::::: using ::::::::::: MagicBrush. While noncompositional methods are likely to change unrelated pixels, compositional methods are more con- In ::: this ::::::: section, :::: we :::::: present :::: the :::::: results :: of :::::::: applying :::: our ::::::: method :: to ::: the ::::: open :::::: LLM. :::::::::: Specifically, ::we :::::::::: substituted :::::::::::: GPT-3.5-turbo :::: with ::::::::::::::: LLama2-chat-13b (Touvron et al., 2023) . :::: The :::::::: outcome :: of ::: this ::::::::: substitution :: is :::::::: displayed ::: in Tab. 7 : . ::: We ::: are :::::: thrilled :: to ::::::: discover :::: that ::: our ::::::: method ::: can ::::: yield :::::::: significant :::::::::::improvements :: in ::::: open ::::: LLM.To demonstrate the generalizability of our method, we apply our method to another compositional method, ViperGPT, which composes available modules by generating Python codes. We equip ViperGPT with our method and test its performance on the GQA dataset. We show the results in Table 8 . We find the performance boost is less significant than which on VISPROG. We analyze this due to ViperGPT provides a few examples in the demonstration and it turns the parameter of the code-generation model to make it deterministic to generate subroutines. In other words, ViperGPT benefits little from our reasoning trace-searching strategy. 
conclusion: 
 In this work, we identify two key types of errors in existing compositional methods: planning errors and module errors. To address these errors, we introduce an innovative verification framework EXOVIP. This framework verifies the correctness of vision module predictions. It corrects module errors by calibration and refines the planning process through tree searching. During this process, it considers both verification scores and the self-correctness of LLM. Applying the EXOVIP to two existing models, we achieve significant performance improvements across five different tasks. The results reinforce the promise and potential of EXOVIP on various open-world compositional reasoning tasks, marking an important milestone in the realm of multimodal tasks involving complex reasoning. 
: 
 ABSTRACT Compositional visual reasoning methods, which translate a complex query into a structured composition of feasible visual tasks, have exhibited a strong potential in complicated multimodal tasks like visual question answering, languageguided image editing, etc. Empowered by recent advances in large language models (LLMs), this multimodal challenge has been brought to a new stage by treating LLMs as few-shot/zero-shot planners, i.e., visual-language programming (Gupta & Kembhavi, 2023) . Such methods, despite their numerous merits, suffer from challenges due to LLM planning mistakes or inaccuracy of visual execution modules, lagging behind the non-compositional models. In this work, we devise a "plug-and-play" method, EXOVIP, to correct the errors at both the planning and execution stages through introspective verification. We employ verification modules as "exoskeletons" to enhance current vision-language programming schemes. Specifically, our proposed verification module utilizes a mixture of three subverifiers to validate predictions after each reasoning step, subsequently calibrating the visual module predictions and refining the reasoning trace planned by LLMs.Experimental results on two representative vision-language programming methods showcase consistent improvements on five compositional reasoning tasks on standard benchmarks. In light of this, we believe EXOVIP can foster better performance and generalization on open-domain multimodal challenges. 
 
 Appendix: Becasue VISPROG doesn't release their sampled evaluation subset, we do sampling following the VIS-PROG paper and evaluate all the methods on our sampled evaluation set.https://anonymous.4open.science/r/ExoViP-5514 


Decision: The paper introduces ExoViP, a "plug-and-play" method to correct errors in both the planning and execution stages of compositional visual reasoning through introspective verification. The proposed verification module, consisting of three sub-verifiers, validates predictions after each reasoning step and refines the reasoning trace planned by LLMs. The reviewers acknowledge the clear writing, the novelty of the approach, and the extensive experiments demonstrating consistent improvements across various compositional reasoning tasks. Although some concerns were raised regarding the efficiency of the method, the inconsistencies in the number of tasks mentioned, and the lack of clarity in certain aspects of the algorithm and hyperparameter evaluation, the overall consensus is that the paper makes a valuable contribution to the field. The AC believes that the negative sampling part is acceptable, given the explaination of the authors from the rebuttal. Considering the strengths and the potential for improved performance and generalization on open-domain multimodal challenges, I recommend accepting this paper.

[comment from the PCs] The term "antonym" has a very specific linguistic definition, which does not align with how it's used in this paper. This led to misunderstandings by the reviewers, which significantly overshadow the evaluation of the paper. What more, using this term does not help the paper, it just hurts it. For the next version of the paper, we ask the authors to replace this term with something more appropriate. This is critical to meet publication standards.

Reviews: review: paper_summary: paper_summary: The paper presents an "exoskeleton" approach to compositional visual reasoning as an amelioration to the visual programming work. Considering the planning error and the module execution error in the visual reasoning process, this work introduces three verifiers and a series of other mechanisms (TRS, self-correction) to enhance the performance in GQA and many other reasoning tasks.
strength_weakness: strength_weakness: + The introduction of 3 verifiers, together with TRS and self-correction is supposed to cause extra latency at inference. So I am skeptical about whether the gains in performance are worth it.
+ The author(s) claim that the proposed exovip framework can mitigate the planning error and present an error analysis in Appendix, which is informative. However, I suggest that presenting the accuracy of planning as an important intermediate step would be more intuitive.
+  The introduction of 3 verifiers lacks a clear motivation and we have little idea why they should be designed in this way. Maybe a more in-depth analysis of how the verifiers collaborate with each other would be helpful.
questions: questions: See the weaknesses above.
paper_summary: paper_summary: This paper introduces a new plug-and-play approach called 'EXOVIP,' to improve compositional visual reasoning and address two challenges in current visual programming: module prediction error and planning error.
"Exoskleton" verification modules (that consists in a mix of image-text matching verifier, image captioning verifier and VQA verifier) are used to verify the prediction after every step, followed by corrections or refinings.
The experiments conver six compositional visual reasoning tasks covering real and abstract images/videos, and shows consistent improvement.
strength_weakness: strength_weakness: There's a concern about the efficiency of the method, as Table 9 shows that the total inference time of EXOVIP is 9.6x the time of LLaVA-1.5-7B. The authors indicated that Internet latency may be a cause, it could be helpful to have an estimate using open-source LLMs.

There's inconsistencies in the paper about the number of tasks and it's quite confusing: sometimes the paper refers to "the outcome has been a uniform enhancement in performance across six diverse tasks", whereas at multiple other places (including in the abstract and appendix) the paper refers to "consistent improvements on five compositional reasoning tasks".
questions: questions: 1. Could you add a short definition of the prediction error and planning error or a reference to the definitions in the caption of Figure 1, so that the very first figure is more self-contained & easy to understand?

2. It could be clearer for the reader if the authors explicitly specify "NLVR2" in the caption of Table 4, as specified in Section 5.3.

3. Typo in 5.3, paragraph "Text-guided Image Editing": "Appendix ??"

Not a question, but I appreciate the analysis about errors/efficiency conducted in the Appendix.
paper_summary: paper_summary: The work aimed to tackle the planning mistakes and execution errors from existing LLMs.
The proposed method, ExoViP, corrects errors at both the planning and execution stages through introspective verification.
ExoViP employs a verification module with three sub-verifiers to validate predictions and refine the reasoning trace planned by LLMs.
Experimental results show consistent improvements on five compositional reasoning tasks on standard benchmarks.
ExoViP has the potential to improve performance and generalization on open-domain multimodal challenges.
strength_weakness: strength_weakness: There are not many serious weaknesses in the paper.
1. Things to improve include the fact that Figure 1 is not very comprehensive; some of the terms, e.g., LOC, VQA[BOX], are unexplained in the beginning. Also, the writing in section 4 should link to Figure 1 to explain each component regarding its target in the pipeline. It would be better to include a high-level figure at the beginning.
2. Another weakness is that the proposed verification is simply a combination of existing works. Also, the idea is in line with the motivation of the Tree of Thoughts, where performance gain is expected.
3. Some typos need to be fixed, e.g., Appendix ?? at section 5.3
questions: questions: Overall, it is a decent paper. Please address the weakness above.
paper_summary: paper_summary: The paper proposes two plug-and-play components to alleviate limitations in existing vision programming approaches with LLMs for compositional visual reasoning. 
First, to improve accuracy of visual execution modules, it reuses visual modules in visual programs as additional verification modules. Negative sampling is proposed to calibrate scores of the verification modules. 
Second, to reduce LLM planning mistakes, it uses tree-based reasoning trace searching and selects the best path. 
Experimental results on five tasks demonstrate the effectiveness of the proposed method over the baseline VisProg.
strength_weakness: strength_weakness: 1. The negative sampling for score calibration is a bit strange. It uses an “antonym” for each answer for calibration. However, the way of selecting antonyms is simply using cosine similarity of text embeddings. Why does it make sense? It seems strange that the antonym of “night- stand” is “stocking”.

2. The algorithm to reduce planning mistakes is not clear. For example, how does the model sample K steps? How do the verification modules measure the plans?

3. Some ablations are missing.
- The hyper-parameters are not evaluated, such as the scaling factor \tau of the verification scores, the number of selected steps K and P.
- The proposed algorithm 1 should compare with simple approaches such as aggregating multiple paths from LLMs with different initial seeds etc.

4. The presentations of table 1 and 2 are confusing.
- The names of different variants of the EXOVIP model are not straightforward to know what the variant is. The textual descriptions sec 5.1.1 help understand these variants but it is still less clear for Line 3 and 4. Line 3 in Table 3 is a model without TRS but the description in Page 7 says it is a model simply with TRS. Similar for Line 4. 
- The numbers in table 2 do not match any numbers in Table 1. Why?
questions: questions: See reasons to reject.

Comments: title: Response to Reviewer MW2q
Response to the authors
comment: I have read through the reviews and rebuttals. 
Most of my concerns are addressed.
I'll keep my rating.
Thank you for your response. Could you please specify the reason you are not convinced? Negative sampling is a long-standing research topic, and in addition to the works we cite in our paper, there are many related studies such as negative sampling [1-5], adversarial sampling [6-9], and additional data-enhanced sampling in information retrieval [10-14]. One of the main designs of our framework is adopting negative sampling to normalize the confidence score for tree-based search across different tools and agents.  Our experiments also demonstrate its effectiveness.

Regarding the presentation, we have carefully addressed your concerns. Could you please specify any remaining points of confusion? Your suggestions are invaluable for us to further improve our work.

[1] Hard Negative Mixing for Contrastive Learning. 

[2] Bundle Recommendation with Graph Convolutional Networks. SIGIR(2020)

[3] Supervised Contrastive Learning. NIPS(2020)

[4] Hard-Negatives or Non-Negatives? A Hard-Negative Selection Strategy for Cross-Modal Retrieval Using the Improved Marginal Ranking Loss. ICCV(2021)

[5] Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings. EMNLP(2022)

[6] Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation. 

[7] Adversarial training regularization for negative sampling based network embedding. Information Sciences(2021)

[8] Adversarial Caching Training: Unsupervised Inductive Network Representation Learning on Large-Scale Graphs. TNNLS(2021)

[9] Instance-wise Hard Negative Example Generation for Contrastive Learning in Unpaired Image-to-Image Translation. ICCV(2021)

[10] Leveraging Social Connections to Improve Personalized Ranking for Collaborative Filtering. CIKM(2014)

[11] Social Recommendation with Strong and Weak Ties. CIKM(2016) 

[12] Joint Geo-Spatial Preference and Pairwise Ranking for Point-of-Interest Recommendation. ICTAI(2017)

[13] A Personalised Ranking Framework with Multiple Sampling Criteria for Venue Recommendation. CIKM(2017)

[14] Socially-Aware Self-Supervised Tri-Training for Recommendation.
Thank you for your useful comments and responses. As identified in our introduction, our proposed framework is generalizable and training-free, thus we believe such general adaptation capability can mitigate the minor inference overhead as indicated in response to **W1**. Moreover, since our LLMs are mainly used for tool-use, some lightweight tool-based LLMs can further reduce the inference overhead and strengthen the performance gain.
Thanks for the rebuttal. I do not find the explanation for negative sampling convincing, and the presentation of the paper should be largely improved to make it easier to understand the technical details and results. Therefore, I kept my original rating.
Thanks to the authors for the responses and clarification. I will keep my current score for now, given that the potential gains seem to be mitigated by the overhead in inference time.