ExoViP: Step-by-step Verification and Exploration with Exoskeleton Modules for Compositional Visual Reasoning
Authors: Yuxuan Wang
Alan Yuille
Zhuowan Li
Zilong Zheng
Venue: COLM (2024)
Keywords: Compositional Methods, Verification, Muiltimodal Agent, Compositional Visua Reasoning
TL;DR: We propose a new framework ExoViP, aims to sove module error and plan error in compositional methods or agent systems

Abstract: Compositional visual reasoning methods, which translate a complex query into a structured composition of feasible visual tasks, have exhibited a strong potential in complicated multi-modal tasks. Empowered by recent advances in large language models (LLMs), this multi-modal challenge has been brought to a new stage by treating LLMs as few-shot/zero-shot planners, i.e., vision-language (VL) programming.
Such methods, despite their numerous merits, suffer from challenges due to LLM planning mistakes or inaccuracy of visual execution modules, lagging behind the non-compositional models.
In this work, we devise a "plug-and-play" method, ExoViP, to correct errors in both the planning and execution stages through introspective verification. We employ verification modules as "exoskeletons" to enhance current VL programming schemes. Specifically, our proposed verification module utilizes a mixture of three sub-verifiers to validate predictions after each reasoning step, subsequently calibrating the visual module predictions and refining the reasoning trace planned by LLMs. 
Experimental results on two representative VL programming methods showcase consistent improvements on five compositional reasoning tasks on standard benchmarks. In light of this, we believe that ExoViP can foster better performance and generalization on open-domain multi-modal challenges.

Published as a conference paper at COLM 2024
ViP: Step-by-step Verification and Exploration with
Exoskeleton Modules for Compositional Visual Reasoning
Yuxuan Wang! Alan Yuille Zhuowan Li
Zilong Zheng 1x
1 State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China 2 Johns Hopkins University, Baltimore, Maryland, USA
vangyuxuan @bigai. (ayuillei, z1i110) edu
zlzheng@bigai.co ai
Abstract
Compositional visual reasoning methods, which translate a complex query into structured composition of feasible visual tasks, have exhib- ited strong potential in complicated multi-modal tasks. Empowered by recent advances in large language models (LLMs), this multi-modal chal- lenge has been brought to a new stage by treating LLMs as few-shot/ / zero- shot planners, i.e., vision-language (VL) programming (Gupta & Kemb- havi, 2023). Such methods, despite their numerous merits, suffer from challenges due to LLM planning mistakes or inaccuracy of visual execu- tion modules, lagging behind the non-compositional models. In this work, we devise a "plug-and-play" method, EXOVIP, to correct errors in both the planning and execution stages through introspective verification. We employ verification modules as "exoskeletons" to enhance current VI. pro- schemes.Speccficallyy a mixture of three sub-verifiers to valiclate predictions after each reasoning step, subsequently calibrating the visual module predictions and refining the reasoning trace planned by LLMs. Experimental results on two repre- sentative VL programming methods showcase consistent improvements on five compositional reasoning tasks on standard benchmarks. In light of this, we believe that ExoVIP can foster better performance and general- ization on open-domain multi-modal challenges.
gramming schemes. Specifically, our proposed verification module utilizes
Code https: httpa////tthubb.cmbbiaaai
1 Introduction
Compositional visual reasoning methods, due to their interpretability and generalization over complex vision-language (VL) challenges that demand intricate, multi-step visual rea- soning guided by linguistic input, have long been the focus for many researchers. Tradi- tional compositional techniques, exemplified by neural modular networks (Andreas et al., 2015; Hu et al., 2017; Johnson et al., 2017; Hu et al., 2018; Le et al., 2022; Qian et al., 2022), have shown success in breaking down intricate language instructions into manageable vi- sual tasks. However, they tend to falter when confronted with the need for broader gen- eralization across diverse domains. Furthermore, the limitations of these approaches man- ifest in their inability to enhance the interaction and attention between neural modules through supervision or feedback mechanisms, thereby constraining performance to end- to-end training paradigms. Recent advances in large language models (LLM) (Radford & Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020; OpenAl, 2023; Chowdhery et al., 2022) have led to novel methods that harness LLM as zero-shot or few-shot planners ad- dress visual reasoning tasks, notably vision-language programming (VIsPROG) (Gupta & Kembhavi, 2023) and ViperGPT (Dídac et al., 2023). These approaches make use of read- ily available pretrained vision models and systematically assemble them, guided by the reasoning trace provided by LLMs, resulting in interpretable intermediary outcomes and highly adaptable reasoning capabilities.
Corresponding authors.

---

Published as a conference paper at COLM 2024
Compositional Visual Question Answering
Question
Ground-truth
both the coffee table that looks lan and the nightstand to the right of the IVV made of wood?
Visual Programming (Gupta Kembhavi, 2023)
Answer:
X
Planning Error
BOX3 =
BOX1
BOX2
A2 VQA[BOX0]
Answer =
LOC
CROP_ RIGHT[[OOO]]
VQA[BOX0]
LOC
CROP
CROP
LOC
what does the coffee table look like?
what material is the nightstand made of?
"wood"
tan coffee table
TV
night stand
and A2 "wood"
Prediction Error
small
wood
Correct it!
EXOVIP (Ours)
Answer: yes
verify
vertfy
verity
verily
Re-plan
Verify with ox-exeert
VQA BOXo]
VQA[BOX3]
Image-ted teat matching verilier
what does the coffee table look like?
what material is the nightstand made of?
A1 == "wood" and A2 = "wood"
0.55
image captioning verifier
VQA
wood
wood
yes
Figure 1: An overview of EXOVIP. The prediction after each step is verified by the pro- posed "exoskeleton" verification modules, which contain mix of three sub-verifiers. The verified scores help correct the errors in the vision module predictions or refine the reason- ing programs planned by LLM.
Despite their merits, current visual programming approaches encounter persistent chal- lenges, often resulting from shortcomings in the planning processes of LLMs or the capa- bilities of visual modules. More precisely, they often fall short of the performance achieved by non-compositional models; refer to Fig. for exemplar error cases. To investigate these limitations, manual examination of 100 randomly selected failure cases (Section 4.5.1) of VISPROG (Gupta & Kembhavi, 2023) on the GQA dataset (Hudson & Manning, 2019) for visual question answering was conducted. The analysis revealed two primary failure categories: Firstly, approximately 30% of these failures were attributed planning errors, where the LLM failed to parse the language query into programs correctly, preventing the formulation of a solvable program. Secondly, over 40% of the failures were attributed to module errors, as the visual modules were incapable of executing the program accurately. The remaining failure cases (less than 30%) stemmed from issues like synonym usage (e.g., "woman" us. "lady") question ambiguity.
Motivated by these failure modes, in this work, we introduce EXOVIP, a method that uses "exoskeleton" verification modules to verify the reasoning results step by step, thus correcting the module errors and refining the LLM planning traces. As depicted in Fig. 1, EXoVIP effectively rectifies both types of errors: the verification module con- tains a mixture of three sub-verifiers, including an image-text matching verifier, an image captioning verifier, and visual question answering (VQA) verifier. These sub-verifiers meticulously validate the accuracy of the predictions generated by the visual modules, thereby correcting module errors. For refining the planning traces, a reasoning trace tree is constructed based the verification scores, along with the self-correctness score (Pan et al., 2023) obtained from LLMss. The methodology involves searching through the tree to identify the optimal trace with the highest score.
and-p
To demonstrate the effectiveness of ExOVIP, we apply our method to two recent visual programming methods: self-defined programs, i.e., VISPROG (Gupta & Kembhavi, 2023) and Python code programs, i.e., ViperGPT (Didac et al., 2023). Our experiments encom- pass six compositional visual reasoning tasks, including compositional image question an- swering, referring expression understanding, natural language for visual reasoning, visual abstract reasoning, language-guided image editing, and spatial-temporal reasoning. The
2

---

Published as a conference paper at COLM 2024
experimental results consistently indicate notable improvements in performance for both models. In light of this, we believe EXOVIP can foster better performance on open-world compositional reasoning tasks.
In summary, our main contributions are as follows:
We introduce "exoskeleton" verification modules tailored for module error and plan error in existing compositional visual reasoning methods, which systematically validate the accuracy of vision module predictions in step-by-step manner.
We illustrate the synergistic integration of our proposed verification modules with a tree- based search algorithm, enhanced by the self-correcting capabilities of the LLM. This collaborative design effectively tackles both the module error and plan error. The tree- based search is informed by a verification score, which serves as measure of confidence in the search process. Concurrently, this verification score is dynamically refined as the search progresses, ensuring a more accurate and reliable verification process.
We have implemented our methodology within two compositional methods, and the outcome has been a uniform enhancement in performance across six diverse tasks en- compassing both image and video modalities. This underscores the efficacy of EXOVIP in augmenting visual reasoning skills.
2 Related work
LLMs in multi-modal tasks. LLMs have significantly enhanced multi-modal tasks through their adaptability and extensive knowledge. There are three primary methods for applying LLMs to multi-modal challenges. One approach involves integrating ex- tra parameters into LLMs for multi-modal contexts and then fine-tuning with either a fixed (Isimpoukelli et al., 2021; Alayrac et al., 2022; Li et al., 2023b; Gao et al., 2023; Li et al., 2023a; Dai et al., 2023; Zhang et al., 2023d) or an adjustable LLM (Hao et al., 2022; Huang et al., 2023; Peng et al., 2023). Another strategy uses LLMs as knowledge experts, combining them with specialists in other fields like vision and speech to tackle diverse tasks (Zeng et al., 2023; Zhang et al., 2023c; Liu et al., 2023b). Our research concentrates on third method that leverages the LLM's ability to parse complex queries and delegate tasks to expert agents, whether through custom programs (Gupta & Kembhavi, 2023), Python code (Dídac et al., 2023), or dialogue agents (Yang et al., 2023). However, the effectiveness of these approaches limited by the quality of the planning sequences and visual experts.
Compositional multi-modal methods. At an early stage, neural module networks (NMN) (Andreas et al., 2015; Hu al., 2017; Johnson et al. 2017; Hu et al., 2018; Le et al., 2022; Qian et al., 2022; Wang et al., 2024) create end-to-end differentiable networks with neural models, but their pre-set modules struggle with open-domain tasks, and the com- plex embedding and attention mechanisms hinder the creation of intermediate supervision signals. Recently, the presence of LLMs has made it possible to automatically compose var- ious kinds of finetuned neural models (Zeng et al., 2023; Gupta & Kembhavi, 2023; Dídac et al., 2023; Yang et al., 2023; Liu et al., 2023b) or external tools (Parisi et al., 2022; Khot et al., 2023; Schick et al., 2023; Shen et al., 2023; Lu et al., 2023; Qin et al., 2023). These works allow us to diagnose the intermedia rationales of the reasoning process. However, human anno- tation of these intermedia results can be rather time-consuming. In this work, we make ways to correct errors in the intermedia results without any human intervention.
Self-correction in LLMs. Although LLMs achieve great success in various tasks, there are many errors in LLM-based system (Pan et al., 2023): hallucination (Li et al., 2023c; Zhang et al., 2023b), unfaithful reasoning (Golovneva et al., 2022; Ribeiro et al., 2023; LYU et al., 2023), toxic, biased and harmful contents (Shaikh et al., 2022), flawed code. One way to fix these errors is to use LLMs themselves (Madaan et al., 2023; Shinn et al., 2023; Ye et al., 2023; Yan et al., 2023) to obtain feedback to correct the errors. Incorporating self-correction strategies from LLMs, researchers aim to streamline reasoning in multi-modal systems. IPVR (Chen et al., 2023) employs LLMs for rationale generation and cross-modality valida- tion to ensure consistent inference. IdeaGPT (You et al., 2023) uses an LLM to summarize and iteratively refine the output of visual experts. To overcome the inherent limitations of
3

---

Published as a conference paper at COLM 2024
LLM self-correction, approach merges LLM feedback with insights from visual experts to authenticate intermediate results and the reasoning process.
EXOVIP
To address the aforementioned shortcomings, we propose EXOVIP. This framework adopts exoskeleton verification modules to calibrate the prediction of the execution modules and refine the reasoning path with tree searching. In this section, we will first introduce the preliminaries, including task definition and visual programming. Then will show the verification modules, and describe how the verification results are applied to correct the results of execution modules and to search for the reasoning trace. Additionally, we will introduce two mechanisms negative sampling and post-hoc self-correction to alleviate extra errors introduced by verification modules.
3.1 Preliminaries
Task definition. Our work focuses on Visual Compositional Reasoning (VCR) tasks. These VCR tasks require reasoning on a series of steps about image input and text input T, and predict the output, e.g. answer to given question, edited images given language instruction, etc.
Visual programming (VisPROG). VISPROG (Gupta & Kembhavi, 2023) is a zero-shot VCR model that leverages LLMs and pretrained vision models. It transforms complex text into a program operations (P = (o',....,")) using LLMs, which are then executed by various vision models (egg,, object detectors, VQA models). Each operation yields an output a; where a; serves the input for the next operation. The final prediction is made after all operations are executed. However, this approach highlights two key shortcomings of existing approaches: module error, the operation models can not predict the answer correctly; ii) planning error, the LLM might generate unfaithful reasoning.
Framework overview Fig. 1 depicts the overall framework. For each operation of, we get set of candidate answers {at....,ay ..), with probabilities {pi...., pp. Unlike VIsPROG, which directly takes the top answer, we use additional verification modules to verify each candidate answer, thus producing verification scores (sq...., ,. Then we take the verifi- cation score calibrate the original scores. Additionally, use the verification scores to search for a program with high verification scores, in order to refine the execution program by tree-searching.
3.2 Verification modules
The verification modules aims to verify the candidate answers {a!....,aa ) given an oper- ation o'. For example, the LOC (nightstand) operation returns set of candidate bounding boxes containing a nightstand, then the verification module verifies whether each of the returned boxes contains nighistand and produces verification scores. Our verification module is mixture of three off-the-shelf sub-verifiers, The output scores of the three ver- ifiers are combined as the final verification score. It important to emphasize that the verification model does not incorporate any extra pre-trained models; instead, it utilizes the verifiers that are integral the execution modules of VIsPROG. To ensure equitable comparisons between modules, we have deliberately chosen these specific sub-verifiers.
Image-text matching verifier calculates the similarity between the whole images and all candidate sentences, which returns the semantic representation of the image-sentence pair. We construct the candidate sentences Tans by filling the template "a photo of" with candi- date answers. In this work, we select CLIP (Radford et al., 2021) to calculate the similarity between images and sentences, i.e., sitn ITM(Tans. img).
Image captioning verifier leverages natural language to describe the visual details of the image. We first get the caption of the image Cing by BLIIP (L. et al., 2022b). We then construct
4

---

Published as a conference paper at COLM 2024
the descriptions of candidate answers Cans with the template "the image describe". Specif- ically, for candidate question-answer pairs, we initially transform the pair into a sentence before inserting it into the template. After that, we calculate the sentence semantic similar- ity (Reimers & Gurevych, 2019) between the captions and the constructed descriptions as the verification score, i.e., cap sim(Cans, Cimg).
Visual question-answering (VQA) verifier is more flexible than others, which offers us more opportunities evaluate the advanced relationships between image and language, such as entailment and factual consistency. Slightly different from the other two types of models, for the VQA verifier, we design templates w.r.t. the neural modules. For example, we use "Is there any object in the image?" for the object detection model, and use "Does this part look like object ?" for the classification model used in the abstract reason- ing task. We determine the verification score by BLIP (Li et al., 2022b) by calculating the difference in answer probabilities Qans between "yes" and "no"
Zqt aas
VQA Qans, True) VQA(Qans False)
(1)
Verification score After obtaining the scores from each individual verification mod- ule, the verification score is averaged over all scores for each given answer, i.e,, Sans =
San avg(sitng ( Sans Sanse s0g))
Negative sampling. Empirically, we find that directly applying this verification score does not work well, because the score scales for different kinds of candidates are not well- calibrated. Motivated by recent works in truthfulness (Li et al., 2022a), commonsense (Ye et al., 2022), and bias (Ruggeri & Nozza, 2023), we propose to take the difference of can- didate answer dj with its semantic opposites jj as the final verification score. More specif- ically, the semantic opposite n; is selected based on the text embeddings from CLIP Rad- ford et al. (2021), i.e. the word of lowest embedding similarity is selected. For example, the semantic opposite of "nightstand" is "stocking". We then compute the difference of the verification scores of the candidate answer and its semantic opposites, and get the final verification score. Mathematically, given candidate answer al, the final verification score is Sy = Saj Sm.
Calibration using verification scores After obtaining the verification scores of all candi- date answers S {$ (si, we normalize them as weights and calibrate the candidate predictions, P/ wj * Pj. where wj is the normalized verification score. More specifically,
the verification score sj re-scaled to w Smax Symn
S,-Swic -T- 4 1, where is a hyper-
parameter controlling the scaling factor Smin, Smax are the minimum or maximum of all the candidate scores.
3.3 Exploration with reasoning trace
(P-1)
saff correct
To mitigate the planning errors, we further apply the verification scores to refine the reasoning trace predicted by LLMs. Mo-
0.3
LLA
0.3
0.5
tivated by the recent works showing that searching through compositional prob- lem space can greatly improve the per- formance of LLMs for complex tasks (Yao et al., 2023; Khalifa et al., 2023; Hao et al., 2023), we introduce uce our dynamic reason- ing trace searching procedure, which takes advantage both the LLM self-correctness potential and our verification modules. In Appendix A, we show the complete algo- rithm of EXOVIP.
0.2
0.55
Verification Modules
verify(t-2)
verify (6-2)
Figure 2: Search of the reasoning trace. We beam search through the program tree, based on the verification scores as well as the LLM self-correctness.
Tree-based reasoning trace searching (TRS) represented as a tree structure, where each node is reasoning operation. To get better
The reasoning trace searching procedure is
5

---

Published as a conference paper at COLM 2024
reasoning trace, search from the tree using the beam search algorithm (Graves, 2012; Boulanger-Lewandoowki et al., 2013; Sutskever et al., 2014), which has long been proven to be effective in sequence-to-sequueee problems.
More specifically, our trace searching procedure contains two steps. First, in order to gen- erate more diverse reasoning traces search from, we randomly perturb the in-context ex- amples in the prompt for LLM. Second, after we get the result of candidate neural modules, sort them according to the verification scores and select the top K candidate reasoning traces.
Post-hoc self-correction (PSC) In some cases, the verification scores can be very close for the top-rated K traces, which could result in suboptimal. Inspired by the zero-shot ranking ability of LLM Hou et al. (2023), we further use the self-correctness ability of LLMs rank the K traces and select the top P from them (P < K). More details of the prompts used for LLM self-correction are included in Sec. E.2.
Experiments
In this section, we apply the EXOVIP to VISPROG, demonstrating the effectiveness of our approach through results and anlysis derived from six distinct tasks, including visual ques- tion answering, referring expression understanding, visual reasoning using natural lan- guage, abstract reasoning, language-guidee image editing, video question answering. Sub- sequent to this, we delve into an exploration of potential future projections in Sec. 4.5. For additional information regarding the implementation and experiment setups and selection of baselines, please refer to Appendix E.
4.1 Compositional Visual Question Answering
4.1.1 Main Results and Analysis
We evaluate the efficacy of compositional visual question answering task GQA (Hudson Manning, 2019), and benchmark against top vision-language models, including BLIP2-flant5-xxl (Li et al., 2023b), InstructBLIP-flann t5-xl (Dai et al., 2023), and use LLaVA-1.5-13B (Liu al., 2023a) as reference point. Our method boosts VISPROG's score from 57.41 to 61.49, surpassing BLIP2 and InstructBLIP, as detailed in Tab. It's important to note that our method does not incorpo- rate any additional modules or knowledge compared to VIs- PROG. The verification modules that we use are inherent to VISPROG itself. To verify the effectiveness of each component in our method, we run a series of ablation studies on our framework (also in Tab. 1). We have the following observations:
ExoVIP
Methods
Accuracy
BLIP2-xx (Li et al., 2023b)
49.20
InstructBLIP-Raant-- (Dai et al., 2023)
55.39
Llava-1.5-13b- (Liu al., 2023a)
74.56
0 VISPROG (Gupta & Kembhavi, 2023)
57.41
EXOVIP w/o self-correctness & negative
57.11
sampling & search
ExOV1P w self-correctness & search
58.53
EXOVIP w/o self-correctness (TRS)
60.57
4
ExoViP w/o verification (PSC)
60.16
5
EXOVIP
61.49
Table Results of compositional visual question answering on GQA. Llava-1.5-13b* is tuned on GQA training corpora, and evaluated with additional prompt.
Negative sampling enhances the robustness of EXOVIP Merely adding verification modules to a system (Line-1) is not sufficient for achieving better results; it may actu- ally lead to decreased performance. On the other hand, when we implement a negative sampling technique that utilizes semantic opposites in these verification modules (Line-2), there is a marked enhancement in the system's 's performance. We posit that this approach could be instrumental in reducing the likelihood of new errors being introduced.

---

Published as a conference paper at COLM 2024
TRS effectively utilizes the verification score Empirical evidence shows that imple- menting TRS can increase accuracy from an initial 58.53 a subsequent 60.57 (Line-3). This improvement in precision underscores the effectiveness of our verification-based search strategy, which has the potential to resolve numerous planning errors.
Verification mechanism enhance LLM self-correction In Line-4, exclusively utilize the post-hoc self-correction (PSC) of the LLM during the process of trace searching, eschew- ing the use of verification scores. The findings demonstrate enhancement in accuracy of 2.75 when compared to the original VISPROG. However, the implementation of both ver- ification scores and LLM self-correctness concurrently results in a superior performance enhancement.
4.1.2 Analysis Experiments
We experiment to explore how the tree-based search algorithm and verification scores inter- act to improve our approach. The algorithm scores to guide branching, while insights from the search enhance score contrasts, refining path differentiation and aiding in finding the best solution. All experiment settings are aligned with the Appendix B.
Mixture of Sub-verifers. We evaluate the effects of different types of verification modules with the setting of the best demonstration setting. As is illus- trated in Tab. 2, Different verification modules share similar boost gain, but a mixture of these modules can benefit more.
Methods
Accuracy
Base
58.14
Image-text Matching
59.26
Image Caption
59.22
Enhanced Verification through TRS. Our analy- sis, illustrated Fig. 6, shows that implementing our trace-searching strategy significantly improved verification scores. Additionally, the increased vari- ance in scores suggests our method could further re- fine the effectiveness of reasoning traces.
Visual QA
59.35
All
60.03
Table 2: Analysis on the sub- verifiers.
Enhanced Planning Efficacy In our comparative
analysis of the GQA task, we observed that our method, EXOVIP, significantly outper- forms the baseline, VISPROG, in terms of planning efficiency and accuracy. The average number of planning steps required by EXOVIP decreased from 5.92 to 4.77, indicating that the TRS strategy employed by our method streamlines the planning process, allowing for a more direct path to the final plan. We also compute the average inference time, which shown on Appendix D. Furthermore, we noted reduction in the error rate, with the percentage of unexecutable plans dropping from 5.84% to 3.82%. This demonstrates that EXoVIP not only reduces the complexity of the planning process but also enhances the reli- ability of the generated plans, predicting a higher number of executable routines compared to the baseline.
4.2 Abstract Visual Reasoning
We tested model performance the KILOGRAM (li et al., 2022) dataset's text-to-image retrieval task, involving 1,251 tangram puzzles with abstract shape recognition. Accuracy was the main evaluation metric. The CLip-large model (Radford et al., 2021) served as our baseline for the text-to-image retrieval task. As is illustrated in Fig. 18, in our approach, we leverage the LLM to identify potential semantic components of given description. Simultaneously, we segment the image into distinct visual parts. Following this, we align the identified semantic parts with their corresponding visual counterparts to optimize the matching process. Tab. 4 illustrates how our method effectively utilizes conceptual com- ponents to enhance abstract image understanding. However, performance gap is still noticeable when compared to CLIP. Despite our method narrowing this gap, it is still un- able to reach SOTA performance levels. The importance of part identification in human abstraction has been well-established in prior research (Tversky & Hemenway, 1984). We posit that the efficacy our approach could significantly improved by integrating more advanced scene segmentation model.
7

---

Published as a conference paper at COLM 2024
Table 3: Visual referring expression on Ref-
Table 4: Abstract reasoning on KILO-
COCO, RefCOCO+, and RefCOCOg.
GRAM.
Methods
IoU
Methods
Accuracy
Qwen-vl-chat-7b (Bai et al., 2023)
32.54
CLip-large (Radford et al., 2021)
27 27.26
VISPROG (Gupta & Kembhavi, 2023)
27.28
VISPROG (Cupta Kembhavi, 2023)
24.46
EXOVIP
31.50
ExoViP
26.22
Table 5: Visual reasoning on NLVR2.
Table 6: Image editing on MagicBrush.
Methods
Accuracy
Methods
CLIP-1
DINO
OFA-large (Wang et al,, 2022)
58.38
InstructPi<PPix (Brooks et al., 2022)
84.19
69.60
VISPROG (Gupta & Kembhavi, 2023)
VISPROG (Gupta Kembhavi, 2023)
90.82
EXOVIP
67.96
ExoVIP
91.27
83.40
Source
Target
InstructPix2Pix
VisProg
ExoViP (Ours)
Query: have team of sled dogs pulling the snowboarder
Query: open the lid of toilet
Figure 3: Qualitative results of text-guided image editing MagicBrush
4.3 Language-grounded Visual Tasks
Visual Referring Expressions (VRE) Our study on VRE used subset of the RefCOCO, RefCOCO+, and RefCOCOg datasets (Yu et al., 2016; Kazemzadeh et al., 2014), and evalu- ated using intersection-over-reee
(IoU). We benchmarked against the Qwen-vi-chat-76 (Bai et al., 2023) model, tigh--prforming, pre-trained vision-language model. The results pre- sented in Tab. 3 illustrate that, even though our approach does not reach the state-of-the (SOTA) performance achieved by Qwen-vi on the RefCOCO dataset, it nonetheless narrows the gap between VIsPROG and large vision-language models. Qwen-vl is highly complex model, constructed on language learning model (LLM) consisting of 7 billion parameters and trained on corpus of trillions of tokens. In contrast, our approach utilizes a team of specialized experts, whose combined parameters amount less than 1 billion. We are opti- mistic that the performance of our method can be further enhanced by incorporating more sophisticated experts.
Natural Language Visual Reasoning In this work, we use the NLVR2 (Suhr et al., 2019) balanced test set for evaluation and use the OFA-large (Wang al., 2022) as our baseline, unlike many multi-modal language models that struggle with dual-image inputs. Tab. presents our findings. While VISPROC demonstrates a strong capability for complex rea- soning compared to the end-to-end model, our method struggles to enhance its perfor- mance significantly. We attribute this to our sole reliance on VQA modules for solving NLVR problems. Specifically, the efficacy of the decomposed VQA steps is intrinsically con- strained by the performance of the VQA model itself. This limitation becomes especially troublesome when errors accumulate over series of VQA steps, consequently hamper-
8

---

Published as a conference paper at COLM 2024
ing the overall performance. As path forward, we foresee potential advancements in the planning process, which could involve the integration of a wider array of expert inputs.
Text-guided Image Editing We use the MagicBrush dataset (Zhang et al., 2023a)for eval- uation. Image quality is gauged using CLIP-I and DINO embeddings for similarity assess- ment. Our baseline is the GPT3-augmented InstructPix2Pix (Brooks et al., 2022) model. The results from both CLIP-I and DINO are presented in Tab. 6. These results illustrate the ca- pability of our method to enhance the similarity between the edited image and the target image, signifying the precision of our image editing technique. For a more comprehensive evaluation of the editing quality, we have conducted case study. Fig. 3 exhibits some instances using MagicBrush. It is observed that non-compositional methods, i.e. Instruct- Pix2Pix, tend to alter unrelated pixels, whereas compositional methods, i.e. VISPROG and our model, offer more control. Furthermore, when compared to VISPROG, our method ex- cels in two key areas: accurately pinpointing the region that requires editing, and adjusting the image to the appropriate extent. This demonstrates the superiority of our method in both localization and modification of the image.
modificaioonn
4.4 Spatial-Temporal Video Reasoning
We conducted experiments using the subset of AGQA 2.0 dataset (Grunde- McLaughlin et al., 2022). Our reference was the Video-LLaVA (Lin et al., 2023), top- tier vision-language model known for its superior performance numerous bench- marks. In our methodology, we address the question by breaking it down into tempo- ral and spatial components. For the tempo- ral aspect, we aim to find the event ac- tion within a video. This is achieved by uniformly sampling frames from the video and generating corresponding captions. We then compute the sentence similarity between these captions and the input query. Subsequently, we identify the event by locating the video segment with the highest similarity, utilizing a monotónic stack algorithm. By adopting this approach, we can effectively mitigate the Out-of- Vicibulayy (OOV) issue that plagues current action classification models. Regarding the spatial component, it is predominantly addressed by existing VQA models. The experimental outcomes, presented in Tab. 7, indicate that the compositional method yields strong performance. However, the benefits brought by verification are limited. Upon further examination of the results and the un- derlying reasoning paths, we observe that the majority of the unsuccessful cases can be attributed to the performance of the VQA models, trend that aligns with findings from the NLVR task.
Methods
Accuracy
Video-LLaVA (Lin et al., 2023)
30.38
EXOVIP w/o verification EXOVIP
37.88
38.00
Table 7: Results of Spatial-Temporal Reason- ing AGQA.
4.5 Discussion
4.5.1 Error analysis of VISPROG and EXOVIP
We manually analyze 100 randomly sampled failure cases on VISPROC. We find that there are three typical reasons for the failures: (a) vision module prediction error; (b) LLM plan- ning error; (c) others. We demonstrate the statistics of the failure cases in Fig. 4 (left). Fol- lowing the application of our proposed framework, we reassessed the same cases in ?? (right) and were pleased to discover reduction in module errors by 28.87%, and a de- crease planning errors by 42.35%. Nevertheless, our current strategy was unable to rec- tify 69.8% of the errors. When juxtaposed with the data from Tab. 1, our method has en- hanced VISPROG by 7.11%, which is lower than the improvement of the failure cases, This outcome suggests that our approach may give rise to novel challenges. We further demon- strate common errors of our method in Fig. 9 and Fig. 10. We find the majority of these failure cases originate from the VQA module.
Additional error analysis Our methodology acknowledges the possibility that the inclu- sion of verifiers might inadvertently increase the error rate. To counteract this, we have

---

Published as a conference paper at COLM 2024
17.7%
31.3%
20.8%
44.0%
Plan Error
Module Error
Other Emor
Correct Prediction
Figure Distribution of the failure cases of original VISPROG (left), and distribution of the failure cases of EXOVIP (right)
adopted a negative sampling approach, and we have integrated with a verification score and the inherent self-corrective feature of LLMss. The efficacy of these combined strate- gies in reducing the incidence of additional errors is evidenced by the results displayed in Tab. 2. Nonetheless, while our approach successfully diminishes errors related to planning and module execution, it can occasionally lead to the introduction of new errors. Moving forward, we aim to enhance our system by incorporating a greater number of verifiers to more effectively resolve these issues.
4.5.2 Method generalizability
To validate the generalizability of our method, we applied it to ViperGPT, which composes available modules by generating Python codes. We equip ViperGPT with our method and test its performance on the GQA dataset. The results, presented in Tab. 8, reveal less significant perfor- mance boost compared to VISPROG. We at- tribute this to ViperGPT providing only a few demonstration examples and adjusting the parameters of the code-generation model to deterministically generate subroutines. We believe this could be improved by introduc- ing diverse demonstrations, similar to VISPROG.
Methods
Accuracy
ViperGPT (Didac et al., 2023)
45,47
ViperGPT+ExoVip
46.84
Table 8: Results for ViperGIT on GQA.
Conclusion
In this work, identify two key types of errors in existing compositional methods: plan- ning errors and module errors. To address these errors, we introduce an innovative verifica- tion framework EXOVIP. This framework verifies the correctness of vision module predic- tions. It corrects module errors by calibration and refines the planning process through tree searching. During this process, it considers both verification scores and the self-correctness of LLM. Applying the ExoVIP to two existing models, we achieve performance improve- ments across five different tasks. The results reinforce the promise and potential EXOVIP various open-world compositional reasoning tasks, marking an important milestone in the realm of multi-modal tasks involving complex reasoning.
The authors thank the reviewers for their insightful suggestions to improve the manuscript. This work presented herein is supported by the National Natural Science Foundation of China (62376031).
Acknowledgments
10

---

Published as a conference paper at COLM 2024
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. Conference on Computer Vision and Pattern Recognition (CVPR), pp. 39-48, 2015. 1,
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for under- standing, localization, text reading, and beyond. 2023. 8
Nicolas Boulanger-Lewandower
Yoshua Bengio, and Pascal Vincent. Audio chord recog- nition with recurrent neural networks. In International Society for Music Information Re- frieval Conference, 2013. 6
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions, Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392-18402, 2022. 8,9
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, 0t5//2005.111666 2020. 1
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-basee visual reasoning. ArXiv, abs/ 2301 .05226, 2023.
Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems (NeurIPS), pp. 17864-17875, 2021. 21
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Joshua
Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben- ton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur- Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiri- donov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, htss/2204.02111 2022. 1
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. ArXiv,
2023.
3,6 6
11

---

Published as a conference paper at COLM 2024
Surís Didac, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python cution for reasoning. arviv preprint ar arXiv:2303.0812, 2023. 1, 3,
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang- Alphazero-like tree-search can guide large language model decoding and training. CoRR, ftbbs//239.11799 2023. 20
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter V2: parameter-efficient visual instruction model. CoRR, abs/2304.15000 2023. 3
O. Yu. Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-ZZrandi, and Asli Celikyilmaz. Roscoe: A suite of metrics for scoring step-by-step reasoning. ArXiv, abs/2212.07919, 2022. 3
Alex Graves. Sequence transduction with recurrent neural networks.
ArXiv,
abs/1211.3711,2012 6
Madeleine Grunde-MeLaughlin, Ranjay Krishna, and Maneesh Agrawala. 2.0: An updated benchmark for compositional spatio-temporal reasoning. abs/2204.06105, 2022.
AGQA CoRR,
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual rea- soning without training. In Conference on Computer Vision and Puttern Recognition (CVPR), Pp. 14953-14962, June 2023. 1, 2, 3, 4, 6,8
Shibo Hao, Yilan Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. ArXiv, abs/2305.14992, 2023.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are purppsse interfaces. ArXiv, abs 2abp//2200066366 2022. 3
Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender sys- tems. In European Conference on Information Retrieval, 2023.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learn- ing to reason: End-to-end module networks for visual question answering. In Interna- fional Conference Computer Vision (ICCV), pp. 804-813. IEEE Computer Society, 2017. 1,
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural com- putation via stack neural module networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), European Conference on Computer Vision (FCCV), vol- ume 11211 of Lecture Notes in Computer Science, pp. 55-71. Sprinneer, 2018. 1,3
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggar- wal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. CoRR, abs//302.140455 2023.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world vi- sual reasoning and compositional question answering. Conference on Computer Vision and Paliern Recognition (CVPR), pp. 6693-6702, 2019. 2, 6
Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D. Hawkins, and Yoav Artzi Abstract visual reasoning with tangram shapes. In Amnual Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. 7
12

---

Published as a conference paper at COLM 2024
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. Inferring and executing programs for visual reasoning. International Conference Computer Vision (ICCV), pp. 3008-3017, 2017. 1,3
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Re- ferring to objects in photographs of natural scenes. In Annual Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 787-798, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.8115/v1/114-11
Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Ho Hin Lee, and Lu Wang. Discriminator-guidee multi-step reasoning with language models. ArXiv, 0bbs/2305.19933 2023.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving com- plex tasks. In International Conference on Learning Representations (ICLR). OpenReview.net, 2023. 3
Hung Le, Nancy F. Chen, and Steven C. H. Hoi. VGNMN: video-grounded neural module networks for inde--grounded dialogue systems. In Marine Carpuat, Matie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 3377-3393. As- sociation for Computational Linguistics, 2022. 1,3
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jinggangg Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. CoRR, abs/ 2a/5/233500372 2023a. 3
Junbo Li, Xianhang Li, and Cihang Xie. Mitigating lies in vision-language models. In NeurIPS ML Safety Workshop, 2022a. 5
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In Inter- national Conference on Machine Learning (ICML), 2022b. 4, 5,
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint ar XrXi:22301.15500 2023b. 3,6 6
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. CoRR, abs/2305.11747, 2023c. 3
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video- llava: Learning united visual representation by alignment before projection. CoRR, abs/2311.10122, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/ 2310.03444 2023a. 6
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandku- mar. Prismer: A vasion-language model with an ensemble of experts. aix preprint ar XrXi::2033.02566 2023b. 3
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. CoRR, abs/2204.09842, 2023. 3
QING LYU, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. Faithful chain-of-thought reasoning. ArXio, 0tt//2301.33779 2023. 3
13

---

Published as a conference paper at COLM 2024
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bod- hisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback. ArXiv, abs/ 225/2303.11 23611113311111,, 2023. 3
Matthias Minderer, Alexey A. Gritsenko, Austin Stone, Maxim Neumann, Dirk Weis- senborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open- vocabulary object detection with vision transformers. CoRR, abs (bbs/220000630 2022.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2027, 1
Liangming Pan, Michael Stephen Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. ArXiv, abs/ 2200//238,,188, 2023. 2,3
Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. ArXiv, abs/2205.12255 2022.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CoRR, Dos/2306.14824, 2023. 3
Zi Qian, Xin Wang, Xuguang Duan, Hong Chen, and Wenwu Zhu. Dynamic spatio- temporal modular network for video question answering. In João Magalhães, Al- berto Del Bimbo, Shin'ich Satoh, Nicu Sebe, Xavier Alameda-Pineda, Qin Jin, Vincent Oria, and Laura Toni (eds.), MM '22: The 30th ACM International Conference on Multime- dia, Lisboa, Portugal, October 10 14, 2022, Pp. 4466 4477. ACM, 2022. 1,3
Yujia Qin, Shihao Liang Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. CoRR, abs/2307.16789 2023.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018. 1
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Lan- guage models are unsupervised multitask learners. 2019.
Alec Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. 4, 5, 8, 21
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Annual Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 11 2019. 5
Danilo Neves Ribeiro, Shen Wang, Xiaofei Ma, He Zhu, Rui Dong, Deguang Kong, Juli- ette Burger, Anjelica Ramos, William Yang Wang, Zhiheng Huang, George Karypis, Bing Xiang, and Dan Roth. Street: A multi-task structured reasoning and explanation bench- mark. ArXiv, abs/ 2302.06729 2023. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference Compuler Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022. 21
Gabriele Ruggeri and Debora Nozza. A multi-dinennsioal study bias in vision- language models. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023. 5
14

---

Published as a conference paper at COLM 2024
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zet,lemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CoRR, abs /bs//230.04461, 2020, 3
Omar Shaikh, Hongxin Zhang, William B. Held, Michael Bernstein, and Diyi Yang. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning. ArXiv, abs/ 0ts//222.080011 2022. 3
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang- Hugginggpt: Solving Al tasks with chatgpt and its friends in huggingface. CoRR, abs/2303.17580, 2023.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. 2023.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6418-6428, Florence, Italy, July 2019. Association for Computational Linguistics. doi: D0.1863/vv/PP9900
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NeurIPS), Montreal, CA, 2014. 6
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill, and Zacharias Janssen. Multimodal few-shot learning with frozen language models. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3
Barbara Tversky and Kathleen Hemenway. Objects, parts, and categories, Journal of experi- mental psychology. General, 113 2:169-99 97, 1984. 7
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sqquence-t--equence learning framework. In International Conference on Machine Learning (ICML), 2022.
Yueqian Wang, Yuxuan Wang, Kai Chen, and Dongyan Zhao. STAIR: spatial-temporal reasoning with auditable intermediate results for video question answering. Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), AAAI Conference on Artificial Intelligence (AAAI), pp. 19215-19223. AAAI Press, 2024. 3
Hao Yan, Saurabh Srivastava, Yintao Tai, Sida I. Wang, Wen tau Yih, and Ziyu Yao. Learn- ing to simulate natural language feedback for interactive semantic parsing. ArXiv, abs/2305.08195, 2023.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. 2023. 3
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. CoRR, (bbs/2305.1060 2023. 5, 20
Seonghyeon Ye, Yongrae Jo, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, and Min- joon Seo. Selfee: Iterative self-revising Ilm empowered by self-feedback generation. Blog post, May 2023.
Shuquan Ye, Yujia Xie, Dongdong Chen, Yichong Xu, Lu Yuan, Chenguang Zhu, and Jing Liao. Improving commonsense in vision-language models via knowledge graph riddles. Conference Computer Vision and Pattern Recognition (CVPR), pp. 2634 2445, 2022. 5
15

---

Published as a conference paper at COLM 2024
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A. Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. Idealgpt: Iteratively decomposing vision and lan- guage reasoning via large language models, 2023. 3
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. Lecture Notes in Compuler Science, pp. 69-85, 2016. ISSN 1611-3349. doi: 10.1007 10.1007/978-3-31111 /97--999999-44475--666
Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero- shot multimodal reasoning with language. In International Conference Learning Repre- sentations (ICLR), 2023.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. ArXiv, abs//306.10012, 2023a.
Muru Zhang, Ofir Press, Will Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. ArXio, abs/2305.13534, 2023b. 3
Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. Conference Computer Vision and Pattern Recognition (CVPR), 3
Jun 2023c. doi: 10.1109 cvp:52729.2023.012
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. CoRR, abs/230s.... (bss/230.1710,, 2023d. 3
Andy Zhou, Kai Yan, Michal Dalapentokh-Rothman Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. CoRR, /bb///231.4406 2023. 20
16

---

Published as a conference paper at COLM 2024
Appendices
Table of Contents
A Exoskeleton Algorithm
17
Proof-of-concept Pilot Experiments
17
C Enhanced Verification through TRS
19
D Efficiency Analysis
20
Implementation details
E.1 Visual modules.
21
E.2 LLM Prompts
21
E.3 Details of Visual Abstract Reasoning
23
E,4
Implementation details
24
Qualitative study.
24
F.1 Qualitative examples
24
A Exoskeleton Algorithm
We demonstrate the overall algorithm of our method in Algorithm 1. There are mainly two parts: step-by-step verification and exploration with reasoning trace. To be more spe- cific, we fuse the self-correctness ability of LLM into the procedure of tree-based reasoning trace searching, which has shown potential in calibrating the effectiveness of the searching algorithm.
B Proof-of-concept Pilot Experiments
To evaluate the effectiveness of the verification modules, we try to find the relationship between verification scores and accuracy. All experiments are applied the GQA dataset. We first disturb the examples in the demonstrations to get different plan results and cor- responding verification scores. Specifically, we change the order of examples and select different portions of examples with four settings. After evaluation, we calculate the mean of verification scores of steps. As shown in Fig. 5, we are delighted to find the verifica- tion scores positively contribute to final accuracy. However, the trend is decreasing, which means when the verification scores increase to a certain extent, higher verification scores do little contribution to the final accuracy.
17

---

Published as a conference paper at COLM 2024
Algorithm 1: Exoskeleton Algorithm
Input: start step (co), goal node (g), scaling factor (T), verification size (K), rank size (P) Output: Verified reasoning trace and intermedia results
openList + eo
closed +- empty list
path < empty list
while open list not empty do
sort (ipen key = es)
Select top K steps from openList and put it in closedList and empty openList Lisk key = LLM
Select top P steps to update closedList
for e in Lissed do
if e is g then
path.add(e)
return path
else openList.add((enext)
end
end
for e in openList do
= avg(eflem
iitm eap
,ss enn es
uqa D4a
en
e + Verify(NORM(e,,)) T)e)
end end
6.6
Verification Score
6.4
6.2
6.0
5.8
5.6
50
52
54
56
58
60
Accuracy
Figure 5: Accuracy GQA positively correlates with the verification scores.
18

---

Published as a conference paper at COLM 2024
C Enhanced Verification through TRS
Veriteation
Soarth Vielltaaioo
Samples
-0.6
-0.3
0.0
0.2
0.6
0.3
Verification Score
Figure 6: Distribution of verification scores w. and w/o trace searching.
In 6, we present graphical representation illustrating the relationship between the trace- searching strategy and the verification score. The x-axis quantifies the verification score associated with each trace, while the y-axis denotes the number of traces corresponding to each score.
Q: What's the woman holding?
Q: keyboard to the left or to the right the book?
moxo=10Gimage::MG object="book")
BOX0-LOC(inage-IM Abjjec="wwmman no box returned, but there should be one fl after verification, this error is corrected IMAGEO-CROP|Bil IMAGE0-CROP/bbxx-80000
(MAGEE=CROPUbbo:-BOO X
BOX1-LOC[ IMAGEO, object= "keyborrr"""""
no keyboard IMAGEO after verification, the program corrected:
ANSWERO VOA(maae--IMAGG -MAGEO, "eestion="whtte is the woman holding?")
CROP - CROP RIGHT
ANSWERO-COUNT[bboue BOX1]
ANSWER1 EVAL[expressioo=="fff
ANSWERD > else 'right"")
(a) module error
(b) planning error
Figure 7: Existing methods suffer from two types of errors: (a) vision module prediction error and (b) LLM planning error. Our verification modules help correct the errors.
Q: Are the clouds the sky light and white?
Q: the gray door made out metal?
BOXO-LOC/|mag--MMG,
object-"sky""
BOXO-LDCImgg-IMG, object- "gray door")
IMAGED-CROPpbaor
IMAGED-CROP(bbox-BB BOXB) ANSWERD-VOA| VOAVERRRVVA//agg---- IMAGED,
VOAlimag--IMAEE
question= "Whtt color are the ciouds
question= material gray
XASWVHI " ANSWEND
made of? wood
"ight' ANSWERD
'no")
The door atttillyy made of metal
This expression -
only returs no
AGED
corrected ANSWER1 EVAL(cxprestoon If ANSWERO 'metal' else 'no""
verification,
program
ANSWERL-VOM (ASSWEEE-VONAmmag-liMeee waage///mmmmmaaaaaaa mage-IMAG50
question- "Are the dools light")
ANSWERR-EVALLexee
ANSWERO white" and ANSWER1 wn 'yes' else
!no")
(a) planning error
(b) module error
Figure 8: More examples of the two types of errors: (a) vision module prediction error and (b) LLM planning error.
In Figs. and show examples of failure cases of the original VIsPROG.
19

---

Published as a conference paper at COLM 2024
Q: the same color?
BOXG-LOC/magg-IMGGG Sojjjcct=shhe"
IMAGED-CROP@mg-IMMGGG bbox=BOX0)
BOXL=10C|maage IMAGE, object=
IMAGEL=CROP IMAGE, bbor
WMSWERO-VOMMimscelMee question The O0 premiie d5 MI tutrent mhdderhaaee a Tot errors color
shoe?"]
X
recognition
(wW.ERt-VOA/mageeeeeee question="What color is the cloud?"|
ANSWERI-EVAL (exprossion "yes"
ANSWER1 'nd"")
Figure 9: Common failure cases: some modules perform badly on certain tasks, egg. the VQA module performs poorly on color recognition tasks.
Which place
ENSWVER--VAAlmage IMGGE Which The reference answer forest
X
Figure 10: Common failure cases: some queries can not be decomposed into sub-tasks. Our method helps little with these non-decomposable queries.
D Efficiency Analysis
We present the average inference time on the GQA dataset. Generally, the temporal ex- penditure of our tree-based search method significantly surpasses that of VisProg. How- ever, the majority of the time is consumed by the call of the OPENAI API, an issue we posit intrinsic analogous works Yao al. (2023); Feng et al. (2023); Zhou When compared to Depth First Search/ Breadth First Search Yao et al. (2023) Monte Carlo Tree Search Feng et al. (2023); Zhou et al. (2023), we assert that our beam search- based method can achieve an optimal equilibrium between efficiency and effectiveness. In addition, when comparing with End-to-End model, we find that the most significant contributor to the overall time cost is the Planning Time. We have determined that this de- lay is largely attributable to Internet latency, as our system utilizes the GPT-o3--ttrro API. We are confident that this latency would reduce the dependency on network response times. Additionally, we are exploring ways to enhance the parallelism of our system's submodules, which we believe will further improve efficiency.
(2023).
be mitigated by deploying the LLM locally, which
Table Average Inference Time on the GQA Dataset
Methods
Total Infer. Time (s)
Planning Time (s)
Module Infer. Time (s)
BLIP2-Flant5-xxl
0.17
0.17
LLa LLaVA-1.5-7B
0.45
0.45
VISPRoG
1.59
1.10
0.49
EXOVIP
4.32
3.64
0.68
20

---

Published as a conference paper at COLM 2024
E
Implementation details
E.1
Visual modules.
Task
Operation Modules
Verification Modules
CLIP
Image
CROP
CROPLEFT
CROPRIGHT
CROPABOVE
CROPBELOW
VQA
PiL.crop0
PIL.crop
PILcrop0
PILcrop0
PILcrop0
BLIP
Visual Grounding
PILrectangle))
Natural Language
EVAL
CAP
Visual Roasoning
evall]
BLIP
BLIP
Abstract Reasoning
PART
SEG
ALIGN
SELECT
SIM
ChatGPT
Maskformer
CLIP
CLIP
CLIP
Text-guided
Editing
Stable
Figure 11: The neural modules (green) and symbolic modules (pink) used in our experi- ments.
We summarize the operation modules and the verification modules of different tasks in Fig. 11. In practice, the candidate neural modules include OWL-ViT (Minderer et al., 2022), CLIP (Radford et al., 2021), BLIP (Li et al., 2022b), ChatGPT, MaskFormer (Cheng et al., 2021), Stable Diffusion (Rombach et al., 2022). In order to validate the effectiveness of our method and eliminate the benefits of external knowledge such as more advanced vision- language models which are trained on larger datasets. Both operation modules and verifi- cation modules are selected from the same candidate neural module sets. In other words, not all modules verified on the mixture of all three types of modules.
E.2
LLM Prompts
We demonstrate the prompt for self-correctness of all five tasks.
You are a ranker for a planner who use the candidate modules to answer question: QUESTION, select the best solutions for answering the question candidate modules include: LOC: detection, VQA: visual question answering, EVAL: use logic operation, RESULT: wrap up the final result,
CROP CROP/CROP LEFTOF/CROP RIGHTOF/CROP FRONTOF/CROP CRONTOF/CROP_INFRONT/CCOOOO INFRONTOF/CROPPEEEINDO// AHEAD/CROP_BELLC BELOW/CROP_ABBO : crop the image.
Current solutions:
PLAN PLAN
If the modules in the solutions have better cause-and couse-and-effect relations, and likely the question, please rank it first. If you unsure, please keep the original rank. Return sequence number of currently best solution, for example 0,1,2,3, DO NOT RETURN ANYTHING ELSE EXCEPT FOR NUMBERS SPLIT by
Figure 12: Self-correctness prompt of compositional question answering.
21

---

Published as a conference paper at COLM 2024
You are a ranker for a planner who use the candidate modules to carry out the instruction: QUESTION, select the best solutions for carrying out the instruction candidate modules include: LOC: detection, FILTER: filter unrelated objects, TAG: tag the object, RESULT: wrap up the final result
Current solutions:
PLAN PLAN If the modules in the solutions have better cause-and-effect relations, and more likely to answer the question, please rank it first. If you are unsure, please keep the original rank. Return sequence number of currently best solution, for example 0,1,2,3, DO NOT RETURN ANYTHING ELSE EXCEPT FOR NUMBERS SPLIT by
Figure 13: Self-correctness prompt of visual grounding.
You are ranker a planner who use candidate modules to evaluate statement: QUESTION, select the best solutions for evaluating the statement
candidate modules include: VQA: visual question answering, EVAL: use logic operation, RESULT: wrap up the final result \n Current solutions
Current solutions:
PLAN PLAN
If the modules in the solutions have better cause-and-effect relations, and more likely to answer the question, please rank it first. If you are unsure, please keep the original rank. Return sequence number of currently best solution, for example 0,1,2,3, Do NOT RETURN ANYTHING ELSE EXCEPT FOR NUMBERS SPLIT by
Figure 14: Self-correctness prompt of natural language for visual reasoning.
You ranker for planner who the candidate modules carry the instruction: QUESTION, select the best solutions for carrying out the instruction candidate modules include: SEG: segmentation, SELECT: select related object, REPALCE: edit image, RESULT: wrap up the final result Current solutions:
PLAN
PLAN
the modules in the solutions have better cause-and-effect relations, and more likely to answer the question, please rank it first. If you are unsure, please keep the original rank. Return sequence number of currently best solution, for example 0,1,2,3, DO NOT RETURN ANYTHING ELSE EXCEPT FOR NUMBERS SPLIT by
Figure 15: Self-correethess prompt of text-guided image editing.
You are ranker for planner who use the candidate modules to carry out the instruction: QUESTION, select the best solutions for carrying out the instruction candidate modules include: PART: take apart object, SEG: segment, ALIGN: align object with query, RESULT:
up the final result
Current solutions:
PLAN
PLAN If the modules in the solutions have better cause-and-effect relations, and likely answer the question, please rank first. you are unsure, please keep original rank. Return sequence number of currently best solution, for example 0,1,2,3, DO NOT RETURN ANYTHING ELSE EXCEPT FOR NUMBERS SPLIT by
Figure 16: Self-correctness prompt of visual abstract reasoning.
22

---

Published as a conference paper at COLM 2024
E.3
Details of Visual Abstract Reasoning
Query:
Images:
"submarine"
PART
SEG
'bo12'
"submarine"
Parts:
Segments:
gropellent
SELECT
"submarine"" Images
tower"
ALIGN
Segments
Figure 17: Implementation of abstract reasoning.
In Fig. 17, we demonstrate our implementation of compositional methods on KILOGRAM dataset. Given an image, we segment into several parts. At the same time, we adopt LLM to parse the query to several components. After that, we match the visual and textual components by their semantic similarity. Finally, we take the alignment score to retrieve the best matched image.
Images:
Program:
Verify:
PART
Image-Tex' Matching
man in hat and robe
hat, robe, head...
PARTSO
SEGS
man, hat, robe, head, arms, legs, feet
hat:8.6 robe:0.5
OBJO
ALIGN
PARTSO, OBJO
"man in hat and role"
hat:0.55 robe:0.4
Query:
Answer:
man in hat and robe
Figure 18: Implementation of ExOVIP on abstract reasoning.
23

---

Published as a conference paper at COLM 2024
E.4 Implementation details
In practice, for the verification modules, we set the 2.0 for LOC module, 1.5 for SELECT module, and ALIGN module, 1.2 for other modules. For the negative sampling strategy, we select words sharing semantic similarity less than 0.5 to construct the semantic opposite vocabulary and randomly sample one semantic opposite for each answer. In the searching process, we set up K as 4, and as 2. To improve the efficiency of our search algorithm, set the branching factor 3. To make the comparison fair, use the same fewer examples in the prompts for our methods, and select the verification modules from the operation modules. We apply our experiments NVIDIA A100 GPU and NVIDIA 3090Ti GPU.
Qualitative study.
E1
Qualitative examples
We additionally exhibit more examples that can be improved by our method. As is shown in these examples, all five types of tasks could be further improved by our framework.
What material cup the of the laptop, plastic glass?
BOx0-L0C| IMAGE, object- "laptop=
BOX0-LOC[magg-IMA55 object- "laptop
MMAGD--CROPPmggeIMMGG bbox=BOX0)
IMAGED-CROP/(mggIMMGG bbox-BOX0)
BOX1- imggo- IMAGE0, objoct- "cup")
BOXI=LOC(magee IMAGEO, object='cup")
MMAGEL-CROPÚMAS
bbox- BOX1]
IMAGEL-CROPlImge= IMAGEI, bbox=BOX1)
AN IMAGE,
ANSWERR--OOAjmgg--MMMA.
question= What material the cup?"]
X
question= What material is the cup?")
result wrongly predicted
correctly predicted plastic
ANSWERI-EVAL exppresssss "plasiic if
verification DAWEEERLEVVALLssrrssso====""""""
ppasstcc "glass") glass
if
ANSWERO 'plastic' else "glass")) plastic
(a) VisProg
(b) ExoViP
Figure 19: Qualitative study for GQA.
man in hat and robe
PARTSO-PARTIE PATTSO0-PAR((uurr--- "man in and robe") man, hat, robe, head, arms, legs, feet
MARTSO-PARTqqurryy "man and robe") man, hat, robe, head, arms, legs, feet
DeJ0-SEGS/mageeMMMGE
IMAGEO=ALIGNI MaGEE:0=AAGNNNiaaaaeee mage=IMAGE. objects
IMAGEO-ALIGN(ime AAAG--AAAAG(mgA-MMAAGGGG objecte
OBJ0,part-PARTSO bbox="men "men in hat and robe") the the ALTGN can not exactly match
part, partPpart-PRRTSO bbox= "man in hat and robe")
the the ALIGN better match after verification
(a) VisProg
(b) ExoViP
Figure 20: Qualitative study for KILOGRAM.
24

---

Published as a conference paper at COLM 2024
Q: Tag the left zebra
malo MAGE, object="tebra" OoUi-FiLTERLmagg-iMAA obejct=OBJ0,
OBI0-LOC[Iaag-IMMGG "abbr""
BBj-=FILTRR/mage-MMoG obejct-OBIO,
query=
query=
[OBII-FILTER mag.-
redundant after venfication
Equery= bebra")
IMAGE0= BAGGE0--AGG(aage=== IMAGEO, object= OBJ1)
#nredonannt Sep
MAAIa-- IMAGEO, object- OB11)
VisProg
(b) ExoViP
Figure 21: Qualitative study for RefCOcO.
Q: The left image features single fur-trimmed fingerless mitten with small embellishments dotting front, and the right image shows a pair of fur-trimmed half-mitts with no thumb part showing.
ANSWERO-VOA( +LEFT, Howw
ANSWERO-VOA( +LEFT, question="How:
many fur-trinmed fingeriess mittens the
many fur-trimmmed fingeriess mittens Image?"!
image?")
ANSWERI-VQAimn #RIGHT, Image?"1 many tuittmmmeed
"How ANSWERS-VQAjiaag #RIGHT, many nur-trimmed
Image?")
NNSWER2-VQAAmagg #LEFT, question="Are 'Are
ANSWER2-VOAlima #LEFT, question= 'Are embellishments the the mitten?" )*
embellishments dotting the mitten?" )*)
ANSWER3-VQA(mass SRIGHT question="ls part
ANSWER3-VOA(man #RIGHT, question='1s
halfmmitts?"
showing
ANSWERL VAL (al.aaaa (epresssinn ANSWERO
ANSWERA-EVAI/ wtt//v/pe/epsioonnn ANSWERO --
and
ANSWERS-EVAL/ (Alpprrsso/s not ANSWERO
ANSWERS-EVAL/ (//pee/esi ANSWER1
and ANSWER1 -- and not ANSWER3)
and ANSWER3)
ANSSERSSEVVALoopnssdii ANSWER4
SWANERSS-EVA/xxpresssooo ANSWER4 and ANSWERS)
ANSWERS]
X
logic
(a) VisProg
(a) ExoVip
Figure 22: Qualitative study for NLVR2.
25


Reviews:
[{"review_id": "nEMIOsmn7r", "reviewer": "Reviewer_pE53", "summary": "The paper presents an \"exoskeleton\" approach to compositional visual reasoning as an amelioration to the visual programming work. Considering the planning error and the module execution error in the visual reasoning process, this work introduces three verifiers and a series of other mechanisms (TRS, self-correction) to enhance the performance in GQA and many other reasoning tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and acknowledgment of our reasonable idea.\n\n> W1: The introduction of 3 verifiers, together with TRS and self-correction is supposed to cause extra latency at inference. So I am skeptical about whether the gains in performance are worth it.\n\n**A:** In Appendix E, we present an analysis of inference time, as detailed in Table 9. The data indicates that the majority of the cost arises from planning time, which we attribute to the OpenAI API. To mitigate this latency, we plan to deploy a LLM locally. Furthermore, the current performance gains are constrained by our choice of verifier. For the sake of a fair comparison, we have selected the verifier from the operation modules. Future improvements could be achieved by introducing stronger verifiers.\n\n\n\n> W2: The author(s) claim that the proposed exovip framework can mitigate the planning error and present an error analysis in Appendix, which is informative. However, I suggest that presenting the accuracy of planning as an important intermediate step would be more intuitive.\n\n**A:** We thank you for your constructive suggestion. To clarify our approach, we have emphasized the connection to Appendix D in our introduction and will incorporate additional details regarding the accuracy of planning in the main body of our text.\n\n\n> W3.1: The introduction of 3 verifiers lacks a clear motivation and we have little idea why they should be designed in this way. \n\n**A:**  Your query is valid. We have chosen VQA, Caption, and CLIP as verifiers for the following reasons:\n\n- We selected verifiers from the operation modules to ensure a fair comparison.\n- We opted for three verifiers to maintain a balance between computational efficiency and performance.\n- We chose three distinct types of Vision-and-Language (VL) models that are commonly employed in multitask learning[1].\n\n[1] Lu, Jiasen, et al. “12-in-1: Multi-Task Vision and Language Representation Learning.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 10434-10443.\n\n> W3.2 Maybe a more in-depth analysis of how the verifiers collaborate with each other would be helpful.\n\n**A:** We appreciate your recommendation. To provide further clarity, we describe the distinct roles of the verifiers as follows:\n\n- VQA: Offers a discriminative viewpoint for VL comprehension.\n- Caption: Generates results with a broader, open-vocabulary approach.\n- CLIP: Demonstrates alignment and representation learning within VL models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The work introduces verifiers to rectify the errors in the reasoning process, which sounds reasonable. \n+ The writing is clear and easy to follow.\n+ It involves extensive experiments spanning multiple vision benchmarks.", "reasons_to_reject": "+ The introduction of 3 verifiers, together with TRS and self-correction is supposed to cause extra latency at inference. So I am skeptical about whether the gains in performance are worth it.\n+ The author(s) claim that the proposed exovip framework can mitigate the planning error and present an error analysis in Appendix, which is informative. However, I suggest that presenting the accuracy of planning as an important intermediate step would be more intuitive.\n+  The introduction of 3 verifiers lacks a clear motivation and we have little idea why they should be designed in this way. Maybe a more in-depth analysis of how the verifiers collaborate with each other would be helpful.", "questions_to_authors": "See the weaknesses above.", "extra_scores": {}}, {"review_id": "65BnVvwa9y", "reviewer": "Reviewer_cnU5", "summary": "This paper introduces a new plug-and-play approach called 'EXOVIP,' to improve compositional visual reasoning and address two challenges in current visual programming: module prediction error and planning error.\n\"Exoskleton\" verification modules (that consists in a mix of image-text matching verifier, image captioning verifier and VQA verifier) are used to verify the prediction after every step, followed by corrections or refinings.\nThe experiments conver six compositional visual reasoning tasks covering real and abstract images/videos, and shows consistent improvement.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our methods' potential to improve performance and generalization on open-domain multimodal challenges. We carefully address each of your concerns as follow.\n\n\n> W1: There's a concern about the efficiency of the method, as Table 9 shows that the total inference time of EXOVIP is 9.6x the time of LLaVA-1.5-7B. The authors indicated that Internet latency may be a cause, it could be helpful to have an estimate using open-source LLMs.\n\n**A:** We have replaced GPT-3.5-turbo to Llama-2-13B-AWQ. Specifically, we have deployed the original Llama-2-13B-AWQ model on a single Nvidia A800 GPU using the vllm-4.0 framework on a cluster, without employing any additional acceleration strategies. The following results were obtained:\n\n| model   | Total Inference Time (s) |  Planning Time (s)   | Module Inference Time (s) |\n| ------- | -------------------- | --- | -------------------------- |\n| ExoViP(GPT-3.5-turbo)  | 4.32                 |   3.64  | 0.68         |\n| ExoViP(Llama-2-13B-AWQ)  | 3.94                 |   2.99  | 0.95         |\n\nWe observe that the planning time decreased but module time increased. We believe this is attributed to the fact that the LLaMA model induces more planning errors, resulting in a significant rise in search steps. In the future, we aim to implement more sophisticated models, employing GPU acceleration tactics.\n\n> W2: There's inconsistencies in the paper about the number of tasks and it's quite confusing.\n\n**A:** Thank you the catch. This is due to the new task we performed close to the deadline. The total number of tasks is six. We apologize for the confusion. We will carefully revise our presentation accordingly.\n\n\n> Q1: Could you add a short definition of the prediction error and planning error or a reference to the definitions in the caption of Figure 1, so that the very first figure is more self-contained & easy to understand?\n\n**A:** Thank you for your suggestion. We will add a brief definition and references of \"planning error\" and \"module error\" as stated in the Introduction Section to Figure 1.\n\n\n> Q2: It could be clearer for the reader if the authors explicitly specify \"NLVR2\" in the caption of Table 4, as specified in Section 5.3.\n\n**A:** Thank you for your suggestion; we will change \"NLVR\" to \"NLVR2\" accordingly.\n\n> Q3: Typo in 5.3, paragraph \"Text-guided Image Editing\": \"Appendix ??\"\n\n**A:** Thanks. We have fixed these typos and will carefully proofread our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The writing is pretty clear and easy to follow.\n\nThe paper proposes a novel and clear approach using verification modules to deal with the planning and module errors in compositional visual reasoning.\n\nThe paper empirically evaluated the approach on a variety of tasks and shows consistent improvement across the tasks, notable for some (e.g. for the compositional VQA tasks). \nThe evaluations cover both real images and abstract synthetic ones.", "reasons_to_reject": "There's a concern about the efficiency of the method, as Table 9 shows that the total inference time of EXOVIP is 9.6x the time of LLaVA-1.5-7B. The authors indicated that Internet latency may be a cause, it could be helpful to have an estimate using open-source LLMs.\n\nThere's inconsistencies in the paper about the number of tasks and it's quite confusing: sometimes the paper refers to \"the outcome has been a uniform enhancement in performance across six diverse tasks\", whereas at multiple other places (including in the abstract and appendix) the paper refers to \"consistent improvements on five compositional reasoning tasks\".", "questions_to_authors": "1. Could you add a short definition of the prediction error and planning error or a reference to the definitions in the caption of Figure 1, so that the very first figure is more self-contained & easy to understand?\n\n2. It could be clearer for the reader if the authors explicitly specify \"NLVR2\" in the caption of Table 4, as specified in Section 5.3.\n\n3. Typo in 5.3, paragraph \"Text-guided Image Editing\": \"Appendix ??\"\n\nNot a question, but I appreciate the analysis about errors/efficiency conducted in the Appendix.", "extra_scores": {}}, {"review_id": "B7ouYpBt2d", "reviewer": "Reviewer_ouxC", "summary": "The work aimed to tackle the planning mistakes and execution errors from existing LLMs.\nThe proposed method, ExoViP, corrects errors at both the planning and execution stages through introspective verification.\nExoViP employs a verification module with three sub-verifiers to validate predictions and refine the reasoning trace planned by LLMs.\nExperimental results show consistent improvements on five compositional reasoning tasks on standard benchmarks.\nExoViP has the potential to improve performance and generalization on open-domain multimodal challenges.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and acknowledging of our effective method. We carefully address each of your concerns as follow.\n\n\n> W1: Things to improve include the fact that Figure 1 is not very comprehensive\n\n**A:** Thank you for your suggestion. We aim to create a clearer and more polished figure to effectively present our framework. Specifically, we will  include subverifiers and TRS in figure 1. For the high-level figure, we will extend figure2 to our framework overview. Due to the limited page numbers, we will add in the final version (with 1 extra page).\n\n\n> W1: Another weakness is that the proposed verification is simply a combination of existing works.\n\n**A:** Thank you for your feedback, to address your concerns directly and provide a more comprehensive understanding, I will delve deeper into the analysis of the verifiers and tree-based search algorithm with other works:\n\n- **Our verification strategy is more than introduce additional modules:** \n    - **Significance of Negative Sampling**: In Section 5.1.1, our findings indicate that verifiers on their own may perform suboptimally. This observation led us to implement a negative sampling strategy. This score is not an absolute measure but a relative one, designed to robustly reflect the model's confidence in its predictions.\n    - **Synergy of Multiple Subverifiers**: We never introduce new modules from outside, all the verifiers comes from the operation modules. And Table 2 highlights the significance of combining different subverifiers. \n\n- **Our tree-based strategy v.s ToT, MCTS**\n    - **Interplay Between Verifiers and Reasoning Trace**: A key innovation of ours is the integration of verification scores to optimize the reasoning trace. As shown in Figure 3, when we combine the tree-based search algorithm with verification scores, there is an improvement in the variance of these scores. This enhancement is instrumental in differentiating between various reasoning traces, ultimately aiding in the identification and selection of the most accurate one. Thus, the subverifiers and the trace search algorithm mutually reinforce each other's effectiveness.\n    - **Efficiency:** As we stated in Appendix E, we assert that our beam searchbased method can achieve an better equilibrium between efficiency and effectiveness comparing with the previous search strategy.\n\n> W1: Some typos need to be fixed.\n\n**A:** Thank you for your suggestions, we would like to revise our paper carefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The introduced \"plug-and-play\" EXOVIP for correcting errors is straightforward, and ablation studies demonstrate the effectiveness of the components.\nThe verification module uses a mixture of three sub-verifiers to validate predictions, providing a comprehensive and robust approach to error correction.\nExperimental results show consistent improvements on five compositional reasoning tasks, demonstrating the effectiveness of the ExoViP method.", "reasons_to_reject": "There are not many serious weaknesses in the paper.\n1. Things to improve include the fact that Figure 1 is not very comprehensive; some of the terms, e.g., LOC, VQA[BOX], are unexplained in the beginning. Also, the writing in section 4 should link to Figure 1 to explain each component regarding its target in the pipeline. It would be better to include a high-level figure at the beginning.\n2. Another weakness is that the proposed verification is simply a combination of existing works. Also, the idea is in line with the motivation of the Tree of Thoughts, where performance gain is expected.\n3. Some typos need to be fixed, e.g., Appendix ?? at section 5.3", "questions_to_authors": "Overall, it is a decent paper. Please address the weakness above.", "extra_scores": {}}, {"review_id": "hZ1GzE9h9G", "reviewer": "Reviewer_MW2q", "summary": "The paper proposes two plug-and-play components to alleviate limitations in existing vision programming approaches with LLMs for compositional visual reasoning. \nFirst, to improve accuracy of visual execution modules, it reuses visual modules in visual programs as additional verification modules. Negative sampling is proposed to calibrate scores of the verification modules. \nSecond, to reduce LLM planning mistakes, it uses tree-based reasoning trace searching and selects the best path. \nExperimental results on five tasks demonstrate the effectiveness of the proposed method over the baseline VisProg.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments\n\n> W1: strange negative sampling.\n\n**A:** The score computation, influenced by recent research in truthfulness, commonsense, and bias (Section 4.1), is a verifier-derived confidence score that becomes a normalized score through the inclusion of negative samples, enhancing system robustness and accuracy, and allowing unbiased model-blind confidence assessment of planning traces. The term \"antonym\" is defined by the greatest distance in VL semantic space using the CLIP text encoder, leading to unconventional pairings like \"night-stand\" and \"stocking\" based on computational results rather than human knowledge.\n\n\n> W2: The algorithm to reduce planning mistakes is not clear. (k step, measure plan)\n\n\n**A:** We manually assessed the results of applying our algorithm to VisProg, confirming its effectiveness in reducing planning errors (Appendix D). Our model, as described in Section 4.2 and shown in Algorithm 1, selects K steps based on ranked verification scores. Higher verification scores suggest more accurate planning, making them a better measure than existing heuristic methods.\n\n\n\n\n\n> W3.1: missing-ablation: The hyper-parameters are not evaluated\n\n**A:** Per your request, the results are as follows:\n\nK=3: 60.85; K=4: 61.49\n\nP=1: 60.98; P=2: 61.49\n\n\n> W3.2: missing-ablation: vanilla ensemble LLM.\n\n\n**A:** Indeed, we've compared an LLM-enhanced tree-search, as shown in Table 1, Line 4. This method replaces Algorithm 1's sorting, ranking, and selecting steps with LLM, which can serve as a stronger baseline.\n\n\n> W4.1: confusing-presentation: The names of different variants of the EXOVIP model are not straightforward to know what the variant is. (TRS in section 5.1 and table 3)\n\n**A:** To clarify, in line 3, \"ExoVIP w/o self-correctness\" means using the TRS strategy only. In line 4, \"ExoVIP w/o verification\" means using the PSC only. Plans are to clarify this further.\n\n\n> W4.2: table 2 not match Table 1\n\n\n**A:** We separating our tree-search strategy and verification score for clear comparison. Our method uses a prediction score for tree-search, then employs an LLM to pick error-free planning traces, fixing the planning tree. This lets us assess the verification score alone. The verification score isn't part of our framework, hence its absence in Table 1. This matches the example in Appendix B with a value of 60.034, identical to Table 2's mixture score. We can offer more details if space allows.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed two components to alleviate mistakes in visual module execution and LLM planning are intuitive and interesting.\n- The effectiveness of the proposed approach is demonstrated on five benchmark datasets and two baseline methods.", "reasons_to_reject": "1. The negative sampling for score calibration is a bit strange. It uses an “antonym” for each answer for calibration. However, the way of selecting antonyms is simply using cosine similarity of text embeddings. Why does it make sense? It seems strange that the antonym of “night- stand” is “stocking”.\n\n2. The algorithm to reduce planning mistakes is not clear. For example, how does the model sample K steps? How do the verification modules measure the plans?\n\n3. Some ablations are missing.\n- The hyper-parameters are not evaluated, such as the scaling factor \\tau of the verification scores, the number of selected steps K and P.\n- The proposed algorithm 1 should compare with simple approaches such as aggregating multiple paths from LLMs with different initial seeds etc.\n\n4. The presentations of table 1 and 2 are confusing.\n- The names of different variants of the EXOVIP model are not straightforward to know what the variant is. The textual descriptions sec 5.1.1 help understand these variants but it is still less clear for Line 3 and 4. Line 3 in Table 3 is a model without TRS but the description in Page 7 says it is a model simply with TRS. Similar for Line 4. \n- The numbers in table 2 do not match any numbers in Table 1. Why?", "questions_to_authors": "See reasons to reject.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces ExoViP, a "plug-and-play" method to correct errors in both the planning and execution stages of compositional visual reasoning through introspective verification. The proposed verification module, consisting of three sub-verifiers, validates predictions after each reasoning step and refines the reasoning trace planned by LLMs. The reviewers acknowledge the clear writing, the novelty of the approach, and the extensive experiments demonstrating consistent improvements across various compositional reasoning tasks. Although some concerns were raised regarding the efficiency of the method, the inconsistencies in the number of tasks mentioned, and the lack of clarity in certain aspects of the algorithm and hyperparameter evaluation, the overall consensus is that the paper makes a valuable contribution to the field. The AC believes that the negative sampling part is acceptable, given the explaination of the authors from the rebuttal. Considering the strengths and the potential for improved performance and generalization on open-domain multimodal challenges, I recommend accepting this paper.

[comment from the PCs] The term "antonym" has a very specific linguistic definition, which does not align with how it's used in this paper. This led to misunderstandings by the reviewers, which significantly overshadow the evaluation of the paper. What more, using this term does not help the paper, it just hurts it. For the next version of the paper, we ask the authors to replace this term with something more appropriate. This is critical to meet publication standards.
Decision: Accept