Analyzing Modular Approaches for Visual Question Decomposition
Authors: Apoorv Khandelwal
Ellie Pavlick
Chen Sun
Venue: EMNLP 2023 Main (2023)
Keywords: Vision–language
visual question answering
modular
neuro-symbolic
prompting
question decomposition

Abstract: Modular neural networks without additional training have recently been shown to surpass end-to-end neural networks on challenging vision–language tasks. The latest such methods simultaneously introduce LLM-based code generation to build programs and a number of skill-specific, task-oriented modules to execute them. In this paper, we focus on ViperGPT and ask where its additional performance comes from and how much is due to the (state-of-art, end-to-end) BLIP-2 model it subsumes vs. additional symbolic components. To do so, we conduct a controlled study (comparing end-to-end, modular, and prompting-based methods across several VQA benchmarks). We find that ViperGPT's reported gains over BLIP-2 can be attributed to its selection of task-specific modules, and when we run ViperGPT using a more task-agnostic selection of modules, these gains go away. ViperGPT retains much of its performance if we make prominent alterations to its selection of modules: e.g. removing or retaining only BLIP-2. We also compare ViperGPT against a prompting-based decomposition strategy and find that, on some benchmarks, modular approaches significantly benefit by representing subtasks with natural language, instead of code. Our code is fully available at https://github.com/brown-palm/visual-question-decomposition.

Analyzing Modular Approaches for Visual Question Decomposition
Apoorv Khandelwal and Ellie Pavlick and Chen Sun
Brown University
Department of Computer Science (apoorvkh, ellie _alllikk (thensunj@brownn.com edu
easily interpretable and can be executed (leverag- ing distinct image or language processing modules) to solve the task at hand. The most recent such mod- end-to-end neural networks challenging vision-language tasks. The latest such meth- ods simultaneously introduce LLM-based code generation to build programs and number of skill-specific, task-oriented modules to execute them. In this paper, we focus on ViperGPT and ask where its additional performance comes from and how much is due to the (state-of-art, end-to-end) BLIP-2 model subsumes ad- ditional symbolic components. To do so, we conduct controlled study (comparing end-to- end, modular, and prompting-base based methods across several VQA benchmarks). We find that ViperGPTTss reported gains over BLIP-2 can be attributed to its selection of task-specific modules, and when we rum ViperGPT using a task-agnostic selection of modules, these gains go away. ViperGPT retains much of its performance if we make prominent alter- ations its selection of modules: e.g. remov- ing or retaining only BLIP-2. We also compare ViperGPT against a prompting based based decom- position strategy and find that, on some bench- marks, modular approaches significantly ben- cfit representing subtasks with natural lan- guage. instead of code. Our code is fully avail- able at https: thub. www.bbowwwwnnlooooe susuull-uusttionddccmmmmpp tion.
Abstract
Modular neural networks without additional training have recently been shown to surpass
els (Gupta and Kembhavi, 2023; Surís et al., 2023; Subramanian et al.. 2023) are training-free. they leverage large language models (LLMs) to generate programs and subsume powerful neural networks as modules. Such approaches demonstrate strong results and outperform end-to-end neural networks on zero-shot vision language tasks.
These recent modular approaches typically in- clude state-of-the-an end-to-end networks, among a complex schema of other modules and engineer- ing designs. As a result, the contribution of these networks is difficult to disentangle from the mod- ularity of their overall system. Thus, in this pa- per, we analyze ViperGPT, in which BLIP-2 (Li et al., 2023) is a constituent module, represen- tative example of recent and performant modular system for Assion-language tasks. BLIP-2 can par- ticularly (and in contrast from ViperGPT's other modules) solve VQA tasks on its own. We ask: where does its additional performance come from, and how much is due to the underlying BLIP-2 model the additional symbolic components? To answer research questions, we conduct con- trolled study, comparing end-to-end, modular, and prompting-base approaches (Sec. 2) on several VQA benchmarks (Sec. 3). We make the following specific contributions:
1 Introduction
End-to-end neural networks (Li et al., 2023) have been the predominant solution for vision- language tasks, like Visual Question Answering (VQA) (Goyal et al., 2017). However, these meth- ods suffer from lack of interpretability and gener- alization capabilities. Instead, modular (or neuro- symbolic) approaches (Andreas al., 2015; John- son et al., 2017; Hu al., 2017: Yi al., 2018) have been long suggested as effective solutions which address both of these limitations. These methods synthesize symbolic programs that are
Section 4, we find that ViperGPT's advan- tage over BL.I--2 alone is likely due to the task-specific engineering in ViperGPT. Specif- ically, we run ViperGPT in a task-agnostic setting, in which we do not preselect different subsets of modules for each task (as is donc in Surís et al. (2023)). We find that, without the task-specific module selection, the aver- age gain of ViperGPT over BLIIP- disappears (dropping from +8.7% to -0.8%). Moreover, we find that removing the BLIP-2 module re-

---

Prompt
Prompt
API
Question
In-context Examples
Question
Modules
BLIP-2
Image
Large Language
Question
LLM (QA)
Model
Large Language Model
Follow-up Q
Other NNs
Image
BLIP-2 Prediction
Symbolic Algorithms
OR
Code ()
Neural Network
for VQA
Prediction
Follow-up Question
2 Python Interpreter
Prediction
Image
BLIP-2
Prediction
End-to-end (BLIP-2)
Modular (ViperGPT)
Successive Prompting
Figure I: A diagram of the end-to-end, modular, and prompting-basee models (Sec. 2) we explore in this paper. Each setting receives image and question as input and produces prediction as output (at which time will terminate). Similar colors across models in this diagram indeed refer to the same modules.
tains a significant percentage of ViperGPT's task-agnostic performance (i.e. 84% for the direct answer setting and 87% for the multi- ple choice setting). And, retaining only the BLIP-2 module comprises 95% and 122% of ViperGPT's task-agnostic performance in those settings.
lyze the code that is generated by ViperGPT and observe higher runtime error rates for A- OKVQA's direct answer setting (3%) and the multiple choice benchmarks (12-18%) than the in-domain direct answer benchmarks (1- 2%). Finally, while the syntax error rate is 0% for all direct answer benchmarks, it is 1-3% for multiple choice benchmarks.
2. In Section 5, find that prompting prompting-based based (rather than code-based) method for question decomposition still constitutes 92% of the performance of an equivalent ViperGPT vari- ant for direct answer benchmarks. Morcover, this method actually exceeds ViperGPT by + +12% on average for multiple choice bench- marks. (To the best of our knowledge, our prompting-base bbased method also presents the highest score in the multiple choice setting of A-OKVQA (Schwenk et al., 2022) compared to any other training-free free method.) These re- sults suggest that, on some benchmarks, mod- ular approaches significantly benefit by rep- resenting subtasks with natural language, in- stead of code.
Models
In this section, we share specific design decisions and implementation details for each the three model families we assess in this paper (i.e. end-to- end, modular, and prompting- based). We addition- ally visualize these approaches in Fig. 1.
2.1 End-to-end
As the end-to-end model in our analyses, we use BLIP-2 (Li et al., 2023), an open-source state- of-the-art vision-language model which can be used for image captioning and zero-shot VQA. For VQA, this model first encodes images using pre- trained image encoder and projects the resulting encoding into the input space of a pre-trained lan- guage model. That language model is then used to generate textual prediction, given the aforemen- tioned image projection and VQA question.
3. In Section 6, we explore ViperGPT's gener- alization to oft-of-domain benchmarks. Un- like for in-domain datasets, find that pro- viding task-specific in-context examples ac-
As in Li et al. (2023), prompt this model with "Question: O Short answer: []". For the direct answer setting, generate text directly from the language model. For the multiple choice setting, we select the choice with the maximum log likelihood for text generation.
tually leads to performance drop by % for VOK''''s direct answer setting and 2% on average for A-OKVQA and ScienceQA multiple choice settings. We additionally ana-

---

As simple example, the question "How many black cats are in the image?" might be written as: We use the same settings as Surís al. (2023, ViperGPT) (i.e. the modular approach in Sec. 2.2) to load and run the model. Specifically, we use the ViT-g/14 image encoder from EVA-CLIP (Fang et al., 2023) and F5-XT5-XXL encoder-decoder language model (Chung et al., 2022). We make predictions using 8-bit inference (Dettmers et al., 2022) and generate text with beam search (width=
ViperGPT paper (Surís al., 2023), However, as that codebase currently² differs from the original paper in several ways, we have modified it to re-align with the original report. Specifically, we switch the code-generation model from ChatGPT to Codex, revert the module set and prompi text to those in Surís et al. (2023, Appendix A-B), and otherwise make minor corrections to the behavior and execution of ImagePatch and execute_ comcuueecoommnd... However, we find that only the full API prompt is made available- the task-specific prompts-ppeeenninn us from exactly replicating Surís et al. (2023).
5, length penalty -1).
2.2 Modular
In this paper, we use ViperGPT (Surís et al.. 2023), which is recent modular system for vision- language tasks.
Design choices. In our experiments and like Surís et al. (2023), we prompt Codex llode-davinci-eep (Chen al., 2021a) for code generation and use the same set of neural modules: GLIP (Li et al., 2022), MiDaS (Ran- ftl et al., 2020), BLIP-2 (Li et al., 2023), X-VLM (Zeng et al., 2022), and InstructGPT (text (eott-davinci-00) (Ouyang et al., 2022).
ViperGPT prompts language model with VQA question and an API-which is an interface for manipulating images-to to generate a Python program. This API written in code and describes a class ImagePatch and several functions, like .find, simple_queer etc. These functions in- voke both symbolic algorithms (e.g. for iterat- ing through elements, sorting lists, computing Eu- clidean distances, etc.) and trained neural network modules (for object detection, VQA, LLM queries. etc.). When the Python program executed, these functions should manipulate the VQA image and answer the question.
For fairness in comparing with the model in Sec. 2.3, we make a few additional design de- cisions that deviate from Surís et al. (2023). For our task-agnostic variant (Sec. 4), we use the full ImagePatch API and external functions (exclud- ing the VideoSegment class) our prompt. We specify further modifications for our other variants in Sec. 4. We prompt the model with the follow- ing signature: "def execute_commant (image) -> str:" (i.e. we explicitly add "-> str" to bet- ter conform to the VQA task). During multiple choice, Surís et al. (2023) provides another argu- ment, possible_choices. the signature. How- ever, we extend this argument with an explicit list of these choices.
def execute_come seccute_coommnddimmae image_patch
str: (ameeattchiimage
cat_patches = image_patch. fiiihhhffiddd (ind('cat')
black_cat_patches =
choices,
for in cat_patches if
p. ww..erffypropertycC c'at,, "black') ]
For the direct answer setting, we use the text as returned by the program as our predicted an- swer. This text may be generated by a number of modules (e.g. BLIP-2 or InstructGPT) or as a hard- coded string in the program itself. For the multiple choice setting, the returned text is not guaranteed to match a choice in the provided list. So, we map the returned text to the nearest choice by prompting In- structGPT (text-davinci-e00) with "Choices: O Candidate: O Most similar choice: []". We select the choice with the highest log likelihood.
return len(black_cat_pptttll
The program can then be executed using the Python interpreter and several modules. In this case, ImagePatch.fing (objectttt str) -> [isg[[haa utilizes an ob- ject detection module to find all cats and ImagePatch. verify (orpjpptttyobjettnn.ee (object_name: str, property: str) -> bool utilizes text- image similarity for determining whether those cats are black.
We elaborate further on our design choices and how they make our experiments more fair in Ap- pendix B.
Implementation. Our implementation of ViperGPT uses the code! released with the original
"As of Octoher 22, 2023.

---

2.3 Successive Prompting
3.1 Benchmarks
We evaluate methods on a set of five di- verse VQA benchmarks: VQAv2 (Goyal et al.. 2017), GQA (Hudson and Manning, 2019), OK- VQA (Marino et al., 2019), A-OKVQA (Schwenk et al., 2022), and ScienceQA (Lu et al., 2022). We use the following dataset splits as benchmarks: validation (1000 random samples) for VQAv2, test- dev for GQA, testing for OK-VQA, validation for A-OKVQA, and validation (IMG subset, QG
Building programs is not the only way decom- pose problems. Recent work in NLP has found that large language models improve performance at reasoning tasks when solving problems step- by-step (Wei et al., 2022; Kojima et al., 2022). For question answering, decomposing question and answering one sub-question at time leads to further improvements (Press et al., 2022; Zhou al., 2023; Dua al., 2022; Khot et al., 2023). Moreover, recent work has started to invoke vision- langauge models based on the outputs of langauge models.
A format) for ScienceQA.
These datasets vary in the amount of percep- tion, compositionality, knowledge, and reasoning their problems require. More specifically: VQAv2 is longstanding benchmark whose questions quire primitive computer vision skills (e.g. classi- fication, counting, etc). GQA focuses on composi- tional questions and various reasoning skills. OK- VQA requires "outside knowledge" about many categories of objects and usually entails detect- ing an object and asking for knowledge about that object. A-OKVQA features "open-domain" ques- tions that might also require some kind of common- sense, visual, physical reasoning. ScienceQA features scientific questions (of elementary through high school difficulty) that require both background knowledge and multiple steps of reasoning to solve. We elaborate further in Appendix D.
As a convergence of these directions, we introduce a training-fres free method that jointly and successively prompts an LLM (InstructGPT: text-dad daxiniaainc))002) and VLM (BLIP-2) to de- compose visual questions in natural language. We call this "Successive Prompting" (following Dua al. (2022)). At each step, our method uses the LLM to ask one follow-up question at a time. Each follow-up question answered independently by the Languaee model. In the subsequent step, the LLM uses all prior follow-up questions and an- swers to generate the next follow-up question. After some number of steps (as decided by the LLM), the LLM should stop proposing follow-up questions and will instead provide answer to the original question. We constrain the LLM's behavior by ap- pending the more likely prefix of "Follow-up:" or "Answer to the original question:" (i.e. the stopping criteria) to the prompt at the end of each step.
3.2 Metrics: Direct Answer
We evaluate the direct answer setting for VQAv2, GQA, OK-VQA, and A-OKVQA. In this setting. method will predict textual answer given image and question. We report scores using (1) the existing metric for each dataset and (2) the new InstructGPT--eal metric from (Kamalloo et al., 2023).
In order to prompt large language model for this task, we provide a high-level instruction along with three dataset-speciff in- in-context demonstra- tions of visual question decompositions.
Our method generates text directly, which can be used for the direct answer setting. Like with ViperGPT, we also prompt our method with an explicit list of choices during the multiple choice setting. And, for the multiple choice setting, we select the choice with the highest log likelihood as the predicted answer.
We observe that while the general trends (de- termining which models perform better or worse) remain the same between metrics (1) and (2), the actual gap may differ significantly. See Appendix A
for further discussion of why (2) a more robust measure of model performance for our experiments. We include (1) for posterity Tables 1 and 2, but make comparisons in our text using (2), unless specified otherwise.
3 Evaluation
(1) Existing metries. GQA uses exact-match accuracy with single ground truth annota- tion. VQAv2, OK-VQA, and A-OKVQA use the
We evaluate variants of end-to-end, modular, and prompt-based methods (Sec. 2) on a set of VQA benchmarks (Sec. 3.1) using direct answer and mul- tiple choice evaluation metrics (Secs. 3.2 and 3.3).
VQAv2 evaluation metric: a prediction is matched with 10 ground truth annotations (i.e. acc. =

---

max 0, num_matches / 3)). Note: VQAv2 and OK-VQA pre-process answers before matching.
BLIP-2 reduced from +6.2% to +2.1% on GQA and +11.1 1% to -3.6% on OK-VQA (using the ex- isting metrics). 4 We continue to observe that our task-agnostic ViperGPT variant usually does not perform better than BLIP-2 across benchmarks and metrics, with the exception of the multiple choice setting of A-OKVQA, on which ViperGPT does outperform BLIP-2 significantly.
(2) InstructGPT-eval. Kamalloo et al. (2023) find that lexical matching metries for open-domain question answering tasks in NLP perform poorly for predictions generated by large language models. We make similar observations for the existing direct answer metrics in the VQA datasets we benchmark: such scores correlate quite poorly with our intu- itions for open-ended text generated by language models. For example, the prediction "riding horse" would be marked incorrect when ground truth an- swers are variants of "horseback riding". Instead, Kamalloo al. (2023) suggest an evaluation met- ric (InstructGPT-eval) that prompts InstructGPT (text-davinci-003) (Ouyang et al., 2022) with
Since ViperGPT relies on BLIP-2 as one of its modules (i.e. in simple_query for simple visual queries), we wonder how much influence BLIP-2 has in the ViperGPT framework. Moreover, how much does ViperGPT gain from having modules and functions in addition to BLIP-2?
Accordingly, we two ablations: evaluate the performance of ViperGPT without BLIP-2 and with only BLIP-2 (i.e. with no other modules). We also report these evaluations in Table 1.
Question: What is he doing?
Answer: horseback riding
To do so, we modify the full API prompt pro- vided to ViperGPT. For "only BLIP-2", we delete all modules and functions in the prompt besides ImagePatch. eeseeattchsiimpl.. simple_quers As the prompt for this module included in-context demonstrations lying on other (now removed) modules, we had to re- write these demonstrations. We either rewrite the existing problem ("zero-shot") or rewrite three ran- dom training set examples for each dataset ("few- shot"). For "without BLIP-2", we simply delete ImagePatch. ShtgeePacc.simpee.oooo.. simple_query and all references to it from the prompt. We show examples for both procedures in Appendix C.
Candidate: riding a horse
Is the candidate correct? [yes/no]
Kamalloo et al. (2023) demonstrates a substan- tial increase of +0.52 in Kendall's 7 correlation with human judgements using their introduced met- ric instead of exact match on the NQ-Open bench- mark (Lee et al., 2019).
3.3 Metries: Multiple Choice
We evaluate the multiple choice setting for A- OKVQA and ScienceQA. In this setting, a method will similarly be given an image and question, but also list of textual choices. The method is required to select one of those choices its predicted an- swer. We evaluate this setting using the standard accuracy metric.
Because ViperGPT has no other image-to-text modules, we expect that excluding BLIP-2 (i.e. "without BLIP-2") should have a highly detrimen- tal effect on the VQA performance of ViperGPT. However, we instead observe that the variant retains 84% and 87% of the average performance, respec- tively, for the direct answer and multiple choice benchmarks. This indicates that including many modules improves the robustness of the ViperGPT model, in that ViperGPT is able to compensate by using other modules to replace BLIP-2.
4 Selecting Modules in ViperGPT
In Surís et al. (2023), the choice of modules is different for each task. This contrasts with end-to- end models like BLIP-2 which are purported to be cask-agnostic. To draw more direct compari- son, we evaluate ViperGPT's performance when given the full API (Surís et al., 2023, Appendix B) and set of all corresponding modules. We refer this the "task-agnostic" setting (Table 1). We find that, in this case, the gain of ViperGPT over
We find that using Viper with BLIP-2 as the only module (i.c. "only BLIP-2") also retains significant performance in the direct answer setting: i.e. by 95% average. Moreover, this variant actually gains performance (+6% A-OKVQA and +12% on ScienceQA) in the multiple choice setting. This
'A list of or-separated answers is provided when several ground truth annotations are available. Although (Kamalloo et al., 2023) tests if the generated response starts with "yes" or "no", we instead compare the log likelihood of generating "yes" or "no" for additional robustness.
'In this comparison, we use our own replicated result for the performance of BLIP-2 on GQA and OK- OK-VQA for consis- tency, although they may deviate from prior reports (li et al., 2023) by 2-5%.

---

Direct Answer
Multiple Choice
Method
VQAv2
GQA
OK-VQA
A-OKVQA
A-OKVQA
ScienceQA
BLIP-2
60.8
64.8
41.9
51.1
40.8
59.5
31.8
63.5
26.2
37.0
ViperGPT (lask-agnostic)
62.1
63.9
44.0
52.6
37.2
57.1
39.5
61.7
52.9
36.0
-without BLIP-2
45.0
55.0
31.8
46.9
7.7
49.6
4.9
44.9
43.1
33.2
-only BLIP-2 (zcro-shot)
59.3
61.0
34.7
45.5
36.0
53.0
39.3
57.8
58.9
47.7
only BLIP-2 (few-shot)
62.9
64.4
39.0
47.8
36.4
53.8
29.9
47.2
56.6
46.6
BLIP-2 (Li ul., 2023)
44.7
45.9
ViperGPT (Surís et al., 2023)
48.1
51.9
Table I: Our evaluation of BLIP-2 and our ViperGPT variants across VQA benchmarks. For each direct answer entry, we list both existing metrics (left) and the InstructGPT-eval metric (right) described in Sec. 3.2. As explained in Appendix A, we only include existing metrics for posterity and make comparisons in our text using the InstructGPT-eval metric. We run BLIP-2 using the same inference settings as Surís et al. (2023), which differ slightly from Li et al. (2023).
result seems to indicate that the BLIP-2 module is doing most of the heavy-lifting within ViperGPT for the VQA benchmarks.
to ViperGPT on GQA, OK OK- and A-OKVQA in the direct answer setting, and is noticably better (i.e. +4% and +17%) on the multiple choice setting for A-OKVQA and ScienceQA. To the best of our knowledge, our method actually presents the high- est A-OK multiple-choice score compared to any other training- ffee method.
5 Decomposing Problems with Programs Prompting?
One of ViperGPT's major contributions is that it decomposes problems into Python programs: it in- herently gains the compositionality and logical rea- soning that built into programming languages. However, recent work in NLP suggests that ques- tions can also be iteratively decomposed and solved more effectively than end-to-end approaches us- ing step-by-step natural langauge prompting (Press et al., 2022). Here, we measure the gains related to ViperGPT's choice of building logical, executable programs in Python, rather than by using the inter- face of natural language and reasoning implicitly within LLMs.
Our method presents intermediate results that are in the form of natural language expressions and strictly subject to downstream operations by neural networks. On the other hand, ViperGPT can present Pythonic data types, like lists and numbers, as well as image regions. Unlike our prompting method, ViperGPT does result a more diverse set of intermediate reprexentations, some of which can be symbolically manipulated, and is designed to leverage diverse set of neural networks.
But from this experiment, we determine that it is not strictly necessary to decompose problems us- ing programs in order to realize performance gains. Instead, natural language prompting can offer a simpler alternative. While ViperGPT leverages the intrinsic compositonality and logical execution of programming languages, our method uses condi- tional generation on intermediate results and a flex- ible natural language interface for reasoning, while remaining similarly effective.
We want to enable as direct a comparison possible between natural language prompt- ing with our method and program generation with Viper. Thus, we choose the same VLM (BLIP-2) ViperGPT and an analogous LLM to Codex (code-davinci--02) speciiiiall InstructGPT (text-davinci-ent) 002). We present the results of our method ("Successive Prompting") in Table and Fig. and directly compare against the "only BLIP-2" variants of ViperGPT. We have also used the same in- mo-context examples for each dataset for both ViperGPT ("only BLIP-2, few-shot") and Successive Prompting, which helps keep the com- parison more fair.
use
In Appendix G, we tried to identify patterns in questions to determine whether they more suitable for formal natural language-based de- composition. We could not find any clear patterns, following simple question type breakdowns of the original datasets, but are hopeful that future work will explore this further and reveal better insights.
Our prompting method performs comparably (i.e. retaining 92% of the performance on average)

---

Direct Answer
Multiple Choice
Method
VQAv2
GQA
OK-VQA
A-OKVQA
A-OKVQA ScienceQA
ViperGPT (only BLIP-2)
62.9
64.4
39.0
47.8
36.4
53.8
39.3
57.8
58.9
47.7
Successive Prompting
53.9
57.8
37.1
47.7
28.5
45.1
36.3
55.2
63.0
64.6
Table 2:
We evaluate prompting based decomposition strategy ("Successive Prompting") from Sec. and compare against the analogous ViperGPT variant ("only BLIP-2") from Sec. 4 and Table I: both use InstructGPT for decomposition
keep BLIP-2 as the only module. We list the higher score between the zero-shot and few-shot "only BLIP-2" variants here. For each direct answer entry, we list both existing metries (left) and the InstructGPT-eval metric (right) described in Sec. 3.2.
Direct Answer OK-VQA
Multiple Choice
VQA
GQA
A-OKVQA
A-OKVQA
ScienceQA
No Exception 99%
98%
99%
96%
86%
79%
Parsing 0%
0%
0%
0%
1%
3%
Runtime
1%
2%
1%
4%
12%
18%
Table breakdown of failure rates across exception modes for ViperGPT (lask-agnostic) across benchmarks. "No Exception" only indicates completion, not correctness, of executions. Parsing errors occur as SyntaxError in Python. We further breakdown runtime errors in Appendix F.
6 How well does ViperGPT generalize to out-of-distribution tasks?
to the other benchmarks in the direct answer setting, We find that this rate increases by another 3x (i.e. 12%) for A-OKVQA in the multiple choice setting. And, ScienceQA holds the highest rate of runtime failures overall 18%. In the multiple choice set- ting, A-OKVQA and ScienceQA produce code that cannot be parsed (i.e. with syntax errors) 1% and 3% of the time. On the other hand, the rate of pars- ing exceptions is consistently 0% for benchmarks (including A-OKVQA) in the direct answer set- ting.
As we have observed Sec. 4, ViperGPT has been designed around a specific set of tasks (including GQA and OK-VQA), especially in its selection of modules and prompt. On the other hand, core motivation for modular and neuro-symbolic ap- proaches is that these should have better gener- alization capabilities unseen tasks. So, we fur- ther wonder how robust ViperGPT is to out-of- distribution tasks. In particular, we consider A- OKVQA and (cspecially) ScienceQA as out-of- distribution (compared to GQA and OK-VQA).
Related Work
Visual Question Answering. Visual Question Answering common vision-language task with many variants: in our paper, we benchmark sev- eral mainstream VQA datasets (Goyal et al., 2017; Hudson and Manning, 2019; Marino et al., 2019; Schwenk et al., 2022; Lu et al., 2022), requiring broad range of skills: related to computer vision and perception, compositional understanding, out- side and world knowledge, scientific and common- sense reasoning, and more.
First, investigate changes the prompt of ViperGPT. Will adding task-specific in-context ex- amples improve the model's robustness to new tasks? In Table 1, we compare zero-shot and few- shot variants of "ViperGPT (only BLIP-2)". We can see that including few-shot examples consis- tently improves performance on the "in-domain" tasks (VQAv2, GQA, and OK- OK-VQA) by +2% on average. But, this consistently hurts performance on the fouttof-distributtio" tasks (A-OKVQA and ScienceQA) by 11% on A-OKVQA's direct answer setting and 2% on average for their multiple choice settings.
End-to-end models are the predominant approach in deep Icarning. In particu- lar, consider vasion-l models here. We observe that recent, state-of-art vlngon-language models may rely on large-scale pretraining (Rad- ford et al., 2021; Singh et al., 2022; Yu et al., 2022; Wang et al., 2023; Li et al., 2023), be trained on
End-to-end models.
We also look at the correctness of the programs generated by ViperGPT in Table 3. We find that the generated code (on average) 3x as likely to encounter runtime errors for A-OKVQA compared

---

VQAv2
GQA
OK-VQA
A-OKVQA
What's around the window?
Q: Has the food this wornan is preparing been fried?
Q: Which part of this arimal would be in use if was pleying the game would be played with
Q: The company that produced the device her hand from what country?
ViperGPT (only BLIP-2, few-shot)
Generated Code
Generated Code
Generated Code
Generated Code
return simple query
location simple query
animal simple query "What animal is this?"
dovice_ simple.quary "Whet type the holding?
"Hes food this woman proparing been Maad?
"Where the window/"
Artifacts: None
simple query [ "What is around the window that is (ocation)?"
simple. query "Which part of this (animal) would be in use if was playing the game that would be played with the items the man is holding? ?
company _name = simple, query
"Wher company makes (spollo__ypa)s"
return simple, quurr
Artifacts:
"Wher country (campany_nama)
location = "aovee the bed"
Artifacts:
animal "dog"
Artifacts:
device_ sype-yypee-ee_ = "remote control"
company_nand name "Nintendo"
Prediction: "no"
Prediction:
"curtains"
Prediction: "tail"
Prediction: "Japan"
Successive Prompting
Q: What's in the image?
Q: What's in the image?
Q: What's the image?
Q: What's in the image?
A: person is preparing salad on a counter
A: small bedroom
A: woman sitting on couch playing a video game
A: carrying dog Q: What the doing with the dog?
Q: What is around the window in the small bedroom?
Q: Has the lettuce been fried?
Q: What company produces the device in her hand?
A: playing frisbee
A:
A: curtains
A: Nintendo
Prediction: "no"
Prediction: "curtains"
Pred: "the dog's mouth"
Prediction: "Japan"
Figure 2: Examples of decompositions for our "ViperGPT (only BLIP, few-shot)" and "Successive Prompting"
models on our direct answer benchmarks. These examples have been condensed for readability. In Successive Prompting, follow-up questions (Q) are proposed by the LLM (InstructGPT) and answered (A) by BLIP-2. After variable number of follow-ups, the L.LM will provide a prediction to the original question and terminate.
many tasks with a unified architecture (Chen et al., 2021b; Wang al., 2022b; Lu al., 2023), both (Wang et al., 2022a; Alayrac al., 2022; Chen et al., 2022). this paper, we focus on the first cat- egory and find BLIP-2 (Li et al., 2023), which uti- lizes a frozen image encoder and language model, as suitable and effective representative.
2023; Surís al., 2023; Subramanian et al., 2023). Simultaneously, these approaches adopt SOTA neu- ral networks their set of modules. All together, these methods present the first competitive neuro- symbolic solutions for visual reasoning tasks.
Natural language reasoning and tool-use. Re- cently, has been found in NLP that large language models can more effectively perform reasoning tasks by reasoning step-by-step (Wei et al., 2022; Kojima et al., 2022). A few works have extended similar capability in LLMs for complex prob- lem and question decomposition (Press et al., 2022; Zhou et al., 2023; Dua et al., 2022; Khot et al., 2023). Finally, recent works have learned ways
Neuro-symbolic methods. Several prior meth-
ods have attempted neuro-symbolic approaches for visual reasoning tasks (Andreas et al., 2015; John- son et al., 2017; Hu et al., 2017; Yi et al., 2018). However, until now, these methods were learned by training and found middling success in doing so. More recently, a few training-free approaches have been suggested that leverage the powerful in-context and program generation abilities of mod- emn large language models (Gupta and Kembhavi,
prompt language models call tools (Parisi et al., 2022; Schick et al., 2023). All of these emergent research directions jointly enable the prompting

---

approach we present in Sec. 5,
services at the Center for Computation and Visual- ization, Brown University.
8 Conclusion
In this paper, we have analyzed ViperGPT (Surís et al., 2023), a recent and intricate modular approach for vasion-language tasks. We unbox ViperGPT, asking the rescarch question: where does its performance come from? Observing that one of ViperGPT's five modules (i.e. BLIP-2 (Li al., 2023)) often outperforms or constitutes ma- jority of its own performance, we investigate this module's role further. Through our experiments, we find that while ViperGPT's marginal gains over BLIP-2 are direct result of its task-specific mod- ule selection, its modularity improves its overall robustness and it can perform well even without the (scemingly critical) BLIP-2 module. Additionally, we investigate ViperGPT's choice of generating Python programs. We ask this is necessary and, alternatively, propose a method relying on prompt- ing large language models and vision-language models to instead decompose visual questions with natural language. Our method performs compara- bly to the relevant ViperGPT variants and, to the best of our knowledge, even reports the highest multiple choice accuracy on A-OKVQA (Schwenk et al., 2022) by any training-free method to date.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 01123716-23733
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2015. Neural module networks. 2016 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39.48.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan. Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman. Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Moham- mad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, David W. Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alee Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Πlya Sutskever, and Wojciech Zaremba. 2021a. Evaluat- ing large language models trained on code. ArXiv. abs/2107.033 033744.
Limitations
Although the experiments in our paper are self- contained and designed to be directly comparable with each other, the absolute scores we report differ from other reports. For example, our results for BLIP-2 on GQA and OK-VQA are within 2-5% of the original reports. We attribute such differences to possible differences in model inference settings between (Surís et al., 2023)-whil we follow and e.g. runs the model with 8-bit inference- and (Li et al., 2023).
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton, 202 Pix2seq: A language mod- eling framework for object detection. arXiv preprint arXiv:2109.10852
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, al. 2022. Pali: A jointly-scaled mul- tilingual language-image model. arXiv preprint 20X.v:2209.06794
In our "ViperGPT (only BLIP-2)" experiments, we find that the method often calls BLIP-2 directly in a single step and might not offer any mechanisms beyond BLIP-2 in that case.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu. Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai. Hongkun Yu, Slav Petrov, Ed Huai hsin Chi. Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Seal- ing instructionffinttuee language models. ArXiv. abs/2210.1 1416.
Acknowledgments
We would like to thank Michael A. Lepori for his generous feedback this work. This work is par- tially supported by the Samsung Advanced Institute of Technology and a Brown University Presidential Fellowship for Apoorv Khandelwal. Our research was conducted using computational resources and

---

Tim Dettmers, Mike Lewis. Younes Belkada, and Luke Zettlemoyer. 2022. Llm.in 8-bit matrix multipli- cation for transformers at scale. In The 36th Con- ference on Neural Information Processing Systems (NeurlPS), pages 30318-30333
approach for solving complex tasks. In The Eleventh International Conference on Learnnng Representa- tions (ICLR).
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In The 36th Conference on Neural Information Processing Sys- tems (NeurIPS), pages 22199-222113
Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive prompting for decom- posing complex questions. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 1251-1265, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, pages 6086-6098 Florence, Italy. Association for Computational Linguisties.
Yuxin Fang. Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang. Xinlong Wang. and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition (CVPR). pages
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In Proceedings of the 40th Interna- tional Conference on Machine Learning (ICML).
Yash Goyal. Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding visual question answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recugnition (CVPR), pages
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. 2022. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition (CVPR), pages 10965-10975.
Tanmay Gupta and Aniruddha Kembhavi. 2023. Vi- sual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 14953-14962.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2023. Unified- io: A unified model for vision, language, and multi- modal tasks. In The Eleventh International Confer- ence on Learning Representations (ICLR).
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804-813.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neu- ral Information Processing Systems (NeurlPS), pages 2507-2521.
Drew A. Hudson and Christopher D. Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pages 6693- 6702.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual ques- tion answering benchmark requiring external knowl- edge, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3190-3199.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zit- nick, and Ross B. Girshick. 2017. Inferring and exe- cuting programs for visual reasoning. 2017 IEEE In- lernational Conference on Computer Vision (ICCV), pages 3008-3017.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. In The 36th Confer- ence on Neural Information Processing Systems (NeurIPS), pages 27730-27744.
Ehsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, and Davood Rafiei. 2023. Evaluating open-domain question answering in the era of large language mod- els. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (ACL).
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal, 2023. Decomposed prompting: A modular
Aaron Parisi, Yao Zhao, and Noah Fiedel. 2022. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255

---

Ofrr Press, Muru Zhang. Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. ArXiv, abs/2210.03350.
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng. Qiang Liu, Kriti Aggarwal. Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2023. Image as a foreign language: Beit pretraining for vision and vision- lision-language tasks. In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 19175-1999 19186.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- tr. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Con- ference on Machine Learning (ICML), pages 8748- 8763. PMLR.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F, Xia, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elic- its reasoning in large language models. In The 36th Conference on Neural Information Processing Sys- tems (NeurIPS), pages 24824-24837.
René Ranfil, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020. Towards ro- bust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transac- tions on Pattern Analysis and Machine Intelligence (TPAMI), 44(3):1623 163336
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba. Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances neural information processing systems, 31.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint 230.:22302.077..1.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Ye- ung. Mojtaba Seyedhosseini. and Yonghui Wu. 2022, Coca: Contrastive captioners are image-text foun- dation models. Transactions on Machine Learning Research (TMLR).
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answer- ing using world knowledge. In Proceedings of the European Conference Computer Vision (ECCV), pages 146-162.
Yan Zeng. Xinsong Zhang, and Hang Li. 2022. Multi- grained vision language pre-training, Aligning texts with visual concepts. In Proceedings of the 39th International Conference Machine Learning (ICMI.), pages 25994-26009.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami. Guillaume Couairon, Wojciech Galuba, Mareus Rohrbach, and Douwe Kiela. 2022. Flava: A foun- dational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition (CVPR), pages 15638-15650.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Olivier Bousquet, Quec Le, and Ed Chi. 2023. Least- to-most prompting enables complex reasoning large language models. In The Eleventh International Conference on Learning Representations (ICLR).
Sanjay Subramanian, Medhini G. Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani. Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein. 2023. Modular visual question answering via code generation. In Proceedings of the 6l Annual Meet- ing of the Association for Computational Linguistics (ACL).
Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. ArXiv, abs/2303.08
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Li- juan Wang. 2022a. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research (TMLR).
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b. Ofa: Unifying ar- chitectures, tasks, and modalities through a simple sequence-to-sqquence learning framework. In Inter- national Conference on Muchine Learning (ICML), PMLR.
pages

---

A Existing Metries vs. InstructGPT-eval
C
ViperGPT Variants
We observe that existing VQA metrics correlate poorly with (our) human judgments for evaluat- ing model predictions. We analyze predictions for our model types (i.e. "BLIP-2", "ViperGPT (task- agnostic)), and "Successive" in Tables and 2) on random training split subsets (N 50) of VQAv2, GQA, and A-OKVQA. We specifically find that, when the existing and new (InstruciGPT-evall) eval- uation metries disagree, InstructGPT-eval correct 93% of the time. In Fig. 3, we show two exam- ples of such disagreements between existing VQA metrics and the open-ended metric. Therefore, we include the results of existing metrics in paper for posterity, but do not find these reliable (espe- cially for open-ended text generated by models like InstructGPT). We instead make comparisons in our paper using the InstructGPT-eval metric. We ob- serve that trends (i.e. which model performs better) are usually the same for both metrics, but the actual gaps may differ significantly.
>>> Which kind of animal is not eating?
>>> def neecute_command
-> str:
>>> image__pacce = ImagePatch
>>> animal_patches = image_p. ("nimall))
>>> for animal_patch patch in primal_patchess
>>>
if not animal_patch. "animal", "eating"
):
>>>
return animal_patch. soochh.smppl...m simple_query" "What kind of animal is eating?"
# crop would include eating so keep it in the query
>>>
# If no animal is not eating,
query the image directly return image_patch, simple_queryC "Which kind of animal is not eating?"
>>>
>>> Which kind animal is not eating?
>>> def execute_comman peecutt_commann(mmage
str:
>>> image_patch ImagePatch(image)
>>> animals image_patch. simple_querrrreeeeeeeeeeepepeeee
which animals are in the image?
>>> return mmage_patch... simple_ querr("
Which of the (animals) is not eating?
B ViperGPT Design Choices
We make few modifications (listed in Sec. 2.2) to the ViperGPT method (from the original de- sign (Surís et al., 2023)) to improve conformity for the VQA task and ensure fairness when com- paring to our prompting-base approach in Sec. 2.3. We elaborate further here.
Figure 4: Partial prompt for ViperGPT (only BLIP-2, zero-shot), showing how the existing demonstration is re-written with only the simple_ query module.
>>> # Are there both windows and doors this photograph?
As we always expect the executable program to return string for VQA, we explicitly add "-> str" the function signature the prompt. By design, our prompting-buis based approach can similarly only result a string.
>>> def execute_ (imaee str:
>>>
image_pated ImagePatch(image windows_present image_p. simple_
>>>
"Are there windows in this image?"
>>> doors_present = image_p. simpleee simple_query(
For the multiple choice setting, we provide an ex- plicit list of choices in the code-generation prompt (e.g. "# possible answers l'dog', 'cat', 'foo', 'bar'I" after the question prompt). We do this, the code generation model benefits from awareness of these choices when generating the program. Simi- larly, our prompting-base method provided with a list of choices conjunction with the question, prior to proposing follow-up questions.
"Are there doors in this image?"
>>> if windows_present == "yes"
and doors_present = "yes":
>>> return "yes"
>>>
else:
>>> return "no"
Figure 5: Partial prompt ViperGPT (only BLIP-2, few-shot), showing one in-context demonstration writ- ten for GQA.
Unlike the Successive Prompting method, the ViperGPT program is not guaranteed to produce a result that matches one of the multiple choices. So we map this result to the most similar choice using InstructGPT (text-davinci-003) We make this choice because text-davinci by a module in ViperGPT and for the open- open-ended evaluation metric.
D Datasets
VQAv2: This is classic VQA benchmark. Many of the questions involve tasks (like clas- sification, attribute detection, and counting) and require primitive computer vision skills.
already used

---

Question (VQAv2): What does this sign mean?
Answers: twisty narrow road, narrow road (x4), curves/narrow road ahead,
Prediction (BLIP-2): warning of a curve ahead Actual: Correct
Existing metric: "Incorrect"
InstructGPT-eval metric: "Correct"
Question (A-OKVOA): Why is he bent over? pitching stance, follow through (x2), pitching (x4), throwing ball,
Answers:
Prediction (BLIP-2): throw the ball
Actual: Correct
Existing: "Incorrect"
InstructGPT-eval: "Correct"
Figure 3: Two examples disagreements between existing VQA metrics and InstructGPT-eval from Kamalloo et al. (2023). In all, InstructGPT--eval correctly evaluates the predictions. As mention in Sec. 3.2, find evaluations from InstructGPT-eval more accurate in an overwhelming majority of cases.
Direct Answer
Multiple Choice
VQA
GQA
OK-VQA
A-OKVQA
A-OKVQA ScienceQA
NameError
100%
14%
25%
13%
61 %
34%
ErrributeError
0%
42%
25%
63%
22%
36%
IndexError
0%
13%
13%
6%
6%
21 %
TypeError
0%
15%
19%
19%
7%
4%
IndentationError
0%
0%
0%
0%
0%
2%
ValuelError
0%
3%
6%
0%
3%
1%
KeyError
0%%
0%%
0%
0%
0%
1%
ZeroDivisionErrrr
0%
0%
0%
0%
0%
0%
Other
0%
14%
13%
0%
0%
0%
Table 4: Breakdown of failure rates across runtime exceptions for ViperGPT (task (task-agnostic) across our benchmarks.
For example, "What color are the pants?" might entail (1) detecting the pants in the im- age and (2) determining their color.
about many categories of objects. Usually requires detecting an object and asking for knowledge about that object. Example (Surís et al., 2023, Figure 5): "The real live version of this toy does what in the winter?". Involves locating and identifying the toy, then asking about the rest.
2. GQA: This is benchmark that focuses compositional questions. Requires "array of reasoning skills such as object and attribute recognition, transitive relation tracking, spa- tial reasoning, logical inference and compar- isons" (Hudson and Manning, 2019). For ex- ample, "What color are the cups to the left of the tray top of the table?" is a multi-step composition (focusing spatial relationships and attribute recognition).
4. A-OKVQA: A follow-up benchmark to OK- VQA. Instead of asking for closed-domain knowledge about objects, this features "open- domain" questions that might also require some kind of commonsense, visual, or physi- cal reasoning. For example, "Which position will the red jacket most likely finish in?" in-
3. OK-VQA: Requires "outside knowledge"

---

volves (1) identifying the context (a ski race),
G Failure Rates By Question Type
(2) locating all the racers, (3) identifying the racer who is wearing the red jacket, (4) deter- mining the orientation of the race (e.g. left-to- right), and (5) determining the "index" of the red jacket racer among all racers along this ori- entation. The question's 's textual prior appears contextually insufficient and proposing a pro- gram based this alone (as in ViperGPT) could be fragile and quite difficult.
BLIP-2
ViperGPT
Successive
GQA (15)
AhtoseAttr
62%
67%
chooseRel
65%
72%
60%
verifyObj
72%
80%
AutryAttr
43%
50%
40%
verifyAurr
58%
67%
59%
logicalOhj
62%
68%
61% %
logicalAtttr
59%
64%
57%
5. ScienceQA: This benchmark features scien- tific questions (of clementary through high school difficulty) that require both back- ground knowledge and multiple steps of rea- soning to solve. Their example question is "Which type of force from the baby's s hand opens the cabinet door?" (choices: push, pull). The given reasoning is that (paraphrased) "The direction of push is away from and pull is towards the acting object. The baby's 's hand applies force to the cabinet door that causes the door open. The direction of the door opening is towards the baby, so the force is pull." Without seeing the image, is not appar-
queryGlobal
29%
34%
29%
verifyRel
58%
61%
56%
queryCat
50%
46%
42%
queryRel
42%
39%
39%
compareAttr
56%
56%
59%
chooseCat
84%
53%
66%
OK-VQA (11)
(Weather)
65%
62%
45%
6 (Sucial Sci.)
49%
55%
39%
(Food)
63%
58%
43%
Other
63%
58%
45%
7 (People)
63%
60%
46%
8 (Plants)
57%
57%
44%
2 (Brands)
53%
57%
44%
4 (Sports)
63%
62%
52%
9 (Science)
54%
52%
43%
ent, but what determines whether the baby is opening or closing the door is the fine-grained detail that the baby's 's hand is curled over the top of the cabinet door, not grasping the han- dle or pushing the door's surface. The current stage of visual programming models are capable of such difficult multi-step reasoning chains and planning around such fine details. Instead, we find that ViperGPT tends to de- fault to its end-to-end VQA module instead.
(Objects)
60%
56%
47%
(Vehicles)
54%
51%
43%
ScienceQA (15)
us-history
33%
47%
43%
earth-science
51%
56%
62%
biology
40%
56%
chemistry
42%
25%
47%
economics
53%
3%
27%
physies
43%
34%
58%
ecience-practicee
46%
21%
51%
geography
23%
27%
75%
Log likelihood of generating continuations
Table 5: We breakdown failure rates for our three model families by question types (as specified in the original GQA, OK-VQA, and ScienceQA datasets). We remove outliers (i.e. categories with 50 samples) and sort |ViperGPT Successive].
We weighted byte-length normalization for generating the log likelihood of continuation, i.e.
H Successive Prompting Example
mi (n 1) Σ
Long (n-1
log ((z))
Zk=m
Law
Question: Has the food this woman is preparing been fried?
where x is a list of tokens (with m tokens in the prompt and tokens in the continuation) and the byte-length of the token H; is Lu.-
Follow-up: What's s in the image?
Follow-up answer: a person is preparing salad on the counter
Follow-up: Has the lettuce been fried?
F Runtime Failure Rate of ViperGPT
Follow-up answer: no Answer to the original question: no
As an extension to Table 3, we further breakdown ViperGPT failures for runtime errors in Table 4.


Reviews:
[{"review_id": "Fg6O3vMwsd", "reviewer": "Reviewer_9Ggp", "summary": "", "questions": "Error analysis of ViperGPT. It would be interesting to see where the errors come from. For example, the program might be not runnable; the program might be unreasonable; or one component is giving a wrong answer.  Maybe investigate 100 examples and count how many errors each category has.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and consideration! We really value your feedback and agree that these changes will greatly strengthen the paper. Please see how we propose to address your points below.\n\n> The paper would be more interesting if it provides some analysis of the differences between the tasks, and how they influence the gain of visual programming compared with BLIP-2.\n\nWe would be happy to elaborate further about each of our benchmark tasks in revision.\n\nWe’ve measured the failure rates of BLIP-2 / ViperGPT across each benchmark’s question types (sorted by ViperGPT’s improvement over BLIP-2). Please refer to the **Breakdown by Question Type** section below for the details. For space considerations, we have shared the bottom/top 3 entries there (and will include the complete results in revision, including the breakdown for our Prompting method as well as associated qualitative/visual examples).\n\nGenerally, we find that ViperGPT performs better than BLIP-2 at visual recognition / counting tasks due to its vision modules, but worse on certain (module-dependent) skills (e.g. diagram understanding or text recognition) and when generating programs for questions that require complex reasoning or are themselves image-dependent.\n\n> ViperGPT may have a bigger performance gain on counting tasks like TallyQA.\n\nCounting is actually one category of VQAv2 and corresponds with the following row: as we expect, Viper does improve compared to BLIP-2.\n\n|          | BLIP-2 | Viper |\n|----------|--------|-------|\n| how many |   0.36 |  0.47 |\n\n> There could be more analysis on when ViperGPT makes errors.\n\nYou are indeed correct that program-based approaches can be fragile and fail upon parsing or runtime. This is a great point in favor of program-free methods, such as prompting with natural language. We have measured the error-rate along your proposed breakdown over all of ViperGPT’s mistakes: please refer to the **Breakdown for ViperGPT Errors** section below for the details. We will also include qualitative examples in our revision.\n\nWe actually find that the ViperGPT method encounters few parsing and runtime errors for benchmarks that it was designed for (i.e. GQA, OK-VQA — and similarly, VQAv2 and A-OKVQA). We observe a significant increase in the rates of these errors for the A-OKVQA MC subset (which contains additional, ambiguously worded questions that require choices to contextualize correctly) and very different benchmarks, like ScienceQA. Such fragility is detrimental to the overall performance in these out-of-domain tasks, leaving up to 15-20% accuracy on the table.\n\n# Additional details (if interested)\n\n---\n---\n\n## Breakdown by Question Type\n\nYou may also find further details about these question types in each benchmark’s original papers (i.e. VQAv2: Figure 4, GQA: Appendix Table 2, OK-VQA: Figure 4, and ScienceQA: Figure 4). Note: A-OKVQA is categorized by reasoning type (knowledge-based, commonsense, visual, physical) but provides no such annotations in its released dataset.\n\n\n### VQAv2\n\nWe can see that ViperGPT performs better on questions such as “are there” (involving object detection and counting) as compared to  “do” or “why” (which require causal reasoning).\n\n|               | BLIP-2 | Viper    |\n|-----------|--------|----------|\n| why       | 0.50   | 0.25     |\n| do        | 0.75   | 0.50     |\n| what are  | 0.83   | 0.58     |\n| ...       |        |          |\n| do you    | 0.80   | 1.00     |\n| who is    | 0.40   | 0.60     |\n| are there | 0.61   | 0.83     |\n\n\n### GQA\n\nViperGPT performs better on “queryAttr” (i.e. What color is the apple?) and “exist” (i.e. Is there an apple?) than “verifyGlobal” (i.e. Is it cloudy today?).\n\n|                    | BLIP-2 | Viper    |\n|--------------|--------|----------|\n| chooseObject |   0.88 |     0.56 |\n| verifyGlobal |   0.84 |     0.69 |\n| chooseRel    |   0.75 |     0.60 |\n| …            |        |          |\n| exist        |   0.71 |     0.79 |\n| queryAttr    |   0.40 |     0.49 |\n| queryGlobal  |   0.19 |     0.31 |\n\n\n### OK-VQA\n\nQuestions are generally easily decomposable, regardless of the given question types. However, “Brands, Companies & Products” typically require reading logos or text (to identify the brand/product), which certain modules may struggle with. Unlike the end-to-end model, the decomposition approach is very strongly dependent on detecting this attribute correctly.\n\n|                                                           | BLIP-2 | Viper |\n|------------------------------------------|--------|-------|\n| Brands, Companies & Products             |   0.70 |  0.59 |\n| Weather & Climate                        |   0.78 |  0.67 |\n| Vehicles & Transportation                |   0.57 |  0.49 |\n| …                                        |        |       |\n| Plants & Animals                         |   0.55 |  0.54 |\n| Sports & Recreation                      |   0.58 |  0.59 |\n| Geography, History, Language and Culture |   0.35 |  0.54 |\n\n\n### ScienceQA\n\nViperGPT is best at the “word-study” topic (i.e. “Which term matches the picture?” for a given list of choices). This can be very directly written as a 1-step program that uses Viper’s text-image similarity module between the image and choices. On the other hand, the “economics” task depends on whether two individuals can “trade” items based on parsing a diagram with either natural images or text (to determine whether the items are available). Such diagram-based questions are probably well outside the domain of questions that ViperGPT was designed for and generating a task-specific program would be very difficult without further inductive biases.\n\n|                                | BLIP-2 | Viper |\n|-----------------------|--------|-------|\n| civics                |   1.00 |  0.00 |\n| economics             |   0.53 |  0.03 |\n| literacy-in-science   |   0.50 |  0.17 |\n| …                     |        |       |\n| vocabulary            |   0.20 |  0.40 |\n| reading-comprehension |   0.25 |  0.63 |\n| word-study            |   0.20 |  0.60 |\n\n---\n---\n\n## Breakdown for ViperGPT Errors\n\n|                       | VQAv2 | GQA | OK-VQA | A-OKVQA | A-OKVQA (MC) | ScienceQA (MC) |\n|-----------------------|-------|-----|--------|---------|--------------|----------------|\n| Total Incorrect       |   361 | 599 |    439 |     412 |          540 |           1283 |\n| No Exception          |   99% | 99% |    99% |     96% |          86% |            79% |\n| Parsing (SyntaxError) |    0% |  0% |     0% |      0% |           1% |             3% |\n| Runtime               |    1% |  1% |     1% |      4% |          12% |            18% |\n\nWe can specifically breakdown the runtime exceptions as well:\n\n|                  | VQAv2 | GQA | OK-VQA | A-OKVQA | A-OKVQA (MC) | ScienceQA (MC) |\n|------------------|-------|-----|--------|---------|--------------|----------------|\n| NameError        |  100% |  0% |    20% |     13% |          61% |            34% |\n| AttributeError   |    0% | 33% |    40% |     63% |          22% |            36% |\n| IndexError       |    0% |  0% |     0% |      6% |           6% |            21% |\n| TypeError        |    0% | 67% |    40% |     19% |           7% |             4% |\n| IndentationError |    0% |  0% |     0% |      0% |           0% |             2% |\n| ValueError       |    0% |  0% |     0% |      0% |           3% |             1% |\n| KeyError         |    0% |  0% |     0% |      0% |           0% |             1% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides great insight into the recent visual programming framework. This area has been really popular, especially after Visprog won CVPR best paper. However, there lack of work that analyzes where is the performance gain coming from.\n\nThe discoveries in this paper are inspiring and provoke more thinking. It seems that ViperGPT is just a shallow wrapper of BLIP-2. The ablation studies are complete and interesting. The NLP community will be benefited from the message this paper shows.", "reasons_to_reject": "1. The performance gain of visual programming might be really dependent on the task. For example, ViperGPT may have a bigger performance gain on counting tasks like TallyQA. The paper would be more interesting if it provides some analysis of the differences between the tasks, and how they influence the gain of visual programming compared with BLIP-2.\n\n2. There could be more analysis on when ViperGPT makes errors. Neuro-symbolic approaches are typically more fragile. For example, the program maybe simply not runnable and has execution errors. It would be great if more error analysis is made in this paper.", "questions_to_authors": "Error analysis of ViperGPT. It would be interesting to see where the errors come from. For example, the program might be not runnable; the program might be unreasonable; or one component is giving a wrong answer.  Maybe investigate 100 examples and count how many errors each category has.", "paper_topic_and_main_contributions": "The paper provides a thorough analysis of the recent works on visual programs, primarily focusing on ViperGPT. The work conducts experiments across 5 VQA datasets and compares end-to-end and visual programming approaches. The main discoveries are:\n1. BLIP-2 is disproportionately influential in ViperGPT.\n2. ViperGPT can get similar performance by using natural language question decomposition rather than Python programs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are minor typos (e.g. \"langauge\" on line 190, 191, 414). Some proofreading could be helpful."}, "extra_text": {}}, {"review_id": "GHKnB8GLki", "reviewer": "Reviewer_a58D", "summary": "", "questions": "The study is ViperGPT-specific. How is it translatable to other models (and even other tasks)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your efforts in reviewing our paper! We’d like to answer your questions / concerns directly:\n\n> The first part of the study is like an extension of ViperGPT’s ablation study, making it less attractive. Re-organizing to highlight the power of decomposition might be more interesting.\n\nWe would like to clarify that our paper is not an extension of ViperGPT’s ablations, rather a carefully controlled set of experiments that serve as both a reproducibility study and a new set of ablations and hypotheses. In the process, we also propose a natural language-based decomposition strategy as a similarly performant alternative to program generation.\n\nWe do appreciate your advice about the organization of our paper and would like to propose some adjustments. We will:\n - List the main findings in the Introduction.\n - Add a diagram to Section 2, illustrating the differences between end-to-end, neuro-symbolic, and prompting-based methods.\n - Include qualitative/visual results of each method with our experiments.\n - List the research questions in a new subsection 2.1.\n - Highlight the relevant research questions at the beginning of each experimental section.\n\nWe would appreciate your feedback on whether these would address your concerns (or if not, what concerns remain). We will be happy to make the agreed changes for camera-ready.\n\n> This study is ViperGPT-specific. How is it translatable to other models (and even other tasks)?\n\nIt’s true that our experiments are specific to ViperGPT and VQA-style benchmarks. We did this to prioritize depth over breadth and feature experiments most suited for ViperGPT and our proposed prompting method. We are very interested in analyzing other methods and tasks and better understanding how general these results are. This would entail significant model-specific experimental design in each case, so it is something that we cannot realistically fit into this paper, but do hope to pursue in follow-up work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The study explores the power of the neuro-symbolic approach in visual question answering and proposes an strong alternative, revealing the root of the benefit of question decomposition.\n- Experimental results are quite supportive.", "reasons_to_reject": "The manuscript's organization confuses the reader of the purpose of the research. The first part of the study is like an extension of ViperGPT's ablation study, making it less attractive.", "questions_to_authors": "The study is ViperGPT-specific. How is it translatable to other models (and even other tasks)?", "paper_topic_and_main_contributions": "First, This study analyzes ViperGPT's performance gain to explore the power of question decomposition in visual question answering tasks. Then, the authors propose an iterative prompting-based approach for visual question decomposition to achieve similar performance. The proposed method is different from the neuro-symbolic approach that, without the logical and executable program, the large language model can also decompose and tackle complex visual question answering problems.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The manuscript's organization prevents the delivery of the study's core findings. Re-organizing to highlight and focus on the power of decomposition might be more interesting to read."}, "extra_text": {}}, {"review_id": "E5tEwZrlC0", "reviewer": "Reviewer_oZRE", "summary": "", "questions": "see weakness.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for your response!\n\n> I am not sure if the applied benchmarks contain complex reasoning tasks… I humbly ask the authors to introduce more about the benchmarks and their difficulties.\n\nWe chose this set of benchmarks because they require diverse skills and are indeed challenging. We summarize them below and are happy to do so further in our paper. Of course, the benchmarks’ original papers provide even more detail (especially showing examples with images).\n\nFinally, we would like to emphasize that these benchmarks remain unsolved by current SOTA models with large pre-training and billions of parameters. One open-source example is BLIP-2, whose paper we can refer to (BLIP-2, Table 1) and which we include in our own analyses (Table 1). And, we can also refer to closed-source models, such as PaLI-17B, which reports a score (PaLI, Xi Chen et al., ICLR 2023, Table 3) that is far from saturated on OK-VQA. These latest results are a testament to the challenging nature of the benchmarks we have selected.\n\n---\n---\n\n## Summary of the benchmarks used in this paper\n\n**VQAv2:** This is a classic/standard VQA benchmark. It would be the “least challenging” among our selection. Many of the questions involve tasks (like classification, attribute detection, and counting) and require primitive computer vision skills, which could be answered in a small number of steps. For example, “What color are the pants?” might entail (1) detecting the pants in the image and (2) determining their color.\n\n**GQA:** This is a benchmark that focuses on compositional questions, which are very suitable for benchmarking modular approaches. Requires an “array of reasoning skills such as object and attribute recognition, transitive relation tracking, spatial reasoning, logical inference and comparisons” (ref. GQA paper). For example, “What color are the cups to the left of the tray on top of the table?” is a multi-step composition (focusing on spatial relationships and attribute recognition).\n\n**OK-VQA:** Requires “outside knowledge” about many categories of objects. Usually requires detecting an object and asking for knowledge about that object. Example (ViperGPT Figure 5): “The real live version of this toy does what in the winter?”. Involves locating and identifying the toy, then asking about the rest.\n\n**A-OKVQA:** A follow-up benchmark to OK-VQA. Instead of asking for closed-domain knowledge about objects, this features “open-domain” questions that might also require some kind of commonsense, visual, or physical reasoning. For example, “Which position will the red jacket most likely finish in?” involves (1) identifying the context (a ski race), (2) locating all the racers, (3) identifying the racer who is wearing the red jacket, (4) determining the orientation of the race (e.g. left-to-right), and (5) determining the “index” of the red jacket racer among all racers along this orientation. We’d like to point out that the question’s textual prior is insufficient and proposing a program based on this alone (as ViperGPT attempts to do) could be fragile and quite difficult!\n\n**ScienceQA:** This benchmark features scientific questions (of elementary through high school difficulty) that require both background knowledge and multiple steps of reasoning to solve. Their example question is “Which type of force from the baby’s hand opens the cabinet door?” (choices: push, pull). The given reasoning is that (paraphrased) “The direction of push is away from and pull is towards the acting object. The baby’s hand applies a force to the cabinet door that causes the door to open. The direction of the door opening is towards the baby, so the force is pull.” Without seeing the image, it is not apparent, but what determines whether the baby is opening or closing the door is the fine-grained detail that the baby’s hand is curled over the top of the cabinet door, not grasping the handle or pushing the door’s surface. The current stage of visual programming models are not capable of such difficult multi-step reasoning chains and planning around such fine details. Instead, we find that ViperGPT tends to default to its end-to-end VQA module instead.\n\nNote: GQA and OK-VQA were benchmarked in the original ViperGPT paper (please see ViperGPT Figures 4-5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A number of visual language benchmarks are used an the results support the main claim of the paper. Ablating ViperGPT and find out the main contributor of the improved performance is also valuable.", "reasons_to_reject": "My main concern is that in the area of language model, it is widely aware that simple QA tasks does not need complicated reasoning skills, and some simple benchmarks cannot even differentiate between GPT-3 and GPT-4. I am not sure if the applied benchmarks contain complext reasoning tasks, e.g. math, programming, multi-hop qa, etc. If not, then the paper does not bring new insights - it is already known that a pretrained model, e.g. LLaMA and BLIP-2 can handle questions that is not complicated enough without any neural symbolic add-ons.\n\nI humbly ask the authors to introduce more about the benchmarks and their difficulties. Otherwise the conclusion of the paper is not strong or grounded enough.", "questions_to_authors": "see weakness.", "paper_topic_and_main_contributions": "This work compared prompt-based and neural-symbolic methods on several vision-language tasks, and found that the prompt-based method (based on BLIP-2) can achieve comparable results as the neural-symbolic method (ViperGPT).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts a comprehensive analysis of visual programming in the context of visual question answering (VQA), with a particular focus on ViperGPT. The key contributions include highlighting the power of question decomposition in VQA tasks, proposing an iterative prompting-based approach for visual question decomposition, and offering insights into the performance gain achieved by visual programming in ViperGPT. Overall, this paper offers valuable insights into the role of visual programming in VQA and presents an alternative approach to question decomposition. Both the AC and reviewers are excited about this work. With some improvements in organization and additional context in the revision, it can make a significant contribution to the field.
Decision: Accept-Main