Title: Exploring Question Decomposition for Zero-Shot VQA
Venue: NeurIPS 2023
Accepted: yes
Keywords: visual question answering

in-context learning

vision-language
Abstract: Visual question answering (VQA) has traditionally been treated as a single-step task where each question receives the same amount of effort, unlike natural human question-answering strategies. We explore a question decomposition strategy for VQA to overcome this limitation. We probe the ability of recently developed large vision-language models to use human-written decompositions and produce their own decompositions of visual questions, finding they are capable of learning both tasks from demonstrations alone.
However, we show that naive application of model-written decompositions can hurt performance.
We introduce a model-driven selective decomposition approach for second-guessing predictions and correcting errors, and validate its effectiveness on eight VQA tasks across three domains, showing consistent improvements in accuracy, including improvements of >20% on medical VQA datasets and boosting the zero-shot performance of BLIP-2 above chance on a VQA reformulation of the challenging Winoground task. Project Site: https://zaidkhan.me/decomposition-0shot-vqa/

Exploring Question Decomposition for Zero-Shot VQA

Abstract

Visual question answering (VQA) has traditionally been treated as a single-step task where each question receives the same amount of effort, unlike natural human question-answering strategies. We explore a question decomposition strategy for VQA to overcome this limitation. We probe the ability of recently developed large vision-language models to use human-written decompositions and produce their own decompositions of visual questions, finding they are capable of learning both tasks from demonstrations alone. However, we show that naive application of model-written decompositions can hurt performance. We introduce a model-driven selective decomposition approach for second-guessing predictions and correcting errors, and validate its effectiveness on eight VQA tasks across three domains, showing consistent improvements in accuracy, including improvements of > 20 % on medical VQA datasets and boosting the zero-shot performance of BLIP-2 above chance on a VQA reformulation of the challenging Winoground task. Project Site: https://zaidkhan.me/decomposition-0shot-vqa/

1 Introduction

On a question-answering test, humans are able to answer some questions in a single step, while other questions require potential deliberation and second-guessing. Visual question answering (VQA) [1-3] has traditionally been treated as a single-step task. Models only get one chance for each question, and each question receives equal amounts of computation. This is incongruent to the natural human approach to such tasks, where simple perceptual questions are quickly answered, while harder reasoning questions are allocated more time and computation.

The emergence of task decomposition techniques for large language models (LLMs) [4] is a potential solution to this incongruency. Task decomposition techniques prompt a LLM to break down an initial complex task into simpler subtasks that can each be solved independently. However, VQA has not benefited from advances in task decomposition techniques for two reasons. First, many task decomposition techniques [5, 6] have only been effective in the regime of very large unimodal LLMs with parameters in the 30B+ range, while the LLMs underlying vision-language models are typically much smaller, only recently reaching ≈ 13 b parameters for publicly available models[7-9]. Second, existing methods for prompting vision-language models (VLMs) during VQA tasks focus on other use cases, such as providing more examples of the input task [10] or more information about the image [11]. Given the recent emergence of multi-billion scale VLMs, our main research question is:

Can multi-billion scale vision-language models benefit by approaching reasoningheavy VQA as a two-step rather than a single-step problem using decomposition?

To this end, we explore a form of task decomposition called question decomposition as a strategy for zero-shot visual question answering with large VLMs. Although question decomposition has been explored for specific unimodal QA[12-14], it has not been explored as a strategy for multimodal

tasks such as VQA with emerging large VLMs [7-9, 15, 16], and little is known about the in-context learning ability of emerging large VLMs.

First, we probe the in-context learning ability [17-19] of both LMs and VLMs to exploit oracular question decompositions written by humans. We design experiments to understand whether models can learn to use decompositions without explicit training, and whether they are merely exploiting keywords and surface statistics when they use decompositions. Second, we conduct a series of experiments, again using in-context learning, to understand how well models can produce decompositions that correct the errors of a fixed VQA model. Last, we propose and study an entirely model-driven closed-loop approach mimicking a simplified form of a classic human second-guessing strategy: second-guess answers based on how confident you are about them. We conduct experiments across three domains (art, natural images, medical), eight datasets, three model families, and model sizes ranging from 80M to 11B parameters. Our contributions can be listed as follows:

We experimentally demonstrate that large VLMs based on instruction-tuned LLMs can use decompositions to improve their predictions without any training, and are not merely exploiting changes in word statistics introduced by the decomposition. (Sec. 3)

We quantitatively show that generative, instruction-tuned language models are capable of writing effective decompositions zero-shot, without task-specific training. (Sec. 4)

We find that applying decomposition naively to every question instance harms performance rather than helps (Fig. 4), and propose selective decomposition (Fig. 3), a modular, modelagnostic, training-free strategy that treats VQA as a two-step task. (Sec. 5)

We apply selective decomposition to a testbed of 8 datasets and show that it consistently improves performance (Tabs. 3 and 4), with gains of > 20% on medical VQA datasets[20-22], and boosts the performance of BLIP-2[7] above chance on the Winoground[23] benchmark when formulated as a VQA task. (Sec. 5).

2 Background

2.1 Problem Setting

In zero-shot VQA, a model /u1D453 : /u1D463 , /u1D45E → /u1D44E is given an image /u1D463 , a question /u1D45E , and outputs an answer, /u1D44E . Unlike traditional VQA, the model /u1D453 (·) has never seen /u1D463 , /u1D45E , /u1D44E triplets. In practice, such a setting often occurs when /u1D453 (·) is a foundation model that contains several billion parameters and has undergone large scale pretraining. It is undesirable to retrain such an /u1D453 (·) on visual question answering pairs specifically, both for reasons of computational convenience and because finetuning can degrade robustness[24]. The most common case is that /u1D453 (·) is an autoregressive, generative language model that can optionally be conditioned on the visual modality. We restrict ourselves to such models, which approximate Π /u1D441 /u1D458 = 1 /u1D45D ( /u1D461 /u1D458 + 1 | /u1D461 1: /u1D458 , /u1D463 ) , where /u1D463 is an image and /u1D461 1: /u1D458 is a sequence of language tokens. In a zero-shot VQA setting, it is expected that /u1D453 (·) understands that it has been given a question /u1D45E and should produce the correct answer /u1D44E to the question /u1D45E in the context of the image /u1D463 by modeling it as /u1D45D ( /u1D44E | /u1D463 , /u1D45E ) . This setting is common when evaluating very large frozen models, such as in [10, 11], with the exception that in our case, /u1D453 (·) is a vision-language model rather than a language-only model.

2.2 Question Decomposition

Question decomposition is the task of decomposing a complex main question into one or more simpler subquestions that are logically related to the main question, answering those simpler subquestion(s), and then using the answered subquestion(s) to help in composing a final answer to the complex main question. This is a strategy often used by humans for problem solving. For example, consider a human being confronted by a wild animal they have never seen before. To answer the main question ' does this animal pose a threat to me? ' a human might decompose it into subquestions such as ' does the animal have sharp canine teeth? ' and ' does the animal have forward facing eyes typical of a predator? ' Knowing the answer to even one of these subquestions makes answering the main question much easier.

Adopting the terminology of Sec. 2.1, the task of question decomposition consists of decomposing a main visual question /u1D463 , /u1D45E into one or more subquestions ( /u1D460 1 , /u1D460 2 , . . . ) , answering those subquestions to

Q: What country

headquarters this

pl ane company?

Before: Canada

After:   UK

Decomposition

what is the name of the

pl ane company? virgin

atl antic

Q: What could keep

t his food from falling

apart?

Before: knife

After:   bread

Decomposition

what is holding the

food together? bread

Q: What is the woman

using the umbrella to

protect herself from?

Before: rain

After:   the sun

Decomposition

what is the weather like

outside? sunny

Q: What kind of show

i s t he man on?

Before: cricket

After:   talk show

Decomposition

what is the man

wearing? a suit

Q: does the picture

contain lungs?

Before: yes

After:   no

Decomposition

i s t here a person in the

picture? no

Q: is this an x-ray?

Before: no

After:   yes

Decomposition

i s t he pict ure bl ack and

white? yes

Q: is this a transverse

section?

Before: no

After:   yes

Decomposition

i s t his a x- ray of a

person's brain? no, mri

Q: Does adrenal show

gout?

Before: a case report

After:   no

Decomposition

what is the cause of

gout? uric acid crystal s

Q: What is the second

shelf in the fridge

used for?

Before: food

After:   drink

Decomposition

what is the first shelf in

t he f ridge used for?

Q: The time of day

suggests the man will

do what soon?

Before: sleep

After:   wake up

Decomposition

what is the time of

day? morning

obtain the decomposition (( /u1D45E ′ 1 , /u1D44E ′ 1 ) , ( /u1D45E ′ 2 , /u1D44E ′ 2 ) . . . ) , and then using /u1D463 , /u1D45E together with the decomposition (( /u1D45E ′ 1 , /u1D44E ′ 1 ) , ( /u1D45E ′ 2 , /u1D44E ′ 2 ) . . . ) to obtain the final answer /u1D44E .

2.3 What makes a good subquestion?

In Sec. 2.2, we gave a definition of decompositions that is dependent on notions of "simpler" and "logically related". It is challenging to make these notions precise, and difficult to operationalize them to measure whether a sequence of text really is a valid subquestion according to these notions. To sidestep these difficulties, we adopt a consequentialist view of whether a subquestion is 'good', following a common consequentalist tradition in artificial intelligence as a whole [25]. We evaluate the 'goodness' of a subquestion by measuring the effect of the subquestion. Concretely, let /u1D463 , /u1D45E , /u1D44E be a visual question triplet where /u1D463 is the image, /u1D45E is the question, and /u1D44E is the answer. Let /u1D45D /u1D453 ( /u1D44E | /u1D463 , /u1D45E ) be the probability of the ground-truth answer /u1D44E as assessed by a visual question answering system /u1D453 (·) . We regard a decomposition of /u1D463 , /u1D45E consisting of series of subquestions and their answers (( /u1D45E ′ 1 , /u1D44E ′ 1 ) , ( /u1D45E ′ 2 , /u1D44E ′ 2 ) . . . ) as 'good' if /u1D45D /u1D453 ( /u1D44E | /u1D463 , /u1D45E ) < /u1D45D /u1D453 ( /u1D44E | /u1D463 , /u1D45E , (( /u1D45E ′ 1 , /u1D44E ′ 1 ) , ( /u1D45E ′ 2 , /u1D44E ′ 2 ) . . . )) , that is, if seeing the decomposition increases the probability of the ground-truth answer /u1D44E . In practice, we adopt a simpler criterion that takes the consequentalist definition to the limit. We regard a decomposition as 'good' if seeing the decomposition induces the model to produce the true ground-truth answer /u1D44E .

2.4 Scope & Limitations

We only consider in-context learning techniques for zero-shot VQA, and do not explore full model training in this work. The class of model we are interested in are instruction-following vision-language models based on large language models [7-9]. This excludes previous-generation vision-language models that are not based on multi-billion parameter instruction-tuned language models [26-30].

Table 1: Models are capable of using decompositions written by humans to provide more accurate answers. The gray rows are the baseline performance with no decomposition, and each Δ is calculated w.r.t to this baseline. Oracle/Oracle rows denoting oracle subquestions/oracle answers, have the highest Δ . 'Self-Answer' means the model answered oracular subquestions itself, and 'No Answer' indicates the answer was left out entirely. Image+Text indicates a vision-language model (BLIP-2) was tested with multimodal inputs, while Text indicates the corresponding language model inside BLIP-2 (FLAN-T5) was tested with text only inputs. Validation split of VQA Introspect is the dataset (22k reasoning questions with their associated decompositions).

Not all datasets are suitable for exploring question decomposition, as some primarily test low-level perception skills rather than high-level reasoning skills that would benefit from a decomposition. We thus limit our evaluation to datasets that explicitly test for high-level reasoning / knowledge-based ability. We are few-shot for the task of visual question decomposition but zero-shot for the task of visual question answering .

3 How well can models use decompositions?

Our goal in this section is to understand the ability of vision-language models based on large language models to consume decompositions. The hypothesis we test is: When provided with gold-standard decompositions on a VQA task, a model's error rate should be lower than without the gold-standard decompositions. Evaluating this hypothesis presents a number of challenges. First, how can we obtain a set of decompositions that are apriori 'known to be good'? Second, how should the model be fed the decompositions?

To find a source of apriori 'good' decompositions, we turn to the literature on internal consistency in visual question answering. To probe consistency in question answering systems, several datasets [3133] have been proposed. A particularly relevant case of such a dataset is VQA-Introspect [32], which probes consistency along a reasoning-perception axis. Selvaraju et al. [32] annotate each question in the VQAv2[1] validation set as a high-level 'reasoning' question or a low-level 'perception' question. For each 'reasoning' question, Selvaraju et al. [32] write 1-3 'perceptual' subquestions which are implied by the reasoning question. For example, given a high-level reasoning question such as 'Can I eat this banana?' a model that says 'yes' should also reply 'yellow' to the low-level perception question 'what is the color of the banana?' We propose to use the low-level perception questions and answers written for the high-level reasoning questions as an oracular decomposition for the high-level reasoning question, on the basis that the low-level perception questions are simpler than the high-level reasoning question, entail the answer for the high-level reasoning question, and are written by humans.

The second challenge lies in using a decomposition consisting of a series of subquestions and answers (( /u1D45E ′ 1 , /u1D44E ′ 1 ) , ( /u1D45E ′ 2 , /u1D44E ′ 2 ) . . . ) alongside a main visual question ( /u1D463 , /u1D45E ) . Recall that we cannot train the model /u1D453 (·) being used for the visual question answering task, and for any arbitrary model, it is unknown whether the model has ever seen the exact task of decomposition-aided visual question answering. Thus, we rely on the in-context learning ability [18, 19] of large language models to learn to perform the tasks we require from a demonstration of the task. We handcraft a simple prompt to contain a main visual question /u1D463 , /u1D45E from the VQAv2 validation set, along with one human-written oracular subquestion and human-written answer /u1D45E ′ , /u1D44E ′ for the main question /u1D463 , /u1D45E extracted from VQA-Introspect. The prompt is simply

Experiments & Discussion We use BLIP-2 [7]models based on the instruction-tuned FLAN-T5[34] in 3B and 13B sizes. Experiments are run on a combination of A6000s and TPUv3s, on the VQAIntrospect validation set containing 22K reasoning questions and their associated decompositions. The results are shown in Tab. 1. Compared to the baseline with no oracular decompositions, both the 3B/13B vision-language models and their corresponding language models show a clear ability to benefit from decompositions across a variety of question types, with numerical questions benefiting the most. Next, we seek to gain insight into the mechanism by which decompositions aid inference. Is the model merely exploiting changes in surface level statistics? If so, we would expect that perturbations that leave the statistics largely unchanged but significantly alter the meaning and logical structure of the oracle decomposition should not result in significantly different performance from the unaltered oracle decompositions. We remove the answers from the decomposition so that it only contains the subquestions, and test the effect of only using the subquestions. Compared to the oracle, there is a significant 50 %relative decrease in improvement w.r.t to the baseline. Most of the subquestion answers are boolean, so removing them should not significantly change content words in the prompt, though it changes the meaning of the context significantly. Next, we allow the models to answer the subquestions themselves (Oracle/Self-Answer) rather than using the ground-truth questions. The accuracy of all models again decreases relative to the oracle answers, suggesting the answer and question together contribute to the result. Finally, we take the oracle subquestion+answer and scramble the words before providing them to the models. If the model is merely exploiting surface level statistics, the performance difference between the scrambled oracular decompositions and the original decompositions should be minimal, as the words are all the same. Again, we observe a significant drop compared to the original decompositions, suggesting that the models are not merely exploiting changes in the surface level statistics . Furthermore, human-written decompositions help in almost all cases over the no-decomposition baseline. Note: See supplement for complete experimental details for all experiments .

4 Can models produce effective decompositions?

In this section, we conduct experiments to answer the following research questions:

Can language models ≤ 13 B parameters learn to produce effective decompositions purely through demonstrations?

Is question decomposition mostly a linguistic ability, or is being able to see the image important?

Recall that a decomposition of a visual question /u1D463 , /u1D45E is a series of one or more subquestions (( /u1D45E ′ 1 , /u1D44E ′ 1 ) , ( /u1D45E ′ 2 , /u1D44E ′ 2 ) . . . ) and their answers, with the constraint that the subquestions and answers should have the property that /u1D45D /u1D453 ( /u1D44E | /u1D463 , /u1D45E ) < /u1D45D /u1D453 ( /u1D44E | /u1D463 , /u1D45E , (( /u1D45E ′ 1 , /u1D44E ′ 1 ) , ( /u1D45E ′ 2 , /u1D44E ′ 2 ) . . . )) where /u1D45D /u1D453 (·) represents probability assessed by a given vision-language model /u1D453 (·) of the ground-truth answer /u1D44E . We simplify this task to the task of producing a single subquestion /u1D45E ′ given a main visual question /u1D463 , /u1D45E , and denote the process of decomposition with an arbitrary autoregressive language model /u1D454 (·) as /u1D451 /u1D454 ( /u1D463 , /u1D45E ) → /u1D45E ′ . We hereafter refer to the model /u1D454 (·) that generates the decomposition as the decomposer . The subquestion is then answered by the vision-language model /u1D453 ( /u1D463 , /u1D45E ′ ) = /u1D44E ′ to produce the subquestion-answer pair ( /u1D45E ′ , /u1D44E ′ ) . We call the question answering model the recomposer.

Reasoning Question:

i s t his

t own in China?

Perception

Question:

<exemplar demonstrations>

( optional)

what language are the signs

written in?

Recomposer

Recomposer

english

Question:

i s t his t own in

China?

Short Answer:

Context:

what language are

t he signs written in? English

no

Model Generated Decomposition

Recomposition

1

generate subquestion

2

answer subquestion

3

answer main question

aided by decomposition

We then measure the effectiveness of the decomposition by measuring the error correction rate :

where ( /u1D463 /u1D456 , /u1D45E /u1D456 , /u1D44E /u1D456 ) represent the /u1D456 -th image, question, and ground-truth answer respectively, and /u1D45E ′ /u1D456 , /u1D44E ′ /u1D456 represent a subquestion generated by the decomposer model and the answer predicted for the subquestion by the recomposer (VQA) model, and /u1D7D9 [ /u1D450 /u1D45C /u1D45B /u1D451 ] is an indicator function that is equal to 1 when /u1D450 /u1D45C /u1D45B /u1D451 is true and 0 otherwise. Simply put, E /u1D436 /u1D445 measures the number of instances on which /u1D453 (·) initially predicted a wrong answer, but switched to the correct answer after seeing the decomposition generated by /u1D454 (·) . Alternatively, this can be understood as the effectiveness of a decomposer model at correcting the errors of the recomposer model. The error induction rate E /u1D43C /u1D436 is the opposite:

and measures how often the produced decompositions flipped an answer that was initially correct to an incorrect answer. The decomposer can be the same as the recomposer if the model can do both tasks by following different prompts, as in the case of instruction-tuned models [38].

Experiments & Discussion We use BLIP-2 [7] based on the FLAN-T5[34] as the question answering model (recomposer). For the decomposers, we use FLAN-T5[34] models ranging in size from 80 Mparameters to 11 B parameters, as well as the BLIP-2 models themselves. We use four VQA

datasets from three domains: ArtVQA[36] (art), SLAKE[20] (medical), and A-OKVQA[35] and OKVQA [37] (external knowledge VQA on natural images). We then carry out the procedure illustrated in Fig. 2 for each combination of decomposer, recomposer, and dataset. We handcraft three demonstrations of writing a subquestion for a question, in the form ' Reasoning Question: <question>? Perception Question: <subquestion>? ' For each /u1D463 , /u1D45E pair in a dataset, we prompt the decomposer with the demonstration, followed by the question /u1D45E as in Fig. 2, and measure E /u1D436 /u1D445 as in Eq. (1) and E /u1D43C /u1D436 as in Eq. (2) for each dataset. We show the results in Tab. 2. We find that yes, language models ≤ 13B parameters can learn to produce effective decompositions just by viewing examples . Decomposer size correlates positively with E /u1D436 /u1D445 ( /u1D445 2 = 0 . 344 ), and negatively with E /u1D43C /u1D436 ( /u1D445 2 = . 273 ) and the correlations are significant at /u1D6FC = 0 . 05 across a larger collection of eight datasets used in Sec. 5. A human examination of the 'subquestions' produced by smaller models shows that many of them are gibberish and not properly formed questions at all. Despite this, they surprisingly manage to maintain an /u1D438 /u1D436 /u1D445 that is sometimes higher than larger models. Finally, the ability to decompose questions in the evaluated datasets may be a primarily linguistic ability in that it is possible to ask effective subquestions about an image without being able to see the image, and the difference in effectiveness between the Image+Text BLIP-2 models and the text-only FLAN-T5 models of a similar size is on average ≈ 10 %of the base error rate (but this may not be true of other VQA datasets).

5 Selective Decomposition Works Better Than Naive Decomposition

One problem shows up in Sec. 4, which is that applying decompostions to every question can hurt performance, by flipping answers that were initially correct to be incorrect. If we were able to decompose only wrong answers, we would always see a net gain in performance due to the error correction of decompositions. However, in a realistic setting, we do not know apriori that our answers are wrong, and thus run the risk of flipping an answer that was initially correct to be incorrect by applying a decomposition that is misleading. We call this the second-guessing problem.

To deal with the second-guessing problem, we propose following an intuitive human strategy: stick with your initial answer on questions you are confident about, and only second-guess (apply a decomposition) for questions you are not confident about. Language models can be surprisingly well calibrated[39], meaning that the probability they assess to an output sequence they produce is often wellcorrelated with the probability that the produced output sequence is the 'correct' one for a given task. We make use of this property to treat visual question answering as a selective prediction [40] task, using the language models's confidence as a decision score to determine whether we should apply a decomposition to a instance or stick with the original answer.

We describe the algorithm in pseudocode in Fig. 3. The selective decomposition procedure transforms VQA from a single-step task to a two-step task given a decomposer model, a recomposer model, a confidence threshold /u1D70F , and a visual question pair /u1D463 , /u1D45E . An initial answer ˆ /u1D44E and confidence /u1D45D ( ˆ /u1D44E ) is solicited from the recomposer model. If /u1D45D ( ˆ /u1D44E ) < /u1D70F , the decomposition procedure is invoked, and a subquestion and answer pair ( /u1D45E ′ , /u1D44E ′ ) are generated by the decomposer and recomposer working together. The recomposer model is then allowed to 'second-guess' the inital answer ˆ /u1D44E with the decomposition ( /u1D45E ′ , /u1D44E ′ ) as additional context. The decomposer and recomposer can be the same model or different models. We experiment with both scenarios. This introduces an extra hyperparameter /u1D70F into the inference procedure.

Experiments & Discussion In Fig. 4, we show the effect of different values of /u1D70F on the accuracy of selective decomposition with several decomposers. Across all datasets and all models, there is a wide range of /u1D70F (expressed as percentiles) for which selective decomposition improves predictive accuracy. At the same time, we clearly demonstrate the second guessing problem in Fig. 4. Decomposing every question often eventually leads to lower accuracy than decomposing no questions at all, because hallucinations and misleading decompositions can flip an initially correct answer to an incorrect answer.

In Tabs. 3 and 4, we show the highest possible net gain achieved by selective question decomposition on three domains by different decomposers. Selective decomposition consistently improves predictive accuracy regardless of the decomposer and domain. Net gains are larger on datasets (passes t-test with /u1D6FC = 0 . 05 ) containing non-naturalistic images and specialized domains (e.g. medical) than they are on domains containing natural images. The mean optimal surprisal /u1D43C ( /u1D70F ) for second-guessing answers is lower for non-natural domains ( /u1D707 /u1D43C ( /u1D70F ) = 13 . 2 for the natural image datasets vs /u1D707 /u1D43C ( /u1D70F ) = 9 . 0 for the medical and art datasets, confirmed by t-test at /u1D6FC = 0 . 5 ). We further visualize this in Fig. 5. This matches our expectations: you should second guess yourself on domains you understand poorly more than on domains you understand well. A linear regression fit shows that larger decomposers correlate with larger net gains ( /u1D445 2 =0.365, /u1D445 2 =0.342 for natural image domains and medical / art domains respectively, t-test with /u1D6FC =0.05).

We reformulate Winoground as a VQA task by turning each caption into a question with a boolean yes / no answer (does '<caption>' describe the image?) on which chance accuracy is 50 % . As visible in Tab. 3, all BLIP-2 models perform below random chance, in agreement with previous results on Winoground showing that it is extremely difficult for vision-language models. Surprisingly, after decompositions produced by the relatively FLAN-T5-small/base models (80M/200M parameters), the performance of BLIP-2 (13B) rises to significantly above chance ( + 18 %). Upon inspection, many of the decompositions produced by the model appear to be gibberish, yet remarkably, induce the much larger 13B BLIP-2 model to correct over 30 %of its initially wrong answers.

6 Literature Review

Task decomposition [5, 6, 41, 42] improves the performance of large language models on zero-shot reasoning tasks. The only work so far to apply similar techniques for VQA is MM-CoT [43], but it does not explore task decomposition with large vision-language models, choosing to finetune a smaller model instead. The ability to use zero-shot task decompositions may be a property of model scale, emerging at 60-200B parameters [44], or may be a property of large-scale pretraining on code [45]. Such large vision-language models have only been developed recently due to advances in vision-language alignment. The prevailing paradigm in vision-language pretraining was to build vision-language models atop (relatively) small language models [26-29, 46] below 1B parameters. Meanwhile, language models were being scaled from 3B-175B parameters [34, 47-50], with each model family having at least one representative with > 10 B parameters. Because vision-language

pretraining typically requires full model training, aligning these multi-billion parameter models to the visual modality was prohibitively computationally expensive. However, recent discoveries [51, 52] motivated by earlier work with frozen models [53] have shown that the representation spaces of vision models and large-language models are surprisingly close, and rough alignment can be achieved with adapters [15] or linear mapping layers while keeping the language model frozen, and more advanced techniques have given rise to vision-LLMs [7-9]. Our work is closely related to the visual question generation paradigm of [11, 54, 55]. However, we direct our question generation to focus on decompositions rather than general questions.

7 Conclusion

Weshow that question decomposition is already a viable strategy that can be used to implement a more natural approach to VQA. Without any training, instruction-tuned VLMs can learn to produce and use decompositions from a few demonstrations of the task. This approach has many possible future directions. For example, we only consider two-step approaches for visual question answering, where we 'hardcode' the depth of the decomposition. A natural next step would be to extend the two-step approach to a multi-step approach, which remains unexplored for large vision-language models in an open-world visual question answering setting. Second, in-context learning has limitations. Would models benefit from being trained to produce and consume decompositions?

References

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering.

International Journal of Computer Vision , 127:398-414, 2016.

Zaid Khan, B Vijaykumar, Samuel Schulter, Xiang Yu, Yun Raymond Fu, and Manmohan Chandraker. Q: How to specialize large vision-language models to data-scarce vqa tasks? a: Self-train on unlabeled images! In CVPR , 2023.

Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision , 123:4 - 31, 2015. URL https://api.semanticscholar.org/CorpusID:3180429 .

Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey. ArXiv , abs/2302.07842, 2023.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS , 2022.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR , 2023.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv , abs/2301.12597, 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv , abs/2304.08485, 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv , abs/2304.10592, 2023.

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI , 2021.

Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In CVPR , 2023.

Ethan Perez, Patrick Lewis, Wen tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In EMNLP , 2020.

Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. NAACLHLT , 2021.

Pruthvi H. Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. Is a question decomposition unit all we need? In EMNLP , 2022.

Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA - multimodal augmentation of generative models through adapter-based finetuning. In EMNLP Findings , 2022.

Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In ICML , 2023.

Stephanie Chan, Adam Santoro, Andrew K. Lampinen, Jane Wang, Aaditya Singh, Pierre H. Richemond, James L. McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In NeurIPS , 2022.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP , 2022.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In ICLR , 2022.

Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Fang Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In ISBI , 2021.

Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific Data , 5, 2018.

Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286 , 2020.

Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR , 2022.

Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR , 2022.

Stewart Russel and Peter Norvig. Artificial Intelligence: A Modern Approach , chapter 2, page 39. Pearson, 4th edition, 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In ICML , 2022.

Zaid Khan, B Vijaykumar, Xiang Yu, Samuel Schulter, Manmohan Chandraker, and Yun Raymond Fu. Single-stream multi-level alignment for vision-language pretraining. In ECCV , 2022.

Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS , 2021.

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. ArXiv , abs/2206.08916, 2022.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML , 2022.

Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. In ECCV , 2020.

R. R. Selvaraju, P. Tendulkar, D. Parikh, E. Horvitz, M. Tulio Ribeiro, B. Nushi, and E. Kamar. Squinting at vqa models: Introspecting vqa models with sub-questions. In CVPR , 2020.

Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-consistency for robust visual question answering. In CVPR , 2019.

Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. ArXiv , abs/2210.11416, 2022.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV , 2022.

Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura. A dataset and baselines for visual question answering on art. In ECCV Workshops , 2020.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR , 2019.

Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR , 2022.

Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. Language models (mostly) know what they know. ArXiv , abs/2207.05221, 2022.

Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph E. Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In ECCV , 2022.

Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. Least-to-most prompting enables complex reasoning in large language models. In ICLR , 2023.

Tushar Khot, H. Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In ICLR , 2023.

Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923 , 2023.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res. , 2022.

Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 1384-1403, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.emnlp-main.90. URL https://aclanthology.org/2022.emnlp-main.90 .

Zaid Khan, Vijay Kumar BG, Samuel Schulter, Xiang Yu, Yun Fu, and Manmohan Chandraker. Q: How to specialize large vision-language models to data-scarce vqa tasks? a: Self-train on unlabeled images! In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15005-15015, June 2023.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony S. Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArXiv , abs/2211.09085, 2022.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. ArXiv , abs/2205.01068, 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur'elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv , abs/2302.13971, 2023.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, P. Zhang, Yuxiao Dong, and Jie Tang. Glm-130b: An open bilingual pre-trained model. ArXiv , abs/2210.02414, 2022.

Zaid Khan and Yun Fu. Contrastive alignment of vision to language through parameter-efficient transfer learning. In ICLR , 2023.

Jack Merullo, Louis Castricato, Carsten Eickhoff, and Elizabeth-Jane Pavlick. Linearly mapping from image to text space. In ICLR , 2023.

Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In NeurIPS , 2021.

Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. Weaqa: Weak supervision via captions for visual question answering. In ACL Findings , 2020. URL https://api. semanticscholar.org/CorpusID:235262536 .

Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for vqa are image captions. In NAACL , 2022. URL https: //api.semanticscholar.org/CorpusID:248512968 .

Dongxu Li, Junnan Li, Hung Le, Guangsen Wang, Silvio Savarese, and Steven C. H. Hoi. Lavis: A library for language-vision intelligence, 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. ArXiv , abs/1910.03771, 2019.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. In EMNLP , 2022.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An opensource framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116 , 2023. URL https://arxiv.org/abs/2306.01116 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the question decomposition of VQA.\nTo begin with, the authors demonstrate that large VLMs equipped with LLMs can effectively leverage question decomposition to improve answering performance.\nThereafter, some experiments are conducted to prove that instruction-tuned language models are capable of auto-writing decompositions.\nSince some questions do not require decomposition to reach the right answer, the authors then propose a novel selective decomposition method to selectively decide which questions demand decomposition.\n\nSTRENGTHS:\n- The paper is well-written and well-organized. Most parts of this paper are easy to follow.\n- The studied problem is interesting and novel. It helps us to understand the in-context learning capability in large VLMs with instruction-tuned LLMs.\n- To validate the effectiveness of question decomposition, the authors perform comprehensive experiments.\n\nWEAKNESSES:\n- My biggest concern lies in Sec. 4 about whether large models can produce effective decompositions. As shown in Table 2, most $E_{CR}$ values are pretty low while $E_{IC}$ are relatively high. According to the definitions of these two terms, it seems that most decompositions severely hurt the model performance. For example, 10% $E_{CR}$ means only 1 of 10 vqa instances can be corrected by the recomposer. In contrast, the average value of $E_{IC}$ is around 40. This means 4 of 10 vqa instances after decomposition failed. \n- For Sec. 5, have authors tried these newly introduced metrics after the selective decomposition?\n- Have the authors considered using the GQA dataset [1], which involves a longer chain of reasoning?\n- About Table 1, the oracle/no answer outperforms oracle/self-answer by a large performance margin. This result does not make much sense since using more context usually leads to more accurate answer predictions.\n- Moreover, the scrambling is not well-explained. How many words are perturbed? This question is important because a larger perturbation often causes a severe performance drop. In addition, why the authors have not tried scrambling the questions? This is also essential because the well-known bias problem in VQA is from question key words and answers.\n\nTypo:\n- Fig. 1 is not referred in the main manuscript.\n\n[1] GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In CVPR 2019.\n\nQUESTIONS:\nOverall, I believe this is a good paper.\nHowever, the effectiveness of question decomposition from large VLMs is confusing.\nIf the authors can address this concern, I will raise my initial score.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo answer the questions that require reasoning or outside knowledge, this paper proposes a question decomposition method which uses LLM to infer subquestions\n\nSTRENGTHS:\n1. The proposed method is intuitive and interesting.\n\n2. The methods conduct comprehensive experiments on BLIP2 and FlanT5, and the results clearly demonstrate the effectiveness of the proposed methods.\n\nWEAKNESSES:\n1. While this research paper serves as an investigative study, it only validates the proposed methods using FlanT5 and BLIP2 models. This limited scope raises questions about the generalizability of the proposed methods. It would enhance the study's breadth if further tests were conducted on a diverse range of zero-shot VQA methods, such as Flamingo [1], Frozen [2], and Img2LLM [3]. Additionally, incorporating various Language Learning Models (LLMs) for decomoposing questions like OPT and the GPT series would further strengthen the results.\n\n2. I acknowledge that the proposed methods are primarily designed for tasks necessitating reasoning. However, given the community's growing interest in open-ended environments, it may be beneficial to conduct some analysis on straightforward VQA tasks. For instance, the experimental results on tasks like VQAv2 or GQA could provide valuable insights.\n\n3. The proposal of self-generated question-answer pairs into VQA tasks is not a novel concept. To improve the understanding of how this method compares with prior approaches, a dedicated section comparing it with earlier methods such as VQ^2A [4], Weak VQA [5], and Img2LLM would be beneficial.\n\n[1] Alayrac, J. B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., ... & Simonyan, K. (2022). Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35, 23716-23736.\n\n[2] Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S. M., Vinyals, O., & Hill, F. (2021). Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34, 200-212.\n\n[3] Guo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., & Hoi, S. C. (2022). From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846.\n\n[4] Changpinyo, S., Kukliansky, D., Szpektor, I., Chen, X., Ding, N., & Soricut, R. (2022). All you may need for vqa are image captions. arXiv preprint arXiv:2205.01883.\n\n[5] Banerjee, P., Gokhale, T., Yang, Y., & Baral, C. (2020). WeaQA: Weak supervision via captions for visual question answering. arXiv preprint arXiv:2012.02356.\n\nQUESTIONS:\nPlease refer to \"Weaknesses\"\n\nLIMITATIONS:\nThe proposed methods lacks the comprehensive analysis on different zero-shot VQA models and LLM models, which raises the concerns about the generalization ability.  In addition, the paper should also  conduct some analysis on straightforward VQA tasks to demonstrate the generalization ability of the proposed methods with different question types.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examine whether the current instruction-tuned VLM (e.g. BLIP2 ) can (1) learn to generate simple perception questions that can help to answer a more complex reasoning visual question; (2) learn to actually benefit from the perception questions.\n\nThe conclusion is that \n\nfor (1): language models (with various sizes) can learn to produce effective decompositions just by viewing examples and adding image as content is not being super helpful.\n\nfor (2): Yes, the performance of the reason question answering is better with golden perception question even without providing answers to the perception question; and achieve good results with selective Decomposition\n\nSTRENGTHS:\n(1) The paper provide very detailed discussion on how the flan model can be used to generate perception questions based on only 2 exemplars. The results are very interesting and surprising in that (a) even small size model can provide \"decomposition\" (sometime gibberish) that helps the recomposer model and visual input does not help a lot\n\n\n(2) test the selective decomposition that actually using the generated perception question to boost the reasoning question without having to know whether the question is requiring the  perception question \n\n\n(3) The paper is well-written and easy to follow\n\nWEAKNESSES:\n(1) It is very surprising that the visual input does not help generate better perception questions. For example, in the supplement material, the reasoning question is \"is it cold outside\" and the perception question is \"are any people wearing jacket\". The perception question is very easy to be generated if we see a person in the picture, but without visual content, there could be lots of perception question that are valid, e.g. \"is there snow\", \"are there green leaves on the tree\",  \"is the lake frozen\" that may be irrelevant to the visual question\n\nMy hypothesis is that the image + text model does not use the image correctly in that the question generation process is not focusing on the image. I wonder if it is easy to see (a) adding prompts that explicit say the perception question should be based on the image (b) converting the image into multiple captions and use the captions as visual inputs\n\n\n(2) It seems the final recomposer should be able to take multiple perception questions as input, it would be good to sample multiple perception questions from the decomposer and use all of perception questions to see if it helps the performance.\n\nQUESTIONS:\nPlease comment on the weakness part\n\n\n\n\n\n\n------------------------------------\npost rebuttal: I read the author's rebuttal and still willing to support the paper as my initial rating\n\nLIMITATIONS:\nThe authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors focus on question difficulty in visual question answering (VQA). In particular, they propose to give subquestions that improve the percentage of correct answers to answer inferential questions more correctly. Using a large-scale vision-language model (VLM), the authors show that giving an oracle subquestion first improves the accuracy of the VQA. They also propose a selective decomposition for solving VQA so that the VLM model generates proper subquestions. This method is shown to improve VQA accuracy on multiple datasets.\n\nSTRENGTHS:\n- The authors utilize a large VLM to decompose questions to improve accuracy for VQA.\n- Validation using oracle subquestions and their answers demonstrates the contribution of question decomposition.\n- The effectiveness of the proposed method is shown for multiple parameter sizes and datasets.\n\nWEAKNESSES:\n- There is little technical novelty in the proposed method in this paper; it only provides an effective prompt for VQA.\n- Furthermore, as the authors acknowledge, despite the claim of question decomposition as its prompt, it also generates many subquestions that do not seem related to the question. As it stands, it remains to be seen why the accuracy of the VQA was improved.\n- The fact that the improvement in accuracy was reported only for BLIP-2 also makes it unclear whether such improvement by question decomposition is common to VLM models in general or whether it is a phenomenon found only in BLIP-2.\n\nQUESTIONS:\n- Please demonstrate that the proposed approach improves the accuracy of VQA in common with multiple VLM models.\n- In doing so, please quantitatively evaluate how many subquestions are generated that are not relevant to VQA, e.g., through user studies. Also, please verify whether such subquestions with little relevance contribute to improving the accuracy of the VQA and provide a discussion.\n- There are a few logic gaps, confusing parts, and areas that need proofreading.\n  - The second line says \"unlike natural human question-answering strategies,\" but the authors need to write the particular human strategies to lead to the subsequent statements.\n  - In Figure 1, there is no ground truth answers, so readers do not know if all the answers shown as After are correct. Alternatively, state that all After answers are correct.\n  - Only the description of Oracle/Oracle (Scrambled) in Table 1 is not present in the caption.\n  - While the VQA system is initially shown as f(v,q)=a, the function f(v,q,(q_i, a_i)) appears in the middle with the same function name without any declarations, causing confusion to the reader. In addition, while the initial discussion includes multiple subquestions and answers for the final answer, the middle of the discussion refers to only a single subquestion. Please clarify how many subquestions are generated and provided.\n  - Please specify what decomposer_parameters is in Figure 4; if it is the number of decomposer parameters, it is odd that it is a decimal.\n  - There is a mysterious indentation on line 234.\n  - In line 267, in the \"yes / no\" section, there is an extra space before and after the slash.\n  - The columns named Acc in Tables 3 and 4 seem unclear what Acc this is.\n  - The formatting of the items in the References is not consistent. There are also references that are cited as arxiv preprints even though they have already been published.\n\nLIMITATIONS:\nThe authors provide an adequate description of the scope and limitations of the proposed method in Section 2.4.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an investigation of a 1-step question decomposition strategy for VQA.\n\nAn example of a decomposition is the sub-question \"What is the name of the plane company?\" for the visual question \"What country headquarters this plane company?\" (Figure 1).\n\nThis paper finds:\n- Including human-annotated decompositions in a vision-language model (VLM) prompt leads to improved performance on VQAv2.\n- LMs/VLMs can generate decompositions that also lead to improved performances on four diverse VQA datasets (A-OKVQA, ArtVQA, OK-VQA, and SLAKE).\n- A \"selective decomposition\" strategy improves performance (i.e. where a model only generates decompositions when it originally produces a low confidence answer), as it should avoid decomposing \"harder to decompose\" questions.\n\nSTRENGTHS:\nThis paper presents several experiments for motivating and investigating the presented method. I believe it is a moderately important research contribution and is a sufficient package for publication.\n\n- The concept of question decomposition (and its use in prompting to improve performance in compositional question answering) is inspired from works in NLP using language models. However, question decomposition has not yet been investigated for visual QA tasks. Prior to this work, it was an open question whether LM/VLMs could decompose *visual* questions or make use of visual question decompositions.\n\n- The paper is generally well written, answers its research questions or demonstrates clear improvements for each experiment, and benchmarks on a diverse set of VQA datasets.\n\nWEAKNESSES:\n- This paper should have also measured the decomposer method (Table 2) on VQAv2. Then, we could have compared with the results from using human-annotated decompositions in Table 1.\n\n- I appreciate that this paper measures text-only biases in their experiments. But I think it is also important to particularly measure the textual biases of *only* generating decomposed questions or the final VQA answer.\n\n- It would have been nice to also see ablations from a more powerful (e.g. 100B-param, code-pretrained) language model to observe an higher bound for the text-only experiments in this paper.\n\nI feel that this paper needs to be a little more careful with some claims:\n\n- It is hypothesized that \"zero-shot task decomposition techniques is likely a property of model scaling\" (L281-282) and this directly informs research question 1 in Sec. 4. However, the cited paper [42] claims that techniques similar to this (e.g. chain of thought, least-to-most prompting) emerge at 60-200B parameters. This work compares models with 11B or fewer parameters. As the decomposition is limited to one step in this work, smaller models (e.g. 3B or 11B size) may plausibly be capable of this task, but the details are not yet clear. Finally, while this is one hypothesis, another line of work suggests that the capability comes (not from model scale but) from pre-training with code (Fu 2022).\n- Technically, the method in this paper is few-shot, since in-context examples are provided (L139).\n- I would adjust the following claim (L133-134): \"It is unlikely the model has ever seen the exact task of decomposition-aided visual question answering\". It is hard to make claims about the specific data that LMs are trained on.\n- I am not confident in the result from the \"Oracle (Scrambled)\" experiment (Table 2, Sec. 3). Do we know that the model is simply not able to override its priors and handle scrambled text effectively after just a few in-context examples? Is this decomposition \"primitive\" or can it be re-scrambled into multiple compositions? I think it's fine to exclude this experiment, as the text-only ablations are likely sufficient.\n- \"Decomposition appears to be a primarily linguistic ability\" (L207) is not necessarily generally true. This has only been shown on this particular set of datasets and perhaps this is an observation of the qualities of existing VQA datasets.\n\nAll being said, I believe adjusting these claims in-text will be sufficient and, although my concerns are detailed, I do not believe these entail any serious methodological issues.\n\nReferences:\n- Yao Fu, Hao Peng and Tushar Khot. (Dec 2022). How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources. Yao Fu’s Notion. https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1\n  - (Maybe there is a printed citation for this, but my point is just that there is an alternate hypothesis, as illustrated here.)\n\nQUESTIONS:\n- How many in-context examples are provided to the model? Have the authors experimented with different numbers of examples? Are these examples possibly distractors when provided to VQA models, as they are irrelevant to the provided image?\n\n- Do the authors know what % of sub-question predictions are accurate (in \"Oracle / Self-Answer\")?\n\nRegarding text:\n\n- Probably better to separate or re-group the rows in Table 1 into different experiments for clarity. I also observe that the improvement gained from using the image in VQAv2 diminishes slightly when using a decomposition and this should probably be mentioned (regarding textual biases of decompositions).\n\n- Would the authors consider rewriting L113-115 more clearly?\n\n- Regarding L307-309, aren't there works that indeed do multi-step question decomposition (like [39])?\n\nLIMITATIONS:\n- This paper formulates decomposition with multiple sub-questions, but only tests one step decomposition. I think saving multiple steps for future work is fine, although the text is slightly misleading. Can just be more up-front that this paper is focusing on one step. But why didn't the authors at least test >1 step human-annotated decompositions from Selvaraju et al. [29] for VQAv2? Seems like low-hanging fruit.\n\n- We should be more clear that there may be more than one good decomposition for a visual question.\n\n- Is a decomposition really \"good\" by the proposed metric? L272-273 indicates that generated decompositions may be gibberish but still lead to improved performance. A more detailed human study is probably necessary for more clarity.\n\n- Recent works (e.g. Kamalloo 2023) indicate that existing QA (and likely VQA metrics), which are based on lexical similarity, correlate poorly with human judgements for open-ended text generations. So I believe \"goodness\" metric here (L100-101), which relies on exact match, may be unsuitable. For that reason, it might have been better to compare the log probabilities of generating the ground truth answer instead (L97-98).\n  - I also observe that this paper did not specify which A-OKVQA setting (multiple choice or direct answer) it used. It is likely the latter, but this dataset's official metric is the VQA metric (i.e. comparing the model prediction to 10 human annotations). Would the authors please clarify which answer they compared against for their exact match setup?\n\nReferences:\n- Ehsan Kamalloo, Nouha Dziri, Charles Clarke and Davood Rafiei. “Evaluating Open-Domain Question Answering in the Era of Large Language Models.” ArXiv abs/2305.06984 (2023).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}