Exploring Question Decomposition for Zero-Shot VQA
Authors: Zaid Khan
Vijay Kumar b g
Samuel Schulter
Manmohan Chandraker
Yun Fu
Venue: NeurIPS 2023 poster (2023)
Keywords: visual question answering
in-context learning
vision-language
TL;DR: Can recent instruction-tuned models improve their own predictions on 0-shot VQA tasks by breaking down the question? Yes.

Abstract: Visual question answering (VQA) has traditionally been treated as a single-step task where each question receives the same amount of effort, unlike natural human question-answering strategies. We explore a question decomposition strategy for VQA to overcome this limitation. We probe the ability of recently developed large vision-language models to use human-written decompositions and produce their own decompositions of visual questions, finding they are capable of learning both tasks from demonstrations alone.
However, we show that naive application of model-written decompositions can hurt performance.
We introduce a model-driven selective decomposition approach for second-guessing predictions and correcting errors, and validate its effectiveness on eight VQA tasks across three domains, showing consistent improvements in accuracy, including improvements of >20% on medical VQA datasets and boosting the zero-shot performance of BLIP-2 above chance on a VQA reformulation of the challenging Winoground task. Project Site: https://zaidkhan.me/decomposition-0shot-vqa/

Exploring Question Decomposition for Zero-Shot VQA
Zaid Khan'
Vijay Kumar BG² Samuel Schulter"
Manmohan Chandraker
Yun Fu²
Northeestern University NEEC Laboratories America
*UC San Diego
{khan, za,uun yun. fuJ@northeestern edu {vijay.kumar, .uumar samuel, manu) Couu-lnne com
Abstract
Visual question answering (VQA) has traditionally been treated as a single-step task where each question receives the same amount of effort, unlike natural human question-answering strategies. We explore question decomposition strategy for VQA to overcome this limitation. We probe the ability of recently developed large vision-language models to use numan-written decompositions and produce their own decompositions of visual questions, finding they are capable of lcarning both tasks from demonstrations alone. However, show that naive application of model-written decompositions can hurt performance. We introduce model-driven selective decomposition approach for second- -uessing predictions and correcting errors, and validate its effectiveness on eight VQA tasks across three domains, showing consistent improvements in accuracy, including improvements of > 20% on medical VQA datasets and boosting the zero-shot performance of BLIP-2 above chance VQA reformulation of the challenging Winoground task. Project Site: https:://zaidda me/ wwteddeooosiiiioonoeeeeeeo
Introduction
On a question-answeriing test, humans are able to answer some questions in a single step, while other questions require potential deliberation and second-guessing. Visual question answering (VQA) 3 has traditionally been treated as a single-step task. Models only get one chance for each question, and each question receives equal amounts of computation. This is incongruent to the natural human approach to such tasks, where simple perceptual questions are quickly answered, while harder reasoning questions allocated more time and computation.
The emergence of task decomposition techniques for large language models (L.LMs) a potential solution to this incongruency. Task decomposition techniques prompt L.LM to break down an initial complex task into simpler subtasks that can each be solved independently. However, VQA has not benefited from advances in task decomposition techniques for two reasons. First, many task decomposition techniques [5, have only been effective the regime of very large unimodal LLMs with parameters in the 30B+ range, while the LLMs underlying vision-language models are typically much smaller, only recently reaching # 13b parameters for publicly available models Second, existing methods for prompting vision- vision-language models (VLMss during VQA tasks focus other use cases, such as providing more examples of the input task 110 or more information about the image [1I]. Given the recent emergence of multi-billion scale VLMs, main research question is:
Can multi-billion seale vision-langee language models benefit by approaching reasoning- heavy VQA as a two-step rather than a single-step problem using decomposition?
To this end, explore form of task decomposition called question decomposition strategy for zero-shot visual question answering with large VLMs. Although question decomposition has been explored for specific unimodal QA (122 14 has not been explored strategy for multimodal
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

tasks such as VQA with emerging large VLMs learning ability of emerging large VLMs.
7 79
15
16.
and little is known about the in-context
of both LMs and VLMs to exploit oracular question decompositions written by humans. We design experiments to understand whether models can learn to use decompositions without explicit training, and whether they are merely exploiting keywords and surface statistics when they use decompositions. Second, we conduct a series of exper- iments, again using in-context learning, to understand how well models can produce decompositions that correct the errors of fixed VQA model. Last, we propose and study an entirely model-driven closed-loop approach mimicking a simplified form of classic human second- guesssing strategy: second- guesss answers based on how confident you are about them. We conduct experiments across three domains (art, natural images, medical), eight datasets, three model families, and model sizes ranging from 80M to 11B parameters. Our contributions can be listed as follows:
First, we probe the in-context learning ability 17 19
1. We experimentally demonstrate that large VLMs based
instruction-tuned LLMs can use decompositions to improve their predictions without any training, and are not merely exploiting changes in word statistics introduced by the decomposition. (Sec. B)
2. We quantitatively show that generative, instruction tunnedd language models are capable of writing effective decompositions zero-shot, without task-specific training. (Sec.
3. We find that applying decomposition naively to every question instance harms performance rather than helps (Fig. D. and propose selective decomposition (Fig. 3 a modular, model- agnostic, training- free strategy that treats VQA as two-step task. (Sec. 50
4. We apply selective decomposition to a testbed of 8 datasets and show that it consistently im- proves performance (Tabs. D and D. with gains of 20% on medical VQA datasets 200 (2). and boosts the performance of [ above chance on the Winoground 23 benchmark when formulated as a VQA task. (Sec. 5
2 Background
2.1 Problem Setting
In zero-shot VQA, a model f: f v.q is given an image v, a question 4, and outputs an answer, a. Unlike traditional VQA. the model f()) has never v. q. a triplets. In practice, such setting often occurs when f() is foundation model that contains several billion parameters and has undergone large scale pretraining. It is undesirable to retrain such an f()) on visual question answering pairs specifically, both for reasons of computational convenience and because finetuning can degrade robustness 24 The most common case is that f()) is autoregressive, generative language model that can optionally be conditioned on the visual modality. We restrict ourselves to such models, which approximate NN ((te-1l-l/ v), where is image and I1 is sequence of language tokens. In zero-shot VQA setting, it is expected that f(.) understands that it has been given a question y and should produce the correct answer
the question q in the context of the image y by modeling it as (alvv, v. 4). This setting is common when evaluating very large frozen models, such as in [T0] I1. with the exception that in our case, f()) is a vision- vision-language model rather than a language-only model.
2.2 Question Decomposition
Question decomposition is the task of decomposing complex main question into one or more simpler subquestions that are logically related to the main question, answering those simpler subquestion(s), and then using the answered subquestion(s) to help in composing final answer to the complex main question. This is a strategy often used by humans for problem solving. For example, consider a human being confronted by a wild animal they have never seen before. To answer the main question "does this animal pose a threat to me?" a human might decompose it into subquestions such as "does the animal have sharp canine teeth?" and "does the animal have forward facing eyes typical of a predator?" Knowing the answer to even one of these subquestions makes answering the main question much easier.
Adopting the terminology of Sec. 2.1 the task of question decomposition consists of decomposing a main visual question v. q into one more subquestions (51, ..). answering those subquestions to

---

Q: What country headquarters plane company?
What could keep this food from falling spart?
woman using the umbrella to protect herself from?
Q: What kind of show the an?
the second shelf in the fridge used for?
Decomposition plane company? virgin
Decomposition
Decomposition
Decompasition
Decompasition
what holding the food together? bread
what the weether like outside? sunny
what is the man wearing? a suit
what is the first the fridge used for?
atlantic
Before: Canada
Before: knife
Before: rain sun
Before: cricket
Before: food After: drink
Q: does the picture contain lungs?
Q: this an x-ray?
Q: is this a transverse section?
Q: Does adrenal show gout?
The time of day suggests the man will
Decomposition
Decomposition picture black and white? yes
Decomposition this x-ray person's brain? no, mrl
Decompasition
Decomposition
there person in the picture?
what the cause of gout? acid crystals
what is the time of
day? morning
Before: yas After: no
no yes
Before: report After:
Before: After:
After:
After: yes
no
wake up
Figure 1: Model-produced decompositions and their error correcting effects. The decompositions and be-
fore/after answers shown above produced by prompting BLIP-2 models based FLAN-T5 to produce subquestion, answering subquestion with the model and feeding the question answered subquestion back to the model: is correcting itself. Before answers are wrong and After answers are correct.
obtain the decomposition (('i, a)) (9',, ..),,.. and then using v, q together with the decomposition a() (4 ay),,))..) ... to obtain the final answer
2.3 What makes good subquestion?
In Sec. gave definition of decompositions that is dependent notions of "simpler" and "logically related". It challenging to make these notions precise, and difficult to operationalize them to measure whether sequence of text really is valid subquestion according to these notions. To sidestep these difficulties, we adopt a consequentialist view of whether a subquestion is "good", following common consequentalist tradition in artificial intelligence subquestion by measuring the effect of the subquestion. Concretely, let v, q, a be visual question triplet where v is the image, q is the question, and a is the answer. Let (a(alv q) be the probability of the ground-trute answer a as assessed by visual question answering system f(-). We regard decomposition of v. consisting of series of subquestions and their answers ((q4,- d)), (q'1. d²)) ...) as "good" if p (alv, q) p (alr, ((44, a)), (q'2 aτ) ..)...)) .)), that is, if seeing the decomposition increases the probability of the ground-trun -ruth answer a. In practice, we adopt a simpler criterion that takes the consequentalist definition to the limit. We regard decomposition as "good" if seeing the decomposition induces the model to produce the true ground-truc truth answer a.
whole 25)). We evaluate
the "goodness"
2.4 Scope & Limitations
We only consider in-context learning techniques for zero-shot VQA, and do not explore full model training in this work. The class of model we are interested in are cssructtio-followinng vision-language models based on large language models 7- This excludes nevious-generatiin vision-language models that are not based on multi-billion parameter instruction-tuned language models 30)
3

---

Image Text (3B)
Image+Texi (13B)
Decomposition
Overall
Boolean
Number
Other
Overall
Booleun
Number
Other
None (Baseline)
82.5
6.8
67.4
79.1
81.9
13.7
Oracle/Oracle
91.4
40.2
79.4
89.8
92.6
45.3
80.4
A w.r.t Baseline
9.6
8.8
33.3
12.1
10.8
10.7
31.6
10.3
Oracle/Sef-AAswer
87.3
21.4
72.8
83.9
87.1
26.5
73.2
& w.r.t Baseline
5
4.8
14.5
5.4
4.8
5.2
12.8
3.1
Oracle/No Answer
83.3
85.9
27.4
74.9
84.1
86.9
27.4
75.2
A w.r. Baseline
4.4
3.4
20.5
7.6
5.1
5
13.7
5.1
Oracle/Oracle (Scrambled)
84.9
87.9
37.6
74.8
86
88.9
39.3
76.2
A w.r. Baseline
5.9
5.4
30.8
7.4
6.9
7
25.6
6
Text (3B)
Text (13B)
Decomposition
Overall
Boolean
Number
Other
Overall
Boolean
Number
Other
None (Baseline)
57.4
64.4
32.2
63.8
71.9
6.8
34.3
Oracle/Oracle
72
75.8
37.6
58.4
81.5
85.1
45.3
69
A w.r. Baseline
14.5
11.4
31.6
26.2
17.8
13.2
38.5
34.7
Oracle/Sel-Answer
62.1
65.8
23.1
48.8
68
72.1
20.5
53.7
A w.r.t Baseline
4.6
1.4
17.1
16.7
4.3
0.2
13.7
19.4
Oracle/No Answer
64.8
68.7
21.4
50.9
75.2
79
26.5
62.2
A w.r. Baseline
7.3
4.3
15.4
18.8
11.4
7.1
19.7
27.9
Oracle/Oracle (Scrambled)
60.5
62.6
28.2
53.3
78.9
83.1
40.2
63.5
A W.E. Baseline
3.1
-1.8
22.2
21.1
15.1
11.3
33.3
29.2
Table 1: Models capahle of using decompositions written by humans provide accurate answers. The gray rows are baseline performance with no decomposition, Oracle/Oracle rows denoting oracle subquestions/oracl answers, have the highest A. "Self-Answer" mcans the model answered oracular subquestions itself. and "No Answer" indicates the answer was left out entirely. Image+ Text indicates vision-language model (BLIP-2) tested with multimodal inputs, while Text indicates the corresponding language model inside BLIP-2 (FLAN-TS) was tested with text only inputs. Validation split VQA Introspect is the datasct (22k reasoning questions with their associated decompositions).
each is calculated w.r.t to this baseline.
Not all datasets are suitable for exploring question decomposition, some primarily test low-level perception skills rather than high-level reasoning skills that would benefit from a decomposition. We thus limit our evaluation datasets that explicitly test for high-level reasoning/ knowledge-baat ability. We are few-shot for the task of visual question decomposition but zero-shot for the task of visual question answering.
How well can models use decompositions?
Our goal in this section is to understand the ability of vision-language models based on large language models to consume decompositions. The hypothesis test is: When provided with gold-standard decompositions on a VQA task, a model'ss error rate should be lower than without the gold-standard decompositions. Evaluating this hypothesis presents a number of challenges. First, how can we obtain a set of decompositions that are apriori "known to be good"? Second, how should the model be fed the decompositions?
To find a source of apriori "good" decompositions, we turn to the literature on internal consistency in visual question answering. To probe consistency in question answering systems, several datasets [31-
[K] have been proposed. particularly relevant case of such dataset is VQA-Introspect
32 which
32 annotate each question in the VQAv2[II validation set as a high-level "reasoning" question or a low-level "perception" question. For each "reasoning" question, Selvaraju et al. [32] write 1-3 "perceptual" subquestions which implied by the reasoning question. For example, given high-level reasoning question such as "Can I this banana?" model that says "yes" should also reply "yellow" the low-level perception question "what is the color of the banana?" We propose to use the low-level perception questions and answers written for the high-level reasoning questions as an oracular decomposition for high-level reasoning question, on the basis that the low-level perception questions are simpler than the ligellevel reasoning question, entail the answer for the high-level reasoning question, and are written by humans.
probes consistency along a
perceettionnn axis. Selvaraju et

---

The second challenge lies in using decomposition consisting of a series of subquestions and answers ((gj,a) (42. a²) ...) alongside a main visual question (v. Recall that we cannot train the model f((-) being used for the visual question answering task, and for any arbitrary model, it is oracular subquestion and human-written answer q', a' for the main question v, 9 extracted from VQA-Introspect. The prompt is simply
unknown whether the model has ever seen the exact task of decomposition-aiided visual question answering. Thus, we rely on the in-context learning ability [8] [9] of large language models to learn to perform the tasks we require from a demonstration of the task. We handcraft a simple prompt to contain main visual question v.q from the VQAv2 validation set, along with one human-writte written
exemplar ar = "Context: is the sky blue? no. are there clouds in the sky? yes. Question: what weather is likely? Short answer: rain" prompt exemplar + "Context: (subquestion)? (subanswer). Question: {
question)? Short answer:"
Experiments & Discussion We use BLIP-2 mmodels based on the instruction-tunee FLAN-T5 34 in 3B and 13B sizes. Experiments are run combination of A6000s and TPUv3s, the VQA- Introspect validation set containing 22K reasoning questions and their associated decompositions. The results are shown in Tab. D Compared to the baseline with no oracular decompositions, both the 3B/13B vision-language models and their corresponding language models show clear ability to benefit from decompositions across variety of question types, with numerical questions benefiting the most. Next, we seek to gain insight into the mechanism by which decompositions aid inference. Is the model merely exploiting changes in surface level statistics? If so, we would expect that perturbations that leave the statistics largely unchanged but sigmificantly alter the meaning and logical structure of the oracle decomposition should not result in significantly different performance from the unaltered oracle decompositions. We remove the answers from the decomposition so that it only contains the subquestions, and test the effect of only using the subquestions. Compared to the oracle, there is a significant 50% relative decrease in improvement w.r.t to the baseline. Most of the subquestion answers are boolean, so removing them should not significantly change content words in the prompt, though it changes the meaning of the context significantly. Next, we allow the models to answer the subquestions themselves (Oracle/Self- Answer))) rather than using the ground-truth questions. The accuracy of all models again decreases relative to the oracle answers, suggesting the answer and question together contribute to the result. Finally, we take the oracle subquestion+ansser and scramble the words before providing them to the models. If the model is merely exploiting surface level statistics, the performance difference between the scrambled oracular decompositions and the original decompositions should be minimal, as the words are all the same. Again, we observe a significant drop compared to the original decompositions, suggesting that the models are not merely exploiting changes in the surface level statisties. Furthermore, human- written decompositions help in almost all cases over the decomposiitoon baseline. Note: See supplement for complete experimental details for all experiments.
4 Can models produce effective decompositions?
In this section, we conduct experiments answer the following research questions:
1. Can language models & 13B parameters learn to produce effective decompositions purely through demonstrations?
2. Is question dccomposition mostly a linguistic ability, or is being able to see the image important?
Recall that a decomposition of visual question v.q is series of one or more subquestions ((9).a)). (g/.a') ...) and their answers, with the constraint that the subquestions and answers should have the property that Py (alv.4) < (alavv q, ((g), (4). a5)...) where (-(-) repre- sents probability assessed by a given vision-language model f()) of the ground-trum -tuth answer a. We simplify this task to the task of producing single subquestion q' given a main visual question v. v.4, and denote the process of decomposition with an arbitrary autoregressive language model g(-) as dg (v. - q'. We hereafter refer to the model g(-) that generates the decomposition as the decomposer. The subquestion then answered by the vision-language model ((v,g') = a' to produce the subquestion-ansee answer pair (q', aa).. We call the question answering model the recomposer.
5

---

english
Context: what language are the signs written in? English
<exemplor demonstrations>
Reasoning Question: this town in China? Perception Question:
anseer subquestion
Question: is this town in China? Short Answer:
Recomposer
onswwer malt qusstiot aided by decompostioe
Recomposer
what language are the signs written
Decomposer
(opeionatj
genterstee sebqueation
no
Model Generated Decomposition
Recomposition
Figure 2:
The procedure we to generate decomposition and use it additional guidance during zero-shot VQA. The recomposer can be any question answering model, and the decomposer can be any generative language model, and some models can perform roles, leading to self-talk. experiments. we test various decomposer candidates can learn to write effective subquestions purely from seeing a demonstration of the task.
B-OKVQAB5
AntVA 36
OK-VQA
SLAKE 201
VQA Model
Decomposer
Fyc J
Fr
ECR
Fjc
Err
Fex
Eyc
Err
FCR T
Eyc
Fm
Params
63.56
66.73
Text
10.42
53.08
9.56
59.81
9:45
52.47
12.15
49.29
250.0M
(30)
Text
7,99
30.76
15.11
12,22
6.06
21.03
41.12
8.64
7.95
29.58
15.01
15.25
16.38
36.83
37.68
780.0M
3.0B
Image+Text
7.81
10.9
4.36
13,08
7.42
12:29
15.96
28,9
3.0B
8.05
Text
11.52
33.44
46.99
42.52
83.15
11.3
34.85
60.31
19.12
30.42
77.38
80.0M
Text
8.92
60.63
9.94
49.07
9.6
58.16
18.15
48.75
250.0M
BLIP2 (1IB)
36.57
40.65
35.5
Text
10.78
20.59
8.33
19.63
9.73
15.43
19.85
35.42
3.0B
Image+Test
26.36
13.54
20.71
30.42
11:08
Text
12,45
30.64
8.05
28.97
12.42
27.11
18.51
32.5
11.08
Table 2: Models of drastically different sizes and multimodal capability produce effective subquestions, as
measured by BeR Eq. ([). their ability to correct errors a VQA Model. However, subquestions produced by larger models are less likely mislead the consuming VQA model, as measured by Eyc in Eq, 2 "Text" indicates a language-only decomposer, while "Image+Texi" indicates a vision-language decomposer. "Params" refers to the parameters of the decomposer. A pink highlight indicates when the decomposer and vqa model are the same (the model talking itself).
We then measure the effectiveness of the decomposition by measuring the error correction rate:
ECR Σ [[((v..)) qi) az A f(v.. 4i. (4/, = a;l
(1)
Σ ([ qi) # a;
where (vi, qt, ai) represent the i-th image, question, and ground-truth answer respectively, and q..af represent subquestion generated by the decomposer model and the answer predicted for the subquestion by the recomposer (VQA) model, and [cond] an indicator function that is equal to I when cond is true and 0 otherwise. Simply put, ECR measures the number of instances initially predicted wrong answer, but switched to the correct answer after seeing the decomposition generated by g()). Alternatively, this can be understood as the effectiveness of decomposer model at correcting the errors of the recomposer model. The error induction rate Erc is the opposite:
which f(-)
ΣN 1[f(v; qi) = a; f(v. qis (q,aa)) ) arl
Ejc₂ =
NN 1[f qi) = a;]
and measures how often the produced decompositions flipped an answer that was initially correct to an incorrect answer. The decomposer be the same the recomposer if the model can do both tasks by following different prompts, as in the case of instruction -uuneednontnedddddee models [8)
Experiments & Discussion We use BLIP-2 2 7 based on the FLAN-T5
the question answering
model (recomposer). For the decomposers, we use FLAN-TS 32 models ranging in size from 80M parameters to 1 IB parameters, as well as the BLIP-2 models themselves. We use four VQA

---

(medical), and A-OKVQA 35 and OKVQA [37 (external knowledge VQA on natural images). We then carry out the procedure illustrated in Fig. 2 for each combination of decomposer, recomposer, and dataset. We handcraft three demonstrations of writing a subquestion for question, in the form "Reasoning Question: <question>? ? Perception Question: <subquestion>? ?" For each 4 pair in dataset, we prompt the decomposer with the demonstration, followed by the question as in Fig. 2 and measure ECR as in Eq. ① and Eic as in Eq. 2 for each dataset. We show the results in Tab. 2 We find that yes, language models & 13B parameters can learn to produce effective decompositions just by viewing examples. Decomposer size correlates positively with ECR (R² 0.344), and negatively with Erc (R² = .273) and the correlations are sigmificant at a 0.05 across a larger collection of eight datasets used in Sec. A human examination of the "subquestions" produced by smaller models shows that many of them are gibberish and not properly formed questions at all. Despite this, they surprisingly manage to maintain an ECR that is sometimes higher than larger models. Finally, the ability to decompose questions in the evaluated datasets may be a primarily linguistic ability that is possible ask effective subquestions about image without being able to see the image, and the difference in effectiveness between the Image+Text BLIP-2 models and the text-only FLAN-TS models of similar size is on average 10% of the base error rate (but this may not be true of other VQA datasets).
datasets from three domains: A 36 (art), SLAKE 20
5 Selective Decomposition Works Better Than Naive Decomposition
One problem shows up in Sec. + which is that applying decompostions to every question can hurt performance, by flipping answers that were initially correct to be incorrect. If we were able to decompose only wrong answers, we would always see a net gain performance due to the error conrection of decompositions. However, in realistic setting, we do not know apriori that our answers are wrong, and thus run the risk of flipping an answer that was initially correct to be incorrect by applying decomposition that is misleading. We call this the second- second-guessing problem.
To deal with the second -roond-ggsisingggg problem, we propose following an intuitive human strategy: stick with your ini- tial answer on questions you are confident about, and only second- recond-guess (apply a decomposition) for questions you are not confident about. Language models can be surpris- ingly well calibrated [39] meaning that the probability they assess to an output sequence they produce often well- correlated with the probability that the produced output sequence is the "correct" one for given task. We make use of this property to treat visual question answering as selective prediction 40 task, using the language models's s confidence
Image q: Question
т: Confidence Threshold
Attempt an initial answer.
å, p()) = recomposer (v, 4)
Selectively decompose q answer à is uncertain.
if p(ā) <= T:
q' = decomposer (v,4)
a' recomposer (v, q²)
Answer again.
decision score to determine whether should apply a decomposition to a instance or stick with the original answer.
recomposer (v, 4 q'.q.''.a'' ,'"""))
Figure 3: Pseudocode for selective decom- position.
We describe the algorithm pseudocode in Fig. 3 The selective decomposition procedure transforms VQA from a single-step task to a two-step task given a decomposer model, a recomposer model, a confidence threshold т, and visual question pair q. An initial answer à and confidence p(ā) is solicited from the recomposer model. and answer pair (q', aa) ) are generated by the decomposer and recomposer working together. The recomposer model is then allowed to "second -ueess the inital answer with the decomposition (q',a') aa) as additional context. The decomposer and recomposer can be the same model or different models. We experiment with both scenarios. This introduces an extra hyperparameter T into the inference procedure.
p(ā) < r, the decomposition procedure is invoked, and a subquestion
Experiments & Discussion In Fig. we show the effect of different values of r on the accuracy of selective decomposition with several decomposers. Across all datasets and all models, there is a wide range of (expressed as percentiles) for which selective decomposition improves predictive accuracy. At the same time, we clearly demonstrate the second guessing problem in Fig. Decomposing every question often eventually leads to lower accuracy than decomposing no questions at all, because hal- lucinations and misleading decompositions can flip an initially correct answer to an incorrect answer.
7

---

SLIPS 1301 INORRVIO
DUP2 AANNOA
DUPD DIOOD
SUPP 1381 SLAKE
DUP2 (28) No0-Mtrospect
8
80.3
coa
large
Accuvey N
95.3-
deCOTT pOo pareTHtars
12.5
706830568
7660226622
27.44
13699006000
Cerfocecc Meronnde
Cardcer Poroedd
Co G5393 Porcertle
Confidencc Porcentlo
Selective decomposition mitigates the problem of misleading decompositions. We decompose questions based on model confidence in the initial answer, and show how accuracy initial rises past the baseline as the model mostly second guesses wrong answers, and then drops below the haseline with no decompositions (horizontal line) if too many questions initially answered correctly are second- wronddguessed...
Figure 4:
In Tabs. and we show the highest possible net gain achieved by selective question decomposition on three do- mains by different decomposers. Selective decomposition consistently improves predictive accuracy regardless of the decomposer and domain. Net gains are larger on datasets (passes t-test with 0.05) containing non-naturalistic images and specialized domains (e.g. medical) than they are on domains containing natural images. The mean op- timal surprisal I(T) for second-gun -oooondggessinggg answers is lower for non-natural domains (1)((r = 13.2 for the natural im- age datasets vs (I(+)) = 9.0 for the medical and art datasets, confirmed by t-test at 0.5). We further visualize this in This matches our expectations: you should second domains you understand poorly more than on domains you understand well. linear regression fit shows that larger decomposers correlate with larger net gains =R2=0,365, R²=0.342 for natural image domains and medical art domains respectively, t-test with a=0.05).
domain
natural other
20
Gain
10-
10
20
30
log2 (+)
Fig. 5
guess yourself
Figure 5: Decomposition is more effective on non-natural image domains, and models are also less confident in these domains. Size of circles is proportional to parameter count.
We reformulate Winoground as a VQA task by turning each caption into a question with a boolean yes / no answer (does "<caption>" describe the image?) on which chance accuracy is 50%. As visible in Tab. 3 all BLIP-2 models perform below random chance, in agreement with previous results on Winoground showing that it is extremely difficult for vision-language models. Surprisingly, after models (80M/200M parameters),
decompositions produced by the relatively TLLAN-TS-small/ee the performance of BLIP-2 (13B) rises significantly above chance (+ (+18%). Upon inspection, many of the decompositions produced by the model appear to be gibberish, yet remarkably, induce the much larger 13B BLIP-2 model correct over 30% of its initially wrong answers.
6 Literature Review
Task decomposition 5 6
improves the performance large language models on zero-shot reasoning tasks. The only work so far to apply similar techniques for VQA is MM-CoT [3], but does not explore task decomposition with large vision-language models, choosing to finetune smaller model instead. The ability to use zero-shot task decompositions may be property of model scale, emerging 60-200B parameters 44 or may be property of large-scale pretraining on code 45 Such large Dision-language models have only been developed recently due to advances in Vision-language alignment. The prevailing paradigm vision-language pretraining was to build lanione models atop (relatively) small language models 226-29 46 below 1B parameters. Meanwhile, language models were being scaled from 3B-175B parameters
50 with each model family having at least one representative with > 10B parameters, Because vision-language
8

---

Decompuser
AokVQA
OLVQA
VQA Introspec
Winoground
VQA
Type
Purams
Acc
%
T(r)
Acc
%.1
f(+)
Acc
%
((T)
Ape
%
(()
3.23
250M
0.44
11
17.58
0.46
13
18,05
3.1
7
13.24
22
99
0.19
780M
0.87
18
12.42
26
11.02
3.18
8
10.64
8.59
3B
3B
49.69
1.14
31
8.13
36.54
0.97
23
12,09
78.64
3.5
5.87
45.81
8.59
I+T
38
1.22
33
7.71
1.29
27
10.7
3,75
20
5.24
0
8,59
7B
3.37
IIB
0.7
11
17.58
1.03
22
12.47
3.6
14
6.27
0
8.59
SOM
3,46
0.22
250M
0.35
22.96
0.81
11
22.13
3.31
7
15.81
22.12
99
0.09
THOM
3B
53.36
0.52
1,05
14
14
17.93
17.93
1.15
18
15.68
15.68
3.73
3.46
8
11.81
11,81
0.06
92
3,83
0.29
T
7B
1.48
18
14.91
1.37
24
12.54
3.69
8
11.61
4.06
99
0.09
10.95
3.68
1.64
1+T BIB
1,92
25
10.95
1.68
24
12.54
3.81
11.81
5.49
Table 3: Increases in accuracy produced selective decomposition at the optimal second-guessing confidence
threshold on external knowledge QA and visual reasoning across several decompasers. I(T) log2(#) the surprisal of z. and is the percent of the questions in the dataset above I(r), or equivalently, below T=FLAN-TS, ++==BIP-22 (based on FLAN-T5) and Tnd -Calaciica
Decomposer
ArtVQA L5
PathVQA
SLAKE X0
VQA Rad
21
VQA
Type
7
Acc
4
[(T)
7
[(r)
T
80M
2.36
50
10.37
15,89
98
2.42
3.86
24
22.12
18.73
97
2.88
250M
2.68
10.37
14.79
16.57
780M
5.28
8.91
12.61
99
1.91
4.15
25
19.65
15.66
16.57
3B
3B
17.17
2,44
10.37
12.45
11.56
98
2.42
33.27
3.96
38
7.33
11.7
18.82
5.76
2.13
69
1.6
T*
7B
2.91
30
17:47
13.9
99
1.91
6.13
50
5,71
19.62
94
4,44
IIB
3.15
8.91
14,08
98
2.42
5.18
31
10.83
19.17
1.6
T
80M
4.33
63
12.09
23.4
99
1.39
8.95
45
14.45
23,89
99
2.95
250M
4.41
19.4
18.46
99
1.39
7.54
39
19.48
23.67
2.95
780M
5.91
6.67
22.62
99
1,39
6.05
41
17.08
24.02
2.95
13B
1B
16.85
4.57
37
20.79
5,56
21.33
99
1.39
22.62
8.67
44
14.8
5.12
23.84
99
2.95
5.12
3.94
20.05
21.56
99
1.39
9.52
44
14.8
26.25
2.95
1+T
11B
5.98
6.35
20.85
99
1,39
10.37
42
16.08
25.13
2.95
Table 4: Increases in accuracy produced by selective decomposition at the optimal second -uoossing confidence
threshold r, across two domains (medical/art) and several decomposers, I(T) = log2( the surprisal of and 77 is the percent of the questions in dataset above ((r), or equivalently, below т. T-FLAN-TS, I+T=BIP--2 (based on FLAN-T5) and T =alaatcti,,..
pretraining typically requires full model training, aligning these multi-billion parameter models the visual modality was prohibitively computationally expensive. However, recent discoveries [5]] 52 motivated by earlier work with frozen models (53] have shown that the representation spaces of vision models and large-languap models are surprisingly close, and rough alignment can be achieved with adapters [S]] or linear mapping layers while keeping the language model frozen, and more advanced techniques have given rise to vision-LLMs [7)) Our work is closely related to the visual question generation paradigm
I 54 [55] However, direct our question generation focus on decompositions rather than general questions.
Conclusion
We show that question decomposition is already a viable strategy that can be used to implement a more natural approach to VQA. Without any training, instruction- -uune VLMs learn to produce and use decompositions from a few demonstrations of the task. This approach has many possible future directions. For example, we only consider two-step approaches for visual question answering, where we "hardcode" the depth of the decomposition. A natural next step would be to extend the two-step approach to multi-step approach, which remains unexplored for large vision-language models in an open-world visual question answering setting. Second, in-context learning has limitations. Would models benefit from being trained to produce and consume decompositions?
References
[1] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering.

---

International Journal of Computer Vision, 127:398 414, 2016.
[2] Zaid Khan, B Vijaykumar, Samuel Schulter, Xiang Yu, Yun Raymond Fu, and Manmohan Chandraker. Q: How to specialize large vision-language models to data-scarce vqa tasks? Self-train on unlabeled images! In CVPR, 2023.
[3] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision, 123:4 - 31, 2015.
watg/Corpuslii :3180442 3180429
[4] Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey. ArXiv, abs/2302.07842, 2023.
[5] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
[6] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
[7] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22 Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023.
[8] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023.
[9] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, abs/2304.10592, 2023.
[10] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-basse based vqa. In AAAI, 2021.
[11] Jiaxian Guo, Jumnan Li. Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images textual prompts: Zero-shot visual question answering with frozen large language models. In CVPR, 2023.
[12] Ethan Perez, Patrick Lewis, Wen tau Yih, Kyunghyun Cho, and Douwe Kiela. Unsupervised question decomposition for question answering. In EMNLP, 2020.
[13] Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Text modular networks: Learning to decompose tasks in the language of existing models. NAACL- HLT, 2021.
[14] Pruthvi H. Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. Is question decomposition unit all we need? In EMNLP, 2022.
[15] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA multimodal augmentation of generative models through adapter- based finetuning. In EMNLP Findings, 2022.
[16] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In ICML, 2023.
[17] Stephanic Chan, Adam Santoro, Andrew K. Lampinen, Jane Wang, Aaditya Singh, Pierre H. Richemond, James L. McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. In NewrIPS, 2022.
[18] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In EMNLP, 2022.
10

---

[19] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In ICLR, 2022.
[20] Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Fang Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhaneed dataset for medical visual question answering. In ISBI, 2021.
[21] Jason J. Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman, A dataset of clinically generated visual questions and answers about radiology images. Scientific Data, 5, 2018.
[22] Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
[23] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022.
[24] Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In CVPR, 2022.
[25] Stewart Russel and Peter Norvig, Artificial Intelligence: A Modern Approach, chapter 2, page 39, Pearson, 4th edition, 2021.
[26] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In ICML, 2022.
[27] Zaid Khan, B Vijaykumar, Xiang Yu, Samuel Schulter, Manmohan Chandraker, and Yun Ray- mond Fu. Single-stream multi-level alignment for vision-language pretraining. In ECCV, 2022.
[28] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, 2021.
[29] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi- modal tasks. ArXiv, abs/2206.08816, 2022.
[30] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai. Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequene learning framework. ICML, 2022.
[31] Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. Vqa-lol: Visual question answering under the lens of logic. In ECCV, 2020.
[32] R. R. Selvaraju, Tendulkar, D. Parikh, E. Horvitz, M. Tulio Ribeiro, B. Nushi, and Kamar. Squinting at vqa models: Introspecting vqa models with sub-questions. In CVPR, 2020.
[33] Meet Shah, Xinlei Chen, Mareus Rohrbach, and Devi Parikh. Cycle-consistency for robust visual question answering. In CVPR, 2019.
[34] Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instructionfineturee language models. ArXiv, abs/2210.11416 2022.
[35] Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, 2022.
11

---

[36] Noa Garcia, Chentao Ye, Zihua Liu, Qingtao Hu, Mayu Otani, Chenhui Chu, Yuta Nakashima, and Teruko Mitamura. dataset and baselines for visual question answering on In ECCV Workshops, 2020.
[37] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
[38] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
[39] Saurav Kadavath, Tom Conerly, Amanda Askell, T. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort. Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kermion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan, Language models (mostly) know what they know. ArXiv, (ts//2207.05221) 2022.
[40] Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph E. Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In ECCV, 2022.
[41] Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Huai hsin Chi. Least-to-most prompting enables complex reasoning in large language models. In ICLR, 2023.
[42] Tushar Khot, H. Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In ICLR, 2023.
[43] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thooee thouhtt reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
[44] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph. Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022.
[45] Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1384-1403, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.emnlp-mane m0in.90. URL htttps://acaatthaeeeee oog.go...ooooooooooooo..ooo..... 2022. mmmlp-main. 90
[46] Zaid Khan, Vijay Kumar BG, Samuel Schulter, Xiang Yu, Yun Fu, and Manmohan Chandraker. Q: How to specialize large lasion-language models to data-scarce vqa tasks? a: Self-train on unlabeled images! In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15005-15015, June 2023.
[47] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony S. Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArXiv, abs/221 1.09085, 2022.
[48] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. ArXiv, abs/2205 200.22255000... 01068, 2022.
12

---

[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume I Lample. Llama: Open and efficient foundation language models. ArXiv, hiss23322.13971, 2023.
[50]
Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng. Xiao Xia. Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, P. Zhang, Yuxiao Dong, and Jie Tang. Glm- 130b: An open bilingual pre-trained model. ArXiv, 10.02220,,2244,, 2022.
[51] Zaid Khan and Yun Fu. Contrastive alignment of vision to language through parameter-effiieen transfer learning. In ICLR, 2023.
[52] Jack Merullo, Louis Castricato, Carsten Eickhoff, and Elizabeth- Pavlick. Linearly mapping from image text space, ICLR, 2023.
[53] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In NeurIPS, 2021.
[54] Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. Weaqa: Weak supervision via captions for visual question answering. In ACL Findings, 2020. URL https://api semanticscholar
[55] Soravit Changpinyo, Doron Kukliansky, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for vqa are image captions. In NAACL, 2022. URL. https
/
//api.semaaaee sew.mmmicccccccola.r....... org.ooorr//D: 248512968
[56] Dongxu Li, Junnan Li, Hung Le, Guangsen Wang. Silvio Savarese, and Steven C. H. Hoi. Lavis:
A library for language-vision intelligence, 2022.
[57] Thomas Wolf, Lysandre Debut, Victor Sanh. Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers:
ofateoof-hhe-nt art natural language processing. ArXiv, abs/ abs/1910.03771, 2019.
[58] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions, 2023.
[59] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[60] Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu, Riprompt: Optimizing discrete text prompts with reinforce- ment learning. EMNLP, 2022.
[61] Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01339, 2023.
[62] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[63] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon I.L:: outperforming curated corpora with web data, and web data only. arXiv preprint arXiw:2306.011 2023. URL (ttpstts:: : /// //arxivv...///////////aa//aaaavr www.ii.wgg/.....aa.o.....a.owott abs httgg//bs2330.000 2306 01116
13


Reviews:
[{"review_id": "d7hStsYvzU", "reviewer": "Reviewer_yYni", "summary": "This paper presents an investigation of a 1-step question decomposition strategy for VQA.\n\nAn example of a decomposition is the sub-question \"What is the name of the plane company?\" for the visual question \"What country headquarters this plane company?\" (Figure 1).\n\nThis paper finds:\n- Including human-annotated decompositions in a vision-language model (VLM) prompt leads to improved performance on VQAv2.\n- LMs/VLMs can generate decompositions that also lead to improved performances on four diverse VQA datasets (A-OKVQA, ArtVQA, OK-VQA, and SLAKE).\n- A \"selective decomposition\" strategy improves performance (i.e. where a model only generates decompositions when it originally produces a low confidence answer), as it should avoid decomposing \"harder to decompose\" questions.", "questions": "- How many in-context examples are provided to the model? Have the authors experimented with different numbers of examples? Are these examples possibly distractors when provided to VQA models, as they are irrelevant to the provided image?\n\n- Do the authors know what % of sub-question predictions are accurate (in \"Oracle / Self-Answer\")?\n\nRegarding text:\n\n- Probably better to separate or re-group the rows in Table 1 into different experiments for clarity. I also observe that the improvement gained from using the image in VQAv2 diminishes slightly when using a decomposition and this should probably be mentioned (regarding textual biases of decompositions).\n\n- Would the authors consider rewriting L113-115 more clearly?\n\n- Regarding L307-309, aren't there works that indeed do multi-step question decomposition (like [39])?", "limitations": "- This paper formulates decomposition with multiple sub-questions, but only tests one step decomposition. I think saving multiple steps for future work is fine, although the text is slightly misleading. Can just be more up-front that this paper is focusing on one step. But why didn't the authors at least test >1 step human-annotated decompositions from Selvaraju et al. [29] for VQAv2? Seems like low-hanging fruit.\n\n- We should be more clear that there may be more than one good decomposition for a visual question.\n\n- Is a decomposition really \"good\" by the proposed metric? L272-273 indicates that generated decompositions may be gibberish but still lead to improved performance. A more detailed human study is probably necessary for more clarity.\n\n- Recent works (e.g. Kamalloo 2023) indicate that existing QA (and likely VQA metrics), which are based on lexical similarity, correlate poorly with human judgements for open-ended text generations. So I believe \"goodness\" metric here (L100-101), which relies on exact match, may be unsuitable. For that reason, it might have been better to compare the log probabilities of generating the ground truth answer instead (L97-98).\n  - I also observe that this paper did not specify which A-OKVQA setting (multiple choice or direct answer) it used. It is likely the latter, but this dataset's official metric is the VQA metric (i.e. comparing the model prediction to 10 human annotations). Would the authors please clarify which answer they compared against for their exact match setup?\n\nReferences:\n- Ehsan Kamalloo, Nouha Dziri, Charles Clarke and Davood Rafiei. “Evaluating Open-Domain Question Answering in the Era of Large Language Models.” ArXiv abs/2305.06984 (2023).", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents several experiments for motivating and investigating the presented method. I believe it is a moderately important research contribution and is a sufficient package for publication.\n\n- The concept of question decomposition (and its use in prompting to improve performance in compositional question answering) is inspired from works in NLP using language models. However, question decomposition has not yet been investigated for visual QA tasks. Prior to this work, it was an open question whether LM/VLMs could decompose *visual* questions or make use of visual question decompositions.\n\n- The paper is generally well written, answers its research questions or demonstrates clear improvements for each experiment, and benchmarks on a diverse set of VQA datasets.", "weaknesses": "- This paper should have also measured the decomposer method (Table 2) on VQAv2. Then, we could have compared with the results from using human-annotated decompositions in Table 1.\n\n- I appreciate that this paper measures text-only biases in their experiments. But I think it is also important to particularly measure the textual biases of *only* generating decomposed questions or the final VQA answer.\n\n- It would have been nice to also see ablations from a more powerful (e.g. 100B-param, code-pretrained) language model to observe an higher bound for the text-only experiments in this paper.\n\nI feel that this paper needs to be a little more careful with some claims:\n\n- It is hypothesized that \"zero-shot task decomposition techniques is likely a property of model scaling\" (L281-282) and this directly informs research question 1 in Sec. 4. However, the cited paper [42] claims that techniques similar to this (e.g. chain of thought, least-to-most prompting) emerge at 60-200B parameters. This work compares models with 11B or fewer parameters. As the decomposition is limited to one step in this work, smaller models (e.g. 3B or 11B size) may plausibly be capable of this task, but the details are not yet clear. Finally, while this is one hypothesis, another line of work suggests that the capability comes (not from model scale but) from pre-training with code (Fu 2022).\n- Technically, the method in this paper is few-shot, since in-context examples are provided (L139).\n- I would adjust the following claim (L133-134): \"It is unlikely the model has ever seen the exact task of decomposition-aided visual question answering\". It is hard to make claims about the specific data that LMs are trained on.\n- I am not confident in the result from the \"Oracle (Scrambled)\" experiment (Table 2, Sec. 3). Do we know that the model is simply not able to override its priors and handle scrambled text effectively after just a few in-context examples? Is this decomposition \"primitive\" or can it be re-scrambled into multiple compositions? I think it's fine to exclude this experiment, as the text-only ablations are likely sufficient.\n- \"Decomposition appears to be a primarily linguistic ability\" (L207) is not necessarily generally true. This has only been shown on this particular set of datasets and perhaps this is an observation of the qualities of existing VQA datasets.\n\nAll being said, I believe adjusting these claims in-text will be sufficient and, although my concerns are detailed, I do not believe these entail any serious methodological issues.\n\nReferences:\n- Yao Fu, Hao Peng and Tushar Khot. (Dec 2022). How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources. Yao Fu’s Notion. https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1\n  - (Maybe there is a printed citation for this, but my point is just that there is an alternate hypothesis, as illustrated here.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review! We have done our best to respond to everything. Please note the user study.\n\n- We do measure the decomposer method on VQAv2 in the supplement (Table 6). This table also includes $E_{IC}$ and $E_{CR}$ for the human-annotated decompositions.\n\n- Thank you for bringing the hypothesis of (Fu 2022) to our attention. We have added to our literature review and will adjust the text (L281-282) to the following:\n\n\"The ability to use zero-shot task decomposition may be a property of model scale, emerging at 60-200B parameters [1], or may be a property of large-scale pretraining on code [2].\"\n\n- [1] Emergent abilities of large language models,\n- [2] Language Models of Code are Few-Shot Commonsense Learners\n\nTo clarify our motivation: we want to understand whether \"smaller\" LLMs (<=11B) are capable of a useful level of task decomposition, even if they are limited compared to their larger counterparts. \n\n- Is the setting technically few-shot?\n\nWe will modify 2.1 (Problem Setting) to explicitly state that we are few-shot for the task of *producing a decomposition*, but zero-shot for task of *visual question answering*. We do provide a single exemplar for VQA, but the exemplar is manually written, static for each dataset, and does not belong to any of the VQA datasets being used. Our setup is very different than those used in \"few-shot\" VQA, such as PIQA [1] and PromptCap [2], which use 16-32 real questions and answers dynamically retrieved using similarity to the test instance from the training split of the test dataset based as in-context examples.\n\n- [1] An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA\n- [2] PromptCap: Prompt-Guided Task-Aware Image Captioning\n\n> I would adjust the following claim (L133-134): \"It is unlikely the model has ever seen the exact task of decomposition-aided visual question answering\".\n\nWe have deleted the claim.\n\n>  Do we know that the model is not able to override its priors and handle scrambled text effectively after just a few in-context examples?\n\nWe don't provide any in-context examples tailored for the scrambled task specifically; we re-use the in-context examples from the unscrambled task. We want to show that the model has some sensitivity to the structure of the decomposition and is not *exclusively* relying on keywords present in the decomposition to guess at the answer.\n\n> \"Decomposition appears to be a primarily linguistic ability\" (L207) is not necessarily generally true. \n\nWe have adjusted it to read: \n\"The ability to decompose questions in the evaluated datasets *may* be a primarily linguistic ability (but this may not be true of other VQA datasets).\"\n\n> How many in-context examples are provided to the model?\n\nA single in-context example for the recomposer, and two in-context examples for the decomposer. We experimented with different numbers of examples, and found that larger numbers of in-context examples degraded the performance of the models evaluated in the main paper. \n\n> What % of sub-question predictions are accurate (in \"Oracle / Self-Answer\")?\n\n 0.81 (VLM 3B) and 0.84 (VLM 11B) respectively, 0.56 (LM 3B) and 0.63 (LM 11B) \n\n> Rewriting L113-115 more clearly?\n\nWe have rewritten this to read:\n\n\"When provided with gold-standard decompositions on a VQA task, a model's error rate should be lower than without them.\"\n\n> aren't there works that indeed do multi-step question decomposition (like [39])?\n\nIndeed, but we focus on open-world unconstrained multimodal visual question answering datasets using open models around the 3B-11B range, while [39] focused on unimodal numerical word problems from SCAN, DROP, and GSMK using the 100B+ parameter, closed GPT3 code-davinci-002 model. \n\nWe have rewritten this sentence to read:\n\"A natural next step would be to extend the two-step approach to a multi-step approach, which remains unexplored for large vision-language models in an open-world visual question answering setting.\"\n\n> why didn't the authors at least test >1 step human-annotated decompositions from Selvaraju et al. [29] for VQAv2?\n\nThere are two reasons : \n\n1. VQA-Introspect doesn't contain multiple decompositions for every question. \n2. The decompositions aren't multi-step, they are multiple possible single step-decompositions.\n\nWe have adjusted L31-32 to read:\n\"Can multi-billion scale vision-language models benefit by approaching reasoning-heavy VQA as a two-step rather than a single-step problem using decomposition?\"\n\n> Is a decomposition really \"good\" by the proposed metric? \n\nWe conduct a human study. For decompositions which correct a wrong answer, $0.72\\pm0.12$ (95% CI) were found to be directly relevant to the question. For decompositions which cause a transition from wrong $\\rightarrow$ right, only $0.45 \\pm 0.13$ were annotated as \"good\". In many cases, a text that seems irrelevant to a human can cause the model to change a wrong answer to the correct one, but phenomena of this nature are catalogued in the LM literature: RLPrompt [1] exploits this to improve classification accuracy, [2] finds seemingly gibberish texts that can break safety guards on arbitrary LLMs.\n\n\n1. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning\n2. Universal and Transferable Adversarial Attacks on Aligned Language Models\n\n> Why not use logprobs to measure goodness?\n\nWe agree this would be ideal. But this is more complicated than it appears. Effects like surface form competition (https://arxiv.org/abs/2104.08315) can cause the probability of a nominally correct answer to go down because a semantically identical surface form has increased significantly. \n\n> Details of A-OKVQA setting?\n\nExact match with the direct answer ground-truth. Accuracy=1 if at least one ground-truth answer is equivalent to the predicted answer. We did not use the VQA metric because it makes comparing accuracies across a wide range of VQA datasets difficult, many of which do not have at least 3 human annotated answers"}, {"review_id": "togKMEcjOp", "reviewer": "Reviewer_F8RC", "summary": "In this paper, the authors focus on question difficulty in visual question answering (VQA). In particular, they propose to give subquestions that improve the percentage of correct answers to answer inferential questions more correctly. Using a large-scale vision-language model (VLM), the authors show that giving an oracle subquestion first improves the accuracy of the VQA. They also propose a selective decomposition for solving VQA so that the VLM model generates proper subquestions. This method is shown to improve VQA accuracy on multiple datasets.", "questions": "- Please demonstrate that the proposed approach improves the accuracy of VQA in common with multiple VLM models.\n- In doing so, please quantitatively evaluate how many subquestions are generated that are not relevant to VQA, e.g., through user studies. Also, please verify whether such subquestions with little relevance contribute to improving the accuracy of the VQA and provide a discussion.\n- There are a few logic gaps, confusing parts, and areas that need proofreading.\n  - The second line says \"unlike natural human question-answering strategies,\" but the authors need to write the particular human strategies to lead to the subsequent statements.\n  - In Figure 1, there is no ground truth answers, so readers do not know if all the answers shown as After are correct. Alternatively, state that all After answers are correct.\n  - Only the description of Oracle/Oracle (Scrambled) in Table 1 is not present in the caption.\n  - While the VQA system is initially shown as f(v,q)=a, the function f(v,q,(q_i, a_i)) appears in the middle with the same function name without any declarations, causing confusion to the reader. In addition, while the initial discussion includes multiple subquestions and answers for the final answer, the middle of the discussion refers to only a single subquestion. Please clarify how many subquestions are generated and provided.\n  - Please specify what decomposer_parameters is in Figure 4; if it is the number of decomposer parameters, it is odd that it is a decimal.\n  - There is a mysterious indentation on line 234.\n  - In line 267, in the \"yes / no\" section, there is an extra space before and after the slash.\n  - The columns named Acc in Tables 3 and 4 seem unclear what Acc this is.\n  - The formatting of the items in the References is not consistent. There are also references that are cited as arxiv preprints even though they have already been published.", "limitations": "The authors provide an adequate description of the scope and limitations of the proposed method in Section 2.4.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors utilize a large VLM to decompose questions to improve accuracy for VQA.\n- Validation using oracle subquestions and their answers demonstrates the contribution of question decomposition.\n- The effectiveness of the proposed method is shown for multiple parameter sizes and datasets.", "weaknesses": "- There is little technical novelty in the proposed method in this paper; it only provides an effective prompt for VQA.\n- Furthermore, as the authors acknowledge, despite the claim of question decomposition as its prompt, it also generates many subquestions that do not seem related to the question. As it stands, it remains to be seen why the accuracy of the VQA was improved.\n- The fact that the improvement in accuracy was reported only for BLIP-2 also makes it unclear whether such improvement by question decomposition is common to VLM models in general or whether it is a phenomenon found only in BLIP-2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We believe there is a misunderstanding in your summary of our work. We do not propose any method for properly generating subquestions with a specific prompt, we propose a method for *selecting which questions to decompose* based on the model's likelihood of the answer being correct.\n\nThis is an empirical study that proposes an inference procedure and studies intrinsic abilities of language and vision-language models, not any specific prompt.\n\n**We include several new experiments, and encourage you to reassess our work in light of our response, which includes detailed answers to all your questions and concerns.**\n\n### We are the first work to explore open-domain unconstrained visual question decomposition, and also the first to explore inter-model dialogue for VQA without using closed-source models like ChatGPT/GPT-4.\n\n### Four reviewers find our work novel and interesting:\n- \"The studied problem is interesting and novel. It helps us to understand the in-context learning capability in large VLMs with instruction-tuned LLMs.\" (TpMW)\n- \"The proposed method is intuitive and interesting.\" (37W7)\n- \"The results are very interesting and surprising\" (zebn)\n- \"Prior to this work, it was an open question whether LM/VLMs could decompose _visual_ questions or make use of visual question decompositions\" (yYni)\n\nThe closest prior work to ours are [1,2,3]. However, [1] focuses on code generation for VQA rather than question decomposition, and uses the closed-source ChatGPT. [2] is concurrent work (released after the NeurIPS submission deadline), and again uses the closed-source ChatGPT, and does not use confidence-based decomposition. [3] does not study question decomposition at all, instead focusing on descriptive question generation.\n\n- [1] Modular Visual Question Answering via Code Generation\n- [2] IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models\n- [3] From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models\n\n### We Explain Why Selective Decomposition Works\nWe dedicate a section in the paper (S5) to explaining this problem and the resolution. First, we carefully measure the error induction rate and error correction rate of decompositions (Table 2). We point out that decompositions can mislead the VQA model and name this problem the second-guessing problem (L217). We then propose the *selective decomposition* strategy (Fig 3), which restricts the decomposition to operating on questions to which the initial answer was likely wrong. This reduces the risk of the decomposition inducing errors while keeping the benefit. Consider the case in which most of the model's low confidence answers are wrong, which will happen if the model is well-calibrated . In this case, there is *no harm to second guessing low-confidence questions if your error correction rate is higher than the error rate on low confidence questions!* \n\nWe demonstrate (Table 3 & Table 4) that following this strategy leads to consistent increases in performance across 8 datasets, 14 decomposers, and two VQA models, for a total of 224 experiments (**this has been expanded even further in the rebuttal with additional VLMs and LLMs, for 300+ experiments**). \n\nAdditionally, we conduct a user study of the generated decompositions on VQA-Introspect, which we summarize here. The $\\pm$ numbers are the 95% confidence intervals according to a population proportion test.\n- For all decompositions, the % of good subquestions is $0.61 \\pm 0.1$.\n- For decompositions which successfully correct a wrong answer (wrong $\\rightarrow$ right), $0.72\\pm12$ were annotated as good.\n\nWe can draw the following conclusions:\n- The majority of subquestions are \"good\" according to humans.\n- Even subquestions that are not \"good\" often result in a flip from wrong $\\rightarrow$ right.\n\nThis may seem unintuitive, but many such phenomena of seemingly irrelevant text that has an effect on the LLMs output exists, see [1] and [2].\n\n1. RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning\n2. Universal and Transferable Adversarial Attacks on Aligned Language Models\n\n### We have added 80 additional experiments with 2 additional VLMS (over 300 total experiments)\nWe add OpenFlamingo (3B) and InstructBLIP (based on Vicuna 7B), and test each with 5 decomposers across 8 datasets for a total of 80 new experiments.\n\n\n### We conduct a user study to analyze the decompositions\nAdditionally, we conduct a user study of the generated decompositions on VQA-Introspect, which we summarize here. The $\\pm$ numbers are the 95% confidence intervals according to a population proportion test.\n- For all decompositions, the % of good subquestions is $0.61 \\pm 0.1$.\n- For decompositions which successfully correct a wrong answer (wrong $\\rightarrow$ right), $0.72\\pm12$ were annotated as good.\n\nThis suggests that roughly $\\approx30$% of subquestions which were not annotated as \"good\" still contribute to increasing the accuracy of the VQA model. We will add a discussion of this user study.\n\n### Clarifications\n- We focus on single-step decompositions only.\n- `decomposer_parameters` in Figure 4 the number of parameters in the decomposer; the decimal is a formatting error that will be removed.\n- \"Acc\" in Tables 3/4 refer to exact match accuracy (1 if the answer matches one of the ground truth answers, zero otherwise).\n- In Fig 1, all the \"After\" answers are correct.\n- Thank you for pointing out the inconsistent formatting of the references and other details, we will fix this.\n"}, {"review_id": "OoWYSgsuUY", "reviewer": "Reviewer_zebn", "summary": "The paper examine whether the current instruction-tuned VLM (e.g. BLIP2 ) can (1) learn to generate simple perception questions that can help to answer a more complex reasoning visual question; (2) learn to actually benefit from the perception questions.\n\nThe conclusion is that \n\nfor (1): language models (with various sizes) can learn to produce effective decompositions just by viewing examples and adding image as content is not being super helpful.\n\nfor (2): Yes, the performance of the reason question answering is better with golden perception question even without providing answers to the perception question; and achieve good results with selective Decomposition", "questions": "Please comment on the weakness part\n\n\n\n\n\n\n------------------------------------\npost rebuttal: I read the author's rebuttal and still willing to support the paper as my initial rating", "limitations": "The authors adequately addressed the limitations", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "(1) The paper provide very detailed discussion on how the flan model can be used to generate perception questions based on only 2 exemplars. The results are very interesting and surprising in that (a) even small size model can provide \"decomposition\" (sometime gibberish) that helps the recomposer model and visual input does not help a lot\n\n\n(2) test the selective decomposition that actually using the generated perception question to boost the reasoning question without having to know whether the question is requiring the  perception question \n\n\n(3) The paper is well-written and easy to follow", "weaknesses": "(1) It is very surprising that the visual input does not help generate better perception questions. For example, in the supplement material, the reasoning question is \"is it cold outside\" and the perception question is \"are any people wearing jacket\". The perception question is very easy to be generated if we see a person in the picture, but without visual content, there could be lots of perception question that are valid, e.g. \"is there snow\", \"are there green leaves on the tree\",  \"is the lake frozen\" that may be irrelevant to the visual question\n\nMy hypothesis is that the image + text model does not use the image correctly in that the question generation process is not focusing on the image. I wonder if it is easy to see (a) adding prompts that explicit say the perception question should be based on the image (b) converting the image into multiple captions and use the captions as visual inputs\n\n\n(2) It seems the final recomposer should be able to take multiple perception questions as input, it would be good to sample multiple perception questions from the decomposer and use all of perception questions to see if it helps the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review! These are great questions and we have considered them ourselves.\n\n### Why doesn't visual input help generate better perception questions?\nWe believe the answer to this is related to the textual biases of the dataset. For example, in the second half of Table 1, we can see that the *zero shot* blind language-only model can achieve 63% accuracy on VQAv2 (SOTA is around ~84%), This is because many of the questions in VQA datasets focus on certain parts of a scene (e.g. the people in the scene) and so with or without an image, the tendency of the model is to ask about people (for example). This bias is quite well documented [1,2] and can be surprisingly strong. In addition, many VQA datasets are based on COCO, which is highly person-centric (for example, *at least* 15k/40k images in COCO validation set contain people [3]), which means that it is usually very reasonable to ask a question about people, even without being able to see the content of the image. \n\n- [1] An analysis of visual question answering algorithms \n- [2] Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering\n- [3] Understanding and Evaluating Racial Biases in Image Captioning\n\nFinally, we do agree with your hypothesis. The models appear to be biased towards the text from some informal experiments. However, it is not easy to alter this with prompts (we have tried). Further training or architectural changes may be required, and this is likely a property of the vision-language pretraining of the models.\n\n### Should we try sampling multiple questions from the decomposer?\nThis is possible in principle. However, the challenge here is sampling diverse questions from the decomposer, which is a nontrivial task from our initial experiments -- there can be repetitions, and increasing the temperature to sample more diverse questions decreases the quality of the decompositions. Also, the risk of distractors and irrelevant questions rises with each subsequent addition to the context of the recomposer. For this reason, **we focused on good single-step decompositions, because a good single-step decomposition is cheap yet can have a huge effect on performance**."}, {"review_id": "441gSdfgF6", "reviewer": "Reviewer_37W7", "summary": "To answer the questions that require reasoning or outside knowledge, this paper proposes a question decomposition method which uses LLM to infer subquestions", "questions": "Please refer to \"Weaknesses\"", "limitations": "The proposed methods lacks the comprehensive analysis on different zero-shot VQA models and LLM models, which raises the concerns about the generalization ability.  In addition, the paper should also  conduct some analysis on straightforward VQA tasks to demonstrate the generalization ability of the proposed methods with different question types.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is intuitive and interesting.\n\n2. The methods conduct comprehensive experiments on BLIP2 and FlanT5, and the results clearly demonstrate the effectiveness of the proposed methods.", "weaknesses": "1. While this research paper serves as an investigative study, it only validates the proposed methods using FlanT5 and BLIP2 models. This limited scope raises questions about the generalizability of the proposed methods. It would enhance the study's breadth if further tests were conducted on a diverse range of zero-shot VQA methods, such as Flamingo [1], Frozen [2], and Img2LLM [3]. Additionally, incorporating various Language Learning Models (LLMs) for decomoposing questions like OPT and the GPT series would further strengthen the results.\n\n2. I acknowledge that the proposed methods are primarily designed for tasks necessitating reasoning. However, given the community's growing interest in open-ended environments, it may be beneficial to conduct some analysis on straightforward VQA tasks. For instance, the experimental results on tasks like VQAv2 or GQA could provide valuable insights.\n\n3. The proposal of self-generated question-answer pairs into VQA tasks is not a novel concept. To improve the understanding of how this method compares with prior approaches, a dedicated section comparing it with earlier methods such as VQ^2A [4], Weak VQA [5], and Img2LLM would be beneficial.\n\n[1] Alayrac, J. B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., ... & Simonyan, K. (2022). Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35, 23716-23736.\n\n[2] Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S. M., Vinyals, O., & Hill, F. (2021). Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34, 200-212.\n\n[3] Guo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., & Hoi, S. C. (2022). From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846.\n\n[4] Changpinyo, S., Kukliansky, D., Szpektor, I., Chen, X., Ding, N., & Soricut, R. (2022). All you may need for vqa are image captions. arXiv preprint arXiv:2205.01883.\n\n[5] Banerjee, P., Gokhale, T., Yang, Y., & Baral, C. (2020). WeaQA: Weak supervision via captions for visual question answering. arXiv preprint arXiv:2012.02356.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive review, and have added the requested experiments and will add the requested references. \n\n### We add experiments with 2 new VLMs and 2 new LLMs.\nThank you for the great suggestion! We have added two additional VLMs (Flamingo and InstructBLIP), and two additional LLMs (Falcon and Galactica). Unfortunately, we do not have the resources at this time to try some of the larger OPT models or use GPT-3 for evaluation. \n\n### We add experiments on GQA.\nWe agree that straightforward VQA tasks could be interesting. VQA-Introspect is actually based on VQAv2 (it contains a subset of the questions in VQAv2 which require a combination of higher-level reasoning and low-level perception). However, we have also added an evaluation on GQA, following the methodology of [1].\n\nOur method still works on GQA, despite the fact that many of the GQA questions are low-level perceptual questions.\n\n| VQA Model | Type | Decomposer Params | Acc | %$\\uparrow$ | $\\eta$ | $I(\\tau)$ |\n|---|---|---|---|---|---|---|\n| 3B | I+T | 3B | 53.79 | 1.52 | 73 | 1.14 |\n|  | T | 80M |  | 0.2 | 3 | 11.4 |\n|  | T | 3B |  | 0.1 | 3 | 11.4 |\n|  | T | 11B |  | 0.81 | 73 | 1.14 |\n|  | Galactica | 7B |  | 0.71 | 8 | 6.51 |\n|  | Falcon | 7B |  | 1.31 | 91 | 0.66 |\n| 13B | I+T | 11B | 55.11 | 0.81 | 49 | 1.95 |\n|  | T | 80M |  | 0.1 | 2 | 16.39 |\n|  | T | 3B |  | 1.42 | 49 | 1.95 |\n|  | T | 11B |  | 0.2 | 2 | 16.39 |\n|  | Galactica | 7B |  | 1.52 | 69 | 1.34 |\n|  | Falcon | 7B |  | 0.4 | 12 | 5.98 |\n\n[1] Visual Programming: Compositional visual reasoning without training\n\n### We will add a discussion relating our work to earlier question-generation work.\nWe agree that our work is closely related to these works, and **we have added a section to our literature review citing VQ2A, Weak VQA, and Img2LLM**. The common unifying theme is that we build on the question-generation paradigm introduced by these works, and the key difference is that we direct our question generation to focus on decompositions rather than general questions (also the two-step VQA process)."}, {"review_id": "DDn29K916V", "reviewer": "Reviewer_TpMW", "summary": "This paper studies the question decomposition of VQA.\nTo begin with, the authors demonstrate that large VLMs equipped with LLMs can effectively leverage question decomposition to improve answering performance.\nThereafter, some experiments are conducted to prove that instruction-tuned language models are capable of auto-writing decompositions.\nSince some questions do not require decomposition to reach the right answer, the authors then propose a novel selective decomposition method to selectively decide which questions demand decomposition.", "questions": "Overall, I believe this is a good paper.\nHowever, the effectiveness of question decomposition from large VLMs is confusing.\nIf the authors can address this concern, I will raise my initial score.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and well-organized. Most parts of this paper are easy to follow.\n- The studied problem is interesting and novel. It helps us to understand the in-context learning capability in large VLMs with instruction-tuned LLMs.\n- To validate the effectiveness of question decomposition, the authors perform comprehensive experiments.", "weaknesses": "- My biggest concern lies in Sec. 4 about whether large models can produce effective decompositions. As shown in Table 2, most $E_{CR}$ values are pretty low while $E_{IC}$ are relatively high. According to the definitions of these two terms, it seems that most decompositions severely hurt the model performance. For example, 10% $E_{CR}$ means only 1 of 10 vqa instances can be corrected by the recomposer. In contrast, the average value of $E_{IC}$ is around 40. This means 4 of 10 vqa instances after decomposition failed. \n- For Sec. 5, have authors tried these newly introduced metrics after the selective decomposition?\n- Have the authors considered using the GQA dataset [1], which involves a longer chain of reasoning?\n- About Table 1, the oracle/no answer outperforms oracle/self-answer by a large performance margin. This result does not make much sense since using more context usually leads to more accurate answer predictions.\n- Moreover, the scrambling is not well-explained. How many words are perturbed? This question is important because a larger perturbation often causes a severe performance drop. In addition, why the authors have not tried scrambling the questions? This is also essential because the well-known bias problem in VQA is from question key words and answers.\n\nTypo:\n- Fig. 1 is not referred in the main manuscript.\n\n[1] GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In CVPR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Great questions, and thank you for the positive appraisal of our work. We have added an experiment for you, as well as a detailed discussion of the of $E_{IC}$ and $E_{CR}$ which should clear up any confusion: we do believe large models can decompose well. \n\n### Why does selective decomposition work in spite of seemingly high error rates?\nThere is a subtle error in your interpretation. The $E_{CR}$ and $E_{IC}$ values have to be interpreted with care, because they are relative to the model's error rate. For example, consider a situation in which there are 100 questions, and the error rate is 90%, so the model only predicts $\\frac{10}{100}$ correct answers. Assume an $E_{IC}=0.5$ and $E_{CR}=0.1$. This would mean that 5 initially correct answers were flipped to incorrect, but 9 initially wrong answers were flipped to correct answers. So the number of total correct answers would go from $10 \\rightarrow 14$, a $40\\%$ increase in accuracy.    \n\nIf we calculated the $E_{IC}$ and $E_{CR}$ absolutely (by making the denominator in Eq 1 & 2) the total number of questions, your interpretation would be correct. However, if we calculate it this way , the $E_{CR}$ and $E_{IC}$ values are very close, which we have computed for you below in the case of BLIP-2 (3B). So in this case, we can see that the $E_{IC}$ and $E_{CR}$ are very close and a small nudge from selective decomposition can increase the $E_{CR}$ over the $E_{IC}$ if it is not already higher.\n\n| Dataset | $E_{CR}$ | $E_{IC}$ | Error Rate |\n|---|---|---|---|\n| A-OKVQA | 3.93 | 5.41 | 50.31 |\n| ArtVQA | 3.62 | 2.2 | 83.15 |\n| OK-VQA | 4.72 | 4.48 | 63.56 |\n| Slake | 10.65 | 9.61 | 66.73 |\n| VQA Introspect | 8.92 | 6.53 | 22.07 |\n\nAs a corrollary, consider the case in which most of the model's low confidence answers are wrong, which will happen if the model is well-calibrated . In this case, there is *no harm to second guessing low-confidence questions if your error correction rate is higher than the error rate on low confidence questions!* Because the error rate on low-confidence questions is very high, most of them are already wrong and there is no penalty induced for changing an answer from wrong $\\rightarrow$ wrong.\n\n### $E_{IC}$ and $E_{CR}$ do increase after selective decomposition\n\nWe have computed these metrics after decomposition (using the absolute formulation so it is easier to interpret), which can be viewed in the _rebuttal PDF_ (Figure 9). In all cases, selective decomposition results an in increased $E_{CR}$ of the decomposed question is higher, and lower $E_{IC}$.\n\n### We add experiments on GQA.\n\n\n| VQA Model | Type | Decomposer Params | Acc | $\\%\\uparrow$ | $\\eta$ | $I(\\tau)$ |\n|---|---|---|---|---|---|---|\n| 3B | I+T | 3B | 53.79 | 1.52 | 73 | 1.14 |\n|  | T | 80M |  | 0.2 | 3 | 11.4 |\n|  | T | 3B |  | 0.1 | 3 | 11.4 |\n|  | T | 11B |  | 0.81 | 73 | 1.14 |\n|  | Galactica | 7B |  | 0.71 | 8 | 6.51 |\n|  | Falcon | 7B |  | 1.31 | 91 | 0.66 |\n| 13B | I+T | 11B | 55.11 | 0.81 | 49 | 1.95 |\n|  | T | 80M |  | 0.1 | 2 | 16.39 |\n|  | T | 3B |  | 1.42 | 49 | 1.95 |\n|  | T | 11B |  | 0.2 | 2 | 16.39 |\n|  | Galactica | 7B |  | 1.52 | 69 | 1.34 |\n|  | Falcon | 7B |  | 0.4 | 12 | 5.98 |\n\n### Does self-answering hurt performance?\n\nThis is only the case for the text-only models. As they cannot see the image, their self-answers to the decomposition are hallucinated and can mislead themselves. For the image+text models, there is only a substantial difference between self-answer and no-answer in one case: BLIP-2 (3B) on \"numerical\" questions. The reason for this is simple: upon manually inspecting the answers, we find that the VQA models make substantial errors when they answer a decomposition. This has also been observed by the line of work building upon VQA-Introspect, which has shown that even VQA models with high accuracy are often inconsistent, providing wrong answers to questions that should be entailed by a main question.\n\nThis effect can be seen clearly when comparing the oracle/oracle rows with the oracle/self-answer rows -- even self-answering with a strong VQA model reduces the benefit of the decomposition, showing that the VQA model is predicting incorrect or different answers than the human-annotated oracle answers.\n\n### Explanation of scrambling\n\nWe scramble the order of all the words in the decomposition. We don't scramble the questions because we are only testing the bias of the VQA model towards keywords in the decomposition. Scrambling the question would be interesting, but it is not directly relevant to our research questions.\n\n### Our statistical analysis shows that larger models produce better decompostions.\nWe've also carried out a statistical analysis, which shows that the correlation between the number of parameters in the decomposer and the resulting net gain after decomposition is statistically significant at a confidence level of 95% with an $R^2=0.396$, again suggesting that larger decomposers are better at decomposing. We also do this analysis for $E_{IC}$ and $E_{CR}$ (L202-203). Finally, we show in the supplement (Section D) that $E_{IC}$ drops roughly .7% for every 1B increase in parameters, while $E_{CR}$ increases .2% for every 1B increase in parameters. Both of these relationships are statistically significant.\n"}]

Author rebuttal:
We thank all reviewers for the detailed feedback and engagement with the paper, and the positive appraisals of our work!

Our rebuttal includes a number of experiments requested by reviewers, and a human study of the decompositions. **Please see the PDF for detailed tables.**

### Additional Experiment With Two New VLMs + Two New LLMs
Reviewers requested experiments with more VLMs and more LLMs. 
**Our evaluation suite now spans 3 VLM families, 9 datasets, and 3 LLM families, with over 300 total configurations tested.**
Please see Table 8 and 9 in the attached PDF, which are similar to Table 3/4 in the main paper. 
Selective decomposition continues to work and produce net gains in VQA Accuracy for all configurations tested. 
- LLMs: Selective decomposition has been shown to work with the FLAN-T5 family, the instruction tuned Falcon-7B, and the instruction tuned Galactica model.
- VLMs: Selective decomposition has been shown to work with the BLIP-2 model based on FLAN-T5, the OpenFlamingo-3B model based on MPT-1B, and InstructBLIP based on VIcuna-7B (itself based on Llama-7b).

### Added Dataset: GQA
Reviewers requested we add an evaluation on the GQA dataset. We show the results here for the BLIP-2 FLAN-T5 family of VLMs. **Although our method was intended for knowledge-intensive reasoning tasks, it remains effective on the visual reasoning tasks of GQA.**
| VQA Model | Type | Decomposer Params | Acc | $\%\uparrow$ | $\eta$ | $I(\tau)$ |
|---|---|---|---|---|---|---|
| 3B | I+T | 3B | 53.79 | 1.52 | 73 | 1.14 |
|  | T | 80M |  | 0.2 | 3 | 11.4 |
|  | T | 3B |  | 0.1 | 3 | 11.4 |
|  | T | 11B |  | 0.81 | 73 | 1.14 |
|  | Galactica | 7B |  | 0.71 | 8 | 6.51 |
|  | Falcon | 7B |  | 1.31 | 91 | 0.66 |
| 13B | I+T | 11B | 55.11 | 0.81 | 49 | 1.95 |
|  | T | 80M |  | 0.1 | 2 | 16.39 |
|  | T | 3B |  | 1.42 | 49 | 1.95 |
|  | T | 11B |  | 0.2 | 2 | 16.39 |
|  | Galactica | 7B |  | 1.52 | 69 | 1.34 |
|  | Falcon | 7B |  | 0.4 | 12 | 5.98 |

### We add a human evaluation of the decompositions
Results are summarized below (all $\pm x$ are 95% CI based on a population proportion test) :
- For all decompositions, the % of relevant subquestions is $0.61 \pm 0.1$.
- % of subquestions answered correctly $0.77 \pm 0.6$. 
- For decompositions which successfully correct a wrong answer (wrong $\rightarrow$ right), $0.72\pm12$ were annotated as relevant.
- For decompositions which induce an error (right $\rightarrow$ wrong), $0.45 \pm 13$ were annotated as relevant. 

We sample ~250 decompositions from VQA-Introspect generated by BLIP-2 (FLAN-T5 3B) and manually inspect each decomposition for relevance to the main question.

### Conclusion
We believe we have answered the main questions from all reviewers. Due to space limitations, we may not have replied to minor points, but all your feedback has been noted. 

Meta-review: The paper studies the idea of question decomposition (which is borrowed from NLP) for VL tasks. The reviewers found the experiments to be comprehensive and results convincing. The reviewers found the paper to be well written and easy to follow.

The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are encouraged to improve the final paper version by following reviewer recommendations.
Decision: Accept (poster)