SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Experiments & Discussion We use BLIP-2 [7] based on the FLAN-T5[34] as the question answering model (recomposer). For the decomposers, we use FLAN-T5[34] models ranging in size from 80M parameters to 11B parameters, as well as the BLIP-2 models themselves. We use four VQA
datasets from three domains: ArtVQA[36] (art), SLAKE[20] (medical), and A-OKVQA[35] and OKVQA [37] (external knowledge VQA on natural images). We then carry out the procedure illustrated in Fig. ② for each combination of decomposer, recomposer, and dataset. We handcraft three demonstrations of writing a subquestion for a question, in the form “Reasoning Question: <question>? Perception Question: <sub>subquestion>?” For each  $\nu, q$  pair in a dataset, we prompt the decomposer with the demonstration, followed by the question  $q$  as in Fig. ② and measure  $E_{CR}$  as in Eq. ① and  $E_{IC}$  as in Eq. ② for each dataset. We show the results in Tab. ②. We find that yes, language models ≤ 13B parameters can learn to produce effective decompositions just by viewing examples. Decomposer size correlates positively with  $E_{CR}$  ( $R^2 = 0.344$ ), and negatively with  $E_{IC}$  ( $R^2 = .273$ ) and the correlations are significant at  $\alpha = 0.05$  across a larger collection of eight datasets used in Sec. ⑤. A human examination of the “subquestions” produced by smaller models shows that many of them are gibberish and not properly formed questions at all. Despite this, they surprisingly manage to maintain an  $E_{CR}$  that is sometimes higher than larger models. Finally, the ability to decompose questions in the evaluated datasets may be a primarily linguistic ability in that it is possible to ask effective subquestions about an image without being able to see the image, and the difference in effectiveness between the Image+Text BLIP-2 models and the text-only FLAN-T5 models of a similar size is on average ≈ 10% of the base error rate (but this may not be true of other VQA datasets).
5 Selective Decomposition Works Better Than Naive Decomposition
One problem shows up in Sec. 4, which is that applying decompositions to every question can hurt performance, by flipping answers that were initially correct to be incorrect. If we were able to decompose only wrong answers, we would always see a net gain in performance due to the error correction of decompositions. However, in a realistic setting, we do not know apriori that our answers are wrong, and thus run the risk of flipping an answer that was initially correct to be incorrect by applying a decomposition that is misleading. We call this the second-guessing problem.
$\nu$  : Image   
 $q$  : Question   
 $\tau$  : Confidence Threshold   
Attempt an initial answer.   
 $\hat{a}$ $p(\hat{a}) =$  recomposer  $(v,q)$    
Selectively decompose  $q$  if   
answer  $\hat{a}$  is uncertain.   
if  $p(\hat{a}) <   = \tau$  ..  $q^{\prime} =$  decomposer  $(v,q)$ $a^\prime =$  recomposer  $(v,q^{\prime})$  Answer again.  $\hat{a} =$  recomposer  $(v,q,q^{\prime},a^{\prime})$
Figure 3: Pseudocode for selective decomposition.
To deal with the second-guessing problem, we propose following an intuitive human strategy: stick with your initial answer on questions you are confident about, and only second-guess (apply a decomposition) for questions you are not confident about. Language models can be surprisingly well calibrated[39], meaning that the probability they assess to an output sequence they produce is often well correlated with the probability that the produced output sequence is the "correct" one for a given task. We make use of this property to treat visual question answering as a selective prediction [40] task, using the language models' confidence as a decision score to determine whether we should apply a decomposition to a instance or stick with the original answer.
We describe the algorithm in pseudocode in Fig. 3. The selective decomposition procedure transforms VQA from
a single-step task to a two-step task given a decomposer model, a recomposer model, a confidence threshold  $\tau$ , and a visual question pair  $\nu, q$ . An initial answer  $\hat{a}$  and confidence  $p(\hat{a})$  is solicited from the recomposer model. If  $p(\hat{a}) < \tau$ , the decomposition procedure is invoked, and a subquestion and answer pair  $(q', a')$  are generated by the recomposer and recomposer working together. The recomposer model is then allowed to "second-guess" the initial answer  $\hat{a}$  with the decomposition  $(q', a')$  as additional context. The recomposer and recomposer can be the same model or different models. We experiment with both scenarios. This introduces an extra hyperparameter  $\tau$  into the inference procedure.
Experiments & Discussion In Fig. 4 we show the effect of different values of  $\tau$  on the accuracy of selective decomposition with several decomposers. Across all datasets and all models, there is a wide range of  $\tau$  (expressed as percentiles) for which selective decomposition improves predictive accuracy. At the same time, we clearly demonstrate the second guessing problem in Fig. 4 Decomposing every question often eventually leads to lower accuracy than decomposing no questions at all, because hallucinations and misleading decompositions can flip an initially correct answer to an incorrect answer.
Figure 4: Selective decomposition mitigates the problem of misleading decompositions. We decompose questions based on model confidence in the initial answer, and show how accuracy initial rises past the baseline as the model mostly second guesses wrong answers, and then drops below the baseline with no decompositions (horizontal line) if too many questions initially answered correctly are second-guessed.
Figure 5: Decomposition is more effective on non-natural image domains, and models are also less confident in these domains. Size of circles is proportional to parameter count.
In Tabs. 3 and 4 we show the highest possible net gain achieved by selective question decomposition on three domains by different decomposers. Selective decomposition consistently improves predictive accuracy regardless of the decomposer and domain. Net gains are larger on datasets (passes t-test with  $\alpha = 0.05$ ) containing non-naturalistic images and specialized domains (e.g. medical) than they are on domains containing natural images. The mean optimal surprisal  $I(\tau)$  for second-guessing answers is lower for non-natural domains ( $\mu_{I(\tau)} = 13.2$  for the natural image datasets vs  $\mu_{I(\tau)} = 9.0$  for the medical and art datasets, confirmed by t-test at  $\alpha = 0.5$ ). We further visualize this in Fig. 5 This matches our expectations: you should second guess yourself on domains you understand poorly more than on domains you understand well. A linear regression fit shows that larger decomposers correlate with larger net gains ( $R^2 = 0.365$ ,  $R^2 = 0.342$  for natural image domains and medical / art domains respectively, t-test with  $\alpha = 0.05$ ).
We reformulate Winoground as a VQA task by turning each caption into a question with a boolean yes / no answer (does “<caption>” describe the image?) on which chance accuracy is  $50\%$ . As visible in Tab. 3, all BLIP-2 models perform below random chance, in agreement with previous results on Winoground showing that it is extremely difficult for vision-language models. Surprisingly, after decompositions produced by the relatively FLAN-T5-small/base models (80M/200M parameters), the performance of BLIP-2 (13B) rises to significantly above chance  $(+18\%)$ . Upon inspection, many of the decompositions produced by the model appear to be gibberish, yet remarkably, induce the much larger 13B BLIP-2 model to correct over  $30\%$  of its initially wrong answers.
6 Literature Review
Task decomposition [5,6,41,42] improves the performance of large language models on zero-shot reasoning tasks. The only work so far to apply similar techniques for VQA is MM-CoT [43], but it does not explore task decomposition with large vision-language models, choosing to finetune a smaller model instead. The ability to use zero-shot task decompositions may be a property of model scale, emerging at 60-200B parameters [44], or may be a property of large-scale pretraining on code [45]. Such large vision-language models have only been developed recently due to advances in vision-language alignment. The prevailing paradigm in vision-language pretraining was to build vision-language models atop (relatively) small language models [26-29, 46] below 1B parameters. Meanwhile, language models were being scaled from 3B-175B parameters [34, 47-50], with each model family having at least one representative with  $>10$  B parameters. Because vision-language
VQA	Decomposer	AokVQA[35]	OkVQA[37]	VQA Introspect[32]	WinogroundVQA[23]
Type	Params	Acc	%↑	η	I(τ)	Acc	%↑	η	I(τ)	Acc	%↑	η	I(τ)	Acc	%↑	η	I(τ)
3B	T	80M	0.79	16	13.57	0.59	15	16.69	3.23	8	10.64	5.94	94	0.41
T	250M	0.44	11	17.58	0.46	13	18.05	3.1	7	13.24	22	99	0.19
T	780M	0.87	18	12.42	0.67	26	11.02	3.18	8	10.64	0	1	8.59
T	3B	49.69	1.14	31	8.13	36.54	0.97	23	12.09	78.64	3.5	16	5.87	45.81	0	1	8.59
I+T	3B	1.22	33	7.71	1.29	27	10.7	3.75	20	5.24	0	1	8.59
T*	7B	1.75	31	8.13	0.73	15	16.69	3.37	8	10.64	0.06	2	7.4
T	11B	0.7	11	17.58	1.03	22	12.47	3.6	14	6.27	0	1	8.59
13B	T	80M	1.14	18	14.91	1.15	18	15.68	3.46	8	11.81	6.94	95	0.22
T	250M	0.35	9	22.96	0.81	11	22.13	3.31	7	15.81	22.12	99	0.09
T	780M	53.36	0.52	14	17.93	39.79	1.15	18	15.68	78.93	3.46	8	11.81	46.5	0.06	2	3.83
T	3B	1.05	14	17.93	1.51	18	15.68	3.73	8	11.81	0.69	92	0.29
T*	7B	1.48	18	14.91	1.37	24	12.54	3.69	8	11.61	4.06	99	0.09
T	11B	1.66	25	10.95	1.66	21	13.98	3.68	8	11.81	0.06	14	1.64
I+T	11B	1.92	25	10.95	1.68	24	12.54	3.81	8	11.81	0	1	5.49
Table 3: Increases in accuracy produced by selective decomposition at the optimal second-guessing confidence threshold  $\tau$ , on external knowledge QA and visual reasoning across several decomposers.  $I(\tau) = \log_2\left(\frac{1}{\tau}\right)$  is the surprisal of  $\tau$ , and  $\eta$  is the percent of the questions in the dataset above  $I(\tau)$ , or equivalently, below  $\tau$ . T=FLAN-T5, I+T=BLIP-2 (based on FLAN-T5) and T\*=Galactica.
VQA	Decomposer	ArtVQA 36	PathVQA 22	SLAKE 20	VQA Rad 21
Type	Params	Acc	%↑	η	I(τ)	Acc	%↑	η	I(τ)	Acc	%↑	η	I(τ)	Acc	%↑	η	I(τ)
3B	T	80M	2.36	50	10.37	15.89	98	2.42	3.86	24	22.12	18.73	97	2.88
T	250M	2.68	50	10.37	11.93	69	14.79	4.05	26	16.62	17.57	69	16.57
T	780M	5.28	55	8.91	12.61	99	1.91	4.15	25	19.65	15.66	69	16.57
T	3B	17.17	2.44	50	10.37	12.45	11.56	98	2.42	33.27	3.96	38	7.33	11.7	18.82	89	5.76
I+T	3B	2.13	29	17.84	11.34	69	14.79	5.84	61	4.71	18.06	98	1.6
T*	7B	2.91	30	17.47	13.9	99	1.91	6.13	50	5.71	19.62	94	4.44
T	11B	3.15	55	8.91	14.08	98	2.42	5.18	31	10.83	19.17	98	1.6
13B	T	80M	4.33	63	12.09	23.4	99	1.39	8.95	45	14.45	23.89	99	2.95
T	250M	4.41	41	19.4	18.46	99	1.39	7.54	39	19.48	23.67	99	2.95
T	780M	5.91	84	6.67	22.62	99	1.39	6.03	41	17.08	24.02	99	2.95
T	3B	16.85	4.57	37	20.79	5.56	21.33	99	1.39	22.62	8.67	44	14.8	5.12	23.84	99	2.95
T*	7B	5.12	60	13.12	21.07	99	1.39	8.29	39	19.48	24.47	99	2.95
T	11B	3.94	39	20.05	21.56	99	1.39	9.52	44	14.8	26.25	99	2.95
I+T	11B	5.98	86	6.35	20.85	99	1.39	10.37	42	16.08	25.13	99	2.95
Table 4: Increases in accuracy produced by selective decomposition at the optimal second-guessing confidence threshold  $\tau$ , across two domains (medical/art) and several decomposers.  $I(\tau) = \log_2\left(\frac{1}{\tau}\right)$  is the surprisal of  $\tau$ , and  $\eta$  is the percent of the questions in the dataset above  $I(\tau)$ , or equivalently, below  $\tau$ . T=FLAN-T5, I+T=BLIP-2 (based on FLAN-T5) and T\*=Galactica.
pretraining typically requires full model training, aligning these multi-billion parameter models to the visual modality was prohibitively computationally expensive. However, recent discoveries [51, 52] motivated by earlier work with frozen models [53] have shown that the representation spaces of vision models and large-language models are surprisingly close, and rough alignment can be achieved with adapters [15] or linear mapping layers while keeping the language model frozen, and more advanced techniques have given rise to vision-LLMs [7-9]. Our work is closely related to the visual question generation paradigm of [11, 54, 55]. However, we direct our question generation to focus on decompositions rather than general questions.
We show that question decomposition is already a viable strategy that can be used to implement a more natural approach to VQA. Without any training, instruction-tuned VLMs can learn to produce and use decompositions from a few demonstrations of the task. This approach has many possible future directions. For example, we only consider two-step approaches for visual question answering, where we "hardcode" the depth of the decomposition. A natural next step would be to extend the two-step approach to a multi-step approach, which remains unexplored for large vision-language models in an open-world visual question answering setting. Second, in-context learning has limitations. Would models benefit from being trained to produce and consume decompositions?
[1] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering.