Large Language Models are Visual Reasoning Coordinators
Authors: Liangyu Chen
Bo Li
Sheng Shen
Jingkang Yang
Chunyuan Li
Kurt Keutzer
Trevor Darrell
Ziwei Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: visual reasoning
large language models

Abstract: Visual reasoning requires multimodal perception and commonsense cognition of the world. Recently, multiple vision-language models (VLMs) have been proposed with excellent commonsense reasoning ability in various domains. However, how to harness the collective power of these complementary VLMs is rarely explored. Existing methods like ensemble still struggle to aggregate these models with the desired higher-order communications. In this work, we propose Cola, a novel paradigm that coordinates multiple VLMs for visual reasoning. Our key insight is that a large language model (LLM) can efficiently coordinate multiple VLMs by facilitating natural language communication that leverages their distinct and complementary capabilities. Extensive experiments demonstrate that our instruction tuning variant, Cola-FT, achieves state-of-the-art performance on visual question answering (VQA), outside knowledge VQA, visual entailment, and visual spatial reasoning tasks. Moreover, we show that our in-context learning variant, Cola-Zero, exhibits competitive performance in zero and few-shot settings, without finetuning. Through systematic ablation studies and visualizations, we validate that a coordinator LLM indeed comprehends the instruction prompts as well as the separate functionalities of VLMs; it then coordinates them to enable impressive visual reasoning capabilities.

Large Language Models are Visual Reasoning Coordinators
Liangyu Chen V Bo Li Sheng Shen
Jingkang Yang"
Chunyuan Li Kurt Keutzer Trevor Darrell Ziwei Liu M.
S-Lab, Nanyang Technological University
University of Californi,, Berkeley Microsoft Research, Redmond (1chen025, 1ibc0013, ziwei Miluininttt edd.. sg
https://github com/ cliagyy/Coo
Abstract
Visual reasoning requires multimodal perception and commonsense cognition of the world. Recently, multiple vision-language models (VLMs) have been proposed with excellent commonsense reasoning ability in various domains. However, how to harness the collective power of these complementary VLMs is rarely explored. Existing methods like ensemble still struggle aggregate these models with the desired higher-order communications. In this work, we propose Cola, a novel paradigm that coordinates multiple VLMs for visual reasoning. Our key insight is that large language model (LLM) can efficiently coordinate multiple VLMs by facilitating natural language communication that leverages their distinct and complementary capabilities. Extensive experiments demonstrate that our instruction tuning variant, Cola-FT, achieves state-of-the- of-the--rtt performance on visual question answering (VQA), outside knowledge VQA, visual entailment, and visual spatial reasoning tasks. Moreover, we show that our in-context learning variant, Cola-Zero, exhibits competitive performance in zero and few-shot settings, without finetuning. Through systematic ablation studies and visualizations, we validate that a coordinator LLM indeed comprehends the instruction prompts as well as the separate functionalities of VLMs; then coordinates them to enable impressive visual reasoning capabilities.
Ensemble
Cola
Action 1 VLMS describe the visual context.
* VLM-1
+Kt VLM-2
8 VLLM1
* VLM-2
A bus is traveling down street with bus.
E
Mass transportation system is the largest public transport system in the world.
Ensemble
LLM
Answer
Answer
Action 2
VLMs answer the question.
(a) Comparison between Paradigms
How many vehicles headlights on?
How many vehicles have headlights on?
One.
Two.
(A) One
Action 3 - LLM coordinates and answers.
(B) None
(C) Two
(D) Three
Two.
(b) Example Visual Reasoning Task: VQA
(c) Cola Solution to the Example VQA
Figure 1: We propose, Cola, using coordinative language model for visual reasoning. Cola coor- dinates multiple pretrained VLMs based on the visual context and plausible answers they provide. 'Equal Contribution.
iProecc Lead.
EsCorresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Visual reasoning is a crucial task that demands models to not only comprehend and interpret visual information but also apply high-level cognition to derive logical solutions [40, 120, 60). The field has received significant attention from the machine learning community because of its potential to enable wide range of intelligent applications, such as intelligent tutoring systems [5, 94, 69], automated image captioning [103], and virtual assistants [88, 50]. To perform visual reasoning effectively, a model must possess both visual perception capabilities and strong logic reasoning abilities.
While classic visual reasoners typically rely on complex architectures [117, 61, 116] or are unable to generalize beyond the training dataset [121, 72], recent advancements in large pretrained models have shown that -ision-language models (VLMs) can achieve impressive performance on visual reasoning tasks even under zero-shot settings [104, 52, 51, 49, 48, 3]. Meanwhile, large language models (LLMM)) have also demonstrated robust zero-shot commonsense reasoning abilities on the natural language processing (NLP) applications [8. 15, 109]. Several recent studies have attempted to combine such complementary VLMs and LLMs for visual reasoning. For example, PICa [115] utilizes image captioning models to generate textual prompts for GPT-3 [8]. and adapts GPT-3 to solve the visual question answering (VQA) tasks in an in-context few-shot learning manner. Socratic Models [123] allow VLMs and LLMs to communicate through prompt engineering to unlock zero-shot multimodal reasoning capabilities.
On the premise that current studies have focused on the interactions among heterogeneous models (specifically, among VLM and LLMs), in this work, we shift to examine how to reconcile homoge- neous expert models (e.g., multiple VLMs) with an LLM in a coordinative paradigm. Inspired by the findings in CICERO [65] that LLMs capture strong strategic planning and negotiation abilities in coordinating multiple agents, we propose if Cola, a novel model ensemble approach that utilizes an LLM as the coordinator in between multiple VLMs. A key finding of this study is that given multiple VLMs with different preferred patterns in describing the visual context and predicting plausi- ble answers in natural languages, an LLM can coorlinate and integrate their respective strengths efficiently and effectively. We present two variants of Cola, namely if Cola-FT and if Cola-Zero, where FT corresponds an instruction finetuning approach and Zero stands for an in-context learning approach to adapt the coordinator LLM for visual reasoning. Figure I provides an overview of Cola and conventional model ensemble approaches.
Existing work on model ensembles usually focuses on manipulating model weights [36] or predictions [111, 22]. while remaining cumbersome, if possible, to implement on prevalent end-to-end black box model APIs, like GPT-4 [70], Google Bard, Anthropic Claude, etc. To address this issue, prompt ensembles [107, 106, 73] sample model outputs (e.g.. rationales) in natural languages to boost chain- of-thought reasoning [109]. Recent studies on augmented LLM such as [86, 93, 58] have delved into developing comprehensive strategy that enables LLMs utilize external tools. These tools comprise multiple off-the-shelf models, web search engines, Python functions [97]. and rule- based modules, which are instrumental in performing complex tasks. Despite these efforts, the power of prompt ensembles to aggregate multiple models remains untouched. In contrast, we show that Cola leverages language prompts generated from multiple expert models to make model ensembles.
Systematic experiments demonstrate that Cola performs at the pinnacle of ability on VQA, out- side knowledge VQA, visual entailment, and visual spatial reasoning tasks. Specifically, Cola- FT achieves state-of-the-art performance on A-OKVQA [89], OK-VQA [63]. e-SNLI-VE [21]. and VSR datasets [56], even when compared with methods that adopt larger models or require more training computations. Cola-FT also demonstrates competitive capabilities on VQA v2 [27], and compositional reasoning tasks (GQA [35] and CLEVR [40]). Perhaps surprisingly, we find that Cola-Zero demonstrates comparable performance without finetuning, as an emerging ability of larger language models. Compared Cola-Zero improve the performance substantially across most datasets. They are even effective with the recent large multimodal models like InstructBLIP [18] which embeds a pretrained LLM inside itself. Besides, we conduct a thorough analysis of perturbed VLM caption or plausible answer labels and saliency visualization to investigate how Cola recognizes each VLM's individual functionalities and then performs coordination behaviors. We conjecture that, in principle, any language-expressing reasoning task can be usefully augmented with coordinative language models that learn to aggregate multiple expert models, even via in-context learning.
single VLM and ensemble modeling, both Cola-FT and

---

In summary, our contributions are as follows:
Cola: a novel paradigm that utilizes a language model as a coordinator between multiple vision-language models integrate their respective strengths for visual reasoning ($2). State-of-theeare performance: Cola achieves the pinnacles on challenging suite of diverse visual reasoning tasks and (atasets($3.22)
Systematic analysis: our experiments reveal how Cola comprehends the instruction prompts, then coordinates them to capture impressive visual reasoning capabilities ($3.3, $3.4, $3.6).
2 Cola
We formulate various visual reasoning tasks as a multi-class classification problem for simplicity, Given image e V and question-like prompt q E Q, the reasoner is required to select answer a from the candidate set A {a}. In the case that the reasoner outputs a text sequence Se.g> we map to prediction P(r, q) (T T(fa))) where T transforms text sequences into text embeddings (we use a a -mppet coseet-baae base peasevv model [81] here), and sim denotes cosine similarity.
Ensemble Modeling aggregates multiple models' predictions in order to improve the overall performance (Figure la). For instance, one common practice is averaging over models:
P(e, ④) - Σ (),,)))
(1)
i=1
where q) denotes the prediction of the 7th model input ().
2.1
Cola & Templates
General Prompt Template
Answer the following multiple-choic question by OFA and BLIP's description and their answers to the visual question. OFA and BLIP two differ- ent vision-language models to provide clues.
An overview of Cola shown in Figure lc. We use OFA [104] and BLIP [52] as the VLMs. LLMs include encoder-decoder (FLAN-T5 [16]) and decoder-only (Vicuna-
OFA's description: <OFA caption- BLIP's description: <BLIP caption> Q: <Quest ion>
1.5 [125], Mistral (39]) transformers. We first prompt each VLM to output captions and plausible answers independently. We then concatenate the instruction prompt, the question with choices, captions, and plausible answers to fuse all contexts for the LLM to reason, coordinate, and answer.
OFA's answer: <OFA answer>
BLIP's answer: <BLIP answer>
Choices: <Choices to the question A:
Table 1: LM prompt template. The LM instructed coordinate VLMs.. Each question set defines visual context, question (and choices), and plausible answers.
Image captioning gives important visual context to reason from. We first employ jth VLM describe each image respectively to get visual descriptions e; (v). We use ofa-large for i ooiionnnnnn wesiinggllrre for BLIP, both implemented by the Hugging Face
OFA and blip-imagee-aaree Transformers library [1]0].
Plausible answers by the VLMs to the question provide clues and patterns of VLMs for the LM to consider and coordinate. Similar to captioning, we prompt each 7th VLM using the image-question pair to get plausible answer (v, g). We use ofa-large for OFA and blip-vga-base for BLIP. Following OFA, our prompt template varies by task category. For the VQA tasks, we leave the original question unchanged. For the visual entailment and visual spatial reasoning tasks, our prompt template is does the image describe "<text premi se>"?".
shown in Table 1. First, we designed an instruction prompt for LM to un- derstand the requirement to coordinate VLMs to answer the visual reasoning question. We then concatenate the captions from each VLM model, with the VLM identification labels in natural languages (referred to VL.M caption labels in Figure 3), such as "OFA" 's description: <OFA capt i on>". Next, the question and its plausible answers provided by VLMMs (with similar identi- fication labels referred to VLM answer labels in Figure 3) are concatenated. We follow [16] to
Prompt template
3

---

include the choices of question (for multiple-choice questions in A-OKVQA, e-SNLI-VE, and VSR) and "A:" to prompt for answers. Overall. the prompt for LLM imput is given by:
Prompt(v, 4) Template( ((c:((()))))))))))))())) ()) = =1 .n)).
(2)
More specific prompt templates each dataset provided in Appendix A.7.
2.2 if Cola-FT
Instruction Tuning of Cola is initialized with pretrained checkpoints. Given the question q based on the image v, the LM predicts the answer in the form of sequence
Sv.g = LLM(Prompt(e,
To optimize the LLM, we use the language modeling loss for next-token prediction, with the teacher forcing strategy. We only finetune the LLM (while not the VLMs) follow the common paradigm of ensemble modeling and simplify the method (Figure 1).
Inference deploys the same prompt as Table to align with instruction tuning. We resort the greedy decoding strategy for conditional sequence generation at both instruction tuning and inference.
2.3 Cola-Zero
In-context learning is emerging ability of the LLM models pretrained on documents of long- range coherence. By learning input and output format from demonstration, in-context learners a prompt consisting of input-output
learn to perform downstream task simply by conditioning examples [114]. The coordinator LLM, finetuned on instruction prompts with examples, capable of in-context shot learning and zero-shot learning (see Figures 6 and 7).
Cola-Zero the in-context www-shot/zero-shot learning variant Cola, without instruction tuning. For in-context shotot learning, we modify the prompt (Table 1) to include k input-output examples sampled from the training set. For zero-shot learning, the prompt remains the same as Table 1.
Experiments
First, the experimental setups and basic methods are described in this section. The main quantitative results are then presented Table 2. Next, we analyze qualitative visualizations and scaling to verify the effectiveness of the Cola paradigm in different settings. Further details datasets, training, evaluation, and experimental analysis can be found in Appendix A.
3.1 Baseline Methods
State-of-the-att Methods are in two broad categories, VLM alone, and VLM combined with LLM. In Table 2, for a fair comparison, we detail the techniques (whether finetuning or in-context learning required) used for training VLMs and LLMs, and the number of training epochs.
Ensemble Modeling can be considered the most basic baseline for aggregating VLMs. It represents the base performance that the combination of VLMs can achieve on the target task when not trained. We implement an averaging ensemble (Equation (1)) of cosine similarity between VLM output and each choice of question as ensemble baseline.
3.2 Overall Performance
In Table 2, we first observe that Cola-FT achieves state-of of-te-of-the-ar (SOTA) performance on four datasets (A-OKVQA, OK-VQA, e-SNLI-VE, VSR), with merely epoch of instruction tuning and a medium-sized language model. In contrast, many previous SOTA methods require finetuning more epochs than Cola-FT (e.g., VLC-BERT, PromptCap on A-OKVQA). Some also use much larger language models, such as GPT-3 (175B) [8] and OPT (175B) [124]. Cola-FT outperforms OFA-X on e-SLLL-VE, although the latter is finetuned much more related tasks and data (c.f.

---

Table 2: Overall performance. Model Spec. denotes specification where summarize the detailed VLMs and LMs adopted in each method and their parameters. FT and ICT denote finetuning and in-context learning, respectively. Downward arrows indicate that fewer FT and ICT are more efficient. The accuracy metric varies slightly in different datasets. In A-OKVQA, we report both val/test accuracies, and val accuracy in VQA v2, OK-VQA, e-SNLI-VE, GQA, and CLEVR; test (zero- shot split) accuracy in VSR. Upward arrows indicate higher accuracy is better. We mark the best performance on each dataset with bold font and second-best with underlines.
Vision-language Model
Large Language Model
Method
Accuracy 1
Model Spec.
FT
Model Spec.
ICL
FT
Visual Question Answering (VQA
MctaLM [31]
Pretrained Encoder BLIP-Caption (446M)
350k steps
MctaLM (1.3B)
350k steps
41.1
PNP-VQA [99]
UnifiedQAv2 [43] (LIB)
O-shot
63,3
BLIP-2 [S1]
CLIP [76] (1.2B trainable)
epochs epochs
FLAN-TS (3B)
81.6
BLIP-2 [51]
CLIP [76] (1.2B truinable)
OPT [124] (6.7B)
82.2
Ensemble
68.0
Cola-Zero
BLIP+OPA (384M-472M)
FLAN-TS (1IB) FLAN-TS (1IB)
2-shol
69.1
Cola-FT
epech
83,7
Outside Knowledge Visual Question Answering, Multiple Choice (A-OKVQA)
PrompiCap [32]
OFA (472M)
epochs
GPT-3 (175B)
0-shot
-/73.2
2 [29]
BLIP (384M)
OPT (175B)
0-shol
42.9
Prophet-MC [90]
MCAN-large [118] (56M)
epochs
GPT-3 (175B)
16-shot
76.4
Ensemble
54.9
Cola-Zero
FLAN-TS (11B)
0-shot
65.4/61.6
BLIP+OFA (384M+472M)
Cola-Zero
FLAN-T5 (1IB)
2-shot
70.4/ 66.5
Cola-IT
FLAN-TS (11B)
epoch
77.7 / 74.0
Cola-Zero
FLAN-TS (1IB)
0-shot
6
Cola-Zero
FLAN-TS (11B)
2-shol
72.3
Cola-IT
InstructBLIP [18]
FLAN-T5 (11B)
epoch
78.1 78.1
Cola-Zero
XL+XXI
Vicuna (7B)
2-shot
63.9/63.0
Cola-IT
(3B+11B)
Vicuna (7B)
epoch
66.6
Cola-Zero
Mistrul (7B)
2-shot
6
Cola-FT
Mistral (7B)
epoch
7
Outside Knowledge Visual Question Answering, Direct Answer (OK-VQA)
PromptCap [32]
OFA (472M)
epochs epochs
GPT-3 (175B)
0-shol
58.8
Prophet [90]
MCAe-large [118] (56M)
GPT-3 (175B)
16-shot
61.1
Ensemble
39.2
Cola-Zero
FLAN-T5 (1IB)
0-shot
39.4 .
BLIP+OFA (384M-472M)
Cola-Zero
FLANNTS (11B)
2-shol
39.4
Cola-IT
FLAN-TS (11B)
epoch
62.4
Visual Entailment (e-SNLI-VE)
G-G [42]
UNITE (86M)
400 epochs
GPT-2
400 epochs
79.5
OFA-X [74]
OFA (472M)
10 epochs
80.9
Ensemble Cola-Zero
48.8
FLAN-TS (1IB)
O-shot
56.2
BLIP+OFA (384M+472M)
Cola-Zero
FLAN-TS (11B)
2-shol
57.8
Cola-FT
FLAN-TS (1IB)
epoch
81.6
Visual Spatial Reasoning (VSR)
VisualBERT [53]
VisualBERT (1 (11)
100 epochs
$4.0
LXMERT [98]
LXMERT (1 10M)
100 epochs
63.2
ViLT [44]
VILT (88M)
30 epochs
62.4
Ensemble Cola-Zero
$1.4
FLAN-TS (1IB) FLAN-TS (1IB)
0-shot 2-shot
55.8
BLIP+OFA (384M+472M)
Cola-Zero
54.9
Cola-FT
FLAN-TS (11B)
epoch
67.0
Compositional Question Answering, Real Images (GQA)
BLIP [52]
BLIP (384M)
41.7
OFA [104]
OFA (472M)
58.0
VisProg [30]
ViLT (88M)
GPT-3 (175B)
8-shot
50.5
Cola-FT
BLIP+OFA (384M+472M)
FLAN-TS (11B)
epoch
60.3
Compositional Question Answering, Synthetic Images (CLEVR)
(IB(
33.7
InstructBLIP [18]
XXL (1IB)
16.6
Cola-Zero Cola-FT
InstructBI.IP
FLAN-TS (1IB)
2-shol
34.4
XL+XXL (3B+1IB)
FLAN-TS (11B)
epoch
54.3
5

---

STE-CARLO
Question
What type of shot the man hitting?
What appliance an appliance that is highly decorated?
Does this image describe "puppy running after a stick grass'
Does this image describe "The truck away from the elephant"
OFA caption
tennis player to tennis player during men's singles second round match at
refrigerator covered variety of stickers.
coyote seen undated file photo. (credit:
elephant loaded onto a truck in yangon. photo: afp
BLIP
man blue shirt playing tennis
a refrigerator with pictures on it
a dog running through the grass field
man riding a motorcycle with a truck behind him
caption
Choices
['orrhaand, 'backhand", serve', 'drossott]]
['mixer,, 'stove/oven", refrigeeraoo", mmcrroavve]]
('yes', 'maybe', 'no')
['yes', 'no']
OFA answer
backhand
stove/oven
yes
yes
BLIP
forehand
microwave
Cola-Zero
forehand
stove/oven
no
no
answer
Cola-FT
forehand
stove/oven
maybe
no
answer
Cola-FT
backhand
microwave
maybe
yes
answer
(swapped
answer labals)
Figure 2: Qualitative examples. The correct choices are underlined. Leftmost: a commonsense question example of A-OKVQA; LLM follows the answer of BLIP. Left: visual question example of A-OKVQA; LLM follows the answer of OFA. Right: an example of e-SNLI-VE: LLM chooses another option after coordination. Rightmost: an example of VSR; LLM predicts based on the caption of OFA and the answer of BLIP. Cola-Zero answers are inferenced in zero-shot settings. The bottom row, Cola-FT (swapped VLM answer labels), indicates that the LLM follows the answer of certain VLMs based on their separate functionalities. LLM answers are associated with the distribution of VLM answer labels.
Cola-FT is trained each one dataset only in Table 2). In addition, the lighter variant Cola-Zero also achieves comparable performance to most baseline methods through in-context few-shot and zero- shot learning, without training any model parameter. To evaluate the performance of Cola with large multimodal models that embed large language models inside, we also assembled InstructBLIP [18] models based on FLAN-T5 XL and XXL and tested on A-OKVQA dataset. In Table 2, we report the multiple-choice accuracies on A-OKVQA- See Appendix A.4 for direct answer results.
3.3 Qualitative Examples
In Figure 2, exhibit several qualitative examples. The language coordinator determines the correctness VLM plausible answers implicitly, given their captions and the caption and answer labels. The leftmost example (a tennis player playing) demonstrates case when captions not informative to guide the LLM for predictions. Between OFA and BLIP's plausible answers, the LLM follows the answer of BLIP. In contrast, in the left example (an next fridge), again with trivial captions, the LLM follows OFA's plausible answer instead.
It's all plausible answers, captions, VLM answer/caption labels, and the world knowledge the LLM encodes in itself, that contribute to the final decision the language coordinator. The rightmost example presents the scenario of inconsistency between captions and answers. OFA describes the image as "an elephant loaded onto truck in yangon." "Though, it agrees that "the truck is away from the elephant". With Cola-FT, The LLM coordinates OFA's correct caption and BLIP's correct answer to make a reasonable prediction.
Notably. we observe a scenario in which captions can be more informative than plausible answers to guide LLM. The right example (a puppy running) presents an uninformative image. Though neither OFA nor BLIP succeeds in answering the question, the LLM chooses to answer with "maybe" based on the given visual context. See Appendix A.5 and Appendix A.6 for more analysis on qualitative examples and failure cases.

---

3.4 Coordination Analysis
Overall, Figure 3 validates the efficacy of Cola to coordinate VLMs. All the ex- periments use the same prompt template as Table unless otherwise stated, On A-OKVQA validation set, the per- formance of single VLM (w/o FLAN-
OFA only FLAN-T5
81
73.97
BLIP FLAN-T5
02
73.27
VLMs plausible
#3
72.4
T5) is 50.83% for BLIP, 54.75% for
VLMs' captions only
84
63.78
OFA. validate the effectiveness of multi-VLM collaboration, we first ab- late single-VLie VLM variants of Cola-FT, shown as #1 (OFA only, without BLIP) and #2 (OFA only, without OFA) from the top. As expected, both fall behind Cola-FT (#8) by a large margin. With both VLMs, ablate VLMs' captions (#3) and VLMs' plausible answers (#4), which reveal that plausible answers are much more significant in helping the LLM answer visual reasoning questions. Next, we perturb caption labels by swap- ping the VLM caption labels instruc- tion tuning and evaluation (#5), specifi- cally "OFA's description: "nnd "BLIP's description: ", by chance of 50%. Un- der such settings, the LLM fails ac- quire the preferred patterns of VLM for captioning, though the overall visual context is preserved. The results underperform Cola-FT, which verifies that VLM caption labels improve Cola-FT performance. Notably, the VLM (plausible) answer labels are more important to LLM's decision: a considerable gap exists between (#6) and Cola-FT, In #6, the LLM fails to learn the separate functionalities of VLM when answer labels are perturbed. This highlights that the performance gained from the coordination between BLIP and OFA, but not the strong reasoning capabilities of the LLM, FLAN-T5.
caption
#5
77.29
Eval: perturb VLM caption labels
FT: perturb VLM answer labals Eval: parturb VLM answer labels
76.16
FT: correct VLM answer labels Eval: swap VLM answer labels
FT: correct VLM answer labels Eval: correct VLM answer labels
(Cola-FT FT)
65
70
75
Accuracy (%)
Figure 3: Ablation study results using single VLM (#1, #2 from top), VLMs' plausible answers only (w/o captions, #3), VLMs' captions (w/o plausible answers, #4), perturbed VLM caption/answer labels at instruction tuning (#5, #6), and swapped answer labels evaluation (#7). In #6, the coordination prior cannot be learned by the LLM. In #7, the coordination prior can learned by the LLM, but cannot be properly applied at evaluation. FT: instruction tuning: Eval: evaluation.
Naturally, we ask what if the LLM can learn the patterns each VLM answers, but they cannot apply at inference? We input correct VLM answer labels at instruction tuning and swap labels at evaluation (#7). Consequently, #7 falls behind Cola-FT with smaller but still considerable margin. The results suggest that learning and applying the separate functionalities of VLMs is important for the coordinator LLM to make predictions. See Appendix A.9 for more ablation studies.
3.5 Scaling f Cola
Methods
A-OKVQA
e-SNLI-VE
Methods
A-OKVQA
e-NNLL-VE
OFA-base (1)
45.76
$2.60
OFA-liny
39.03
buse
OFA OFA-meding medium
42.45
$1.04
OFA-bace hase (3)
45.73
52.13
OFA-base
45.76
52.60
Ensemble (majority voting)
44.79
52.71
Ensemble (uujority voling)
46.71
53,94
Ensemble (average)
46.04
52.25
Ensemble (average)
46.62
54.41
Cola-Zero (2-shot)
47.71
54.42
(2-shot)
49.37
$7.63
Cola-I7
Cola-FT
54.26
63.68
Table 3: Performance of ensemble methods based three identical models.
Table 4:
Performance of ensemble methods based on three different models.
Scaling Cola with More VLMs. By decoding the top-k (k=5) results from three identical (OFA-base) models on the A-OKVQA validation set, both the answers and captions may exhibit slight variations. Cola demonstrated significant performance improvements compared to single VLM or ensemble, as shown Table Furthermore, the performance gap between the ensemble baselines and Cola based on three different models OFA-tinyymeddumm/bee) is even more substantial, depicted in Table 4.

---

We further scale the number of OFA-base and find that Cola-FT satures VLMs (49.77%) and Cola-Zero (2-shot in-context learning) satures at 3 VLMMs (47.71%). We observe that long input harms the performance of Cola-Zero, which negative for scaling with more VLMMs (Figure 4). L.Ms with larger context window size [12, 68, 9] are promising to further improve the performance of Cola, which we leave for future works.
Cola-FT Cola-Zero (2-shot)
Acsuracy
48
Scaling Model Size. We conduct experiments on scaling the coordina- tor LLM size to see if there are ramifications when operating scale. Figure 5 reveals that Cola-FT performance increases as the LLM (FLAN-TS) model size increases. Notably, Cola-FT/sman with only 80M parameters, could achieve 65% MC accuracy on A A-OKVQA validation set, which is far beyond our baseline methods (55%). Cola-Zero, under the in-context learning paradigm, achieves competitive performance when the model grows to billion-panee pallion-parameter scale. This observation on Cola-Zero can be regarded as a proof-of-concep! that potentially reveals Cola-Zero's emerging abilities (inherited from FLAN-T5 [16]) visual reasoning tasks at relatively large scale, Cola-FT is effective with small models, but Cola-Zero is an emerging ability on larger models only.
larger
47
Models (OFA-Base)
Figure 4: Scaling of Models.
78
Cola-FT
Cola-FT
Cola-Zero, in-context 2-shot leamiing
76
Cola-Zero, zero-shot learning
80
Accuracy (96
74
Cola-Zero (in-context learning)
-70
72
Accuracy
80
70
50
68
40
66
Cola-Zery (zero-shot learning)
80M Small
250M Base
3B
11B
XL
XXL
100
10'
102
10a
10
Model size (# parameters)
Exemplar Training Data Size
Figure 5: Cola performances versus the Figure 6: Low-data Cola-FT and Cola-Zero perfor- mances. r-axis is distorted for optimal display.
LLM (FLAN-T5) sizes.
In-context Learning & Low-data Instruction Tuning. We conduct experiments on different data scales to verify Cola's performance varying from zero-shot to full-shot under in-context learning and full-finetune paradigm. As shown in Figure 6, with Cola-Zero, few-shot exemplars substantially improve performance compared to zero-shot learning. As [16, 108] revealed, exemplars potentially help the model better understand the output format and understand the instructions Cola-Zero for in-context few-shot learning outperforms zero-shot learning by a large margin, being on par with low-data Cola-FT without instruction tuning. We also observe Cola-FTT's substantial performance gain when finetuning shots increase 1000 and beyond.
Table 1.
3.6 Saliency Visualization
As shown Figure 7, we visualize the importance of the input prompt tokens by graudegradient saliency feature attribution [19], implementing with Ecco [2]. The input tokens that are more relevant to predict the output token "grass" are highlighted in darker colors. In the given example, both Cola-FT and Cola-Zero predict the correct answer and find the relevant clues from visual context and plausible answers. Figure 7(b) shows that Cola-Zero attributes the output more to the instructions in the prompt template. This explains Cola-Zero's 's competitive performance, consequence of FLAN instruction tuning [108]. After instruction tuning, Cola-FT focuses more on the most informative parts of input: the question, choices, well as VLMs plausible answers.
3.7 Can Cola-FT Explain its Answers?
We modify the prompt template of Cola so that the model would output the logical conduction process, allowing us to observe the specific behaviors of the LLM during its coordination of VLMs.
8

---

Question and choices
Cola-FT Festure Attribution
Ansoee the following multipln choice question by CRRA anb D2I1P a descripttion and their anawecc to the visual comation. OFA and 2LIP ArM
two di flreett vision- Sasoi--aanggan lanquage modell to provide cluda. GEA
deseription: piesic tuble in the park BLEE" description: pienic
table with a pienic table and what type
berond table and grill? OEA'
Elatgrasay
tublu and grill, BEIE anSVAE Cholceaa
Bandyy auday. A:sras Spadi geasayc/a>
Plausibbe answers
(b) Cola-Zero Feature Atribution
foll loving multeppe
description and thaks
gomatiion
two Wis Lon- language modela to peovide cluca. OPA description: picnic table in the pazk EL.22' description: pienic
table vi th picnic calrb und s-t-------- & Mhat Kype of casrain La bayand the table and grill? DEA
SEaREY acea La bayand the
table and
3 Choico
Handy
auddy
Syady grassys/s>
In-context learners focus Instructions and VLM labels.
Figure 7: Visualization of input token saliency. We visualize the relevancy between input tokens and the output token "grass" by feature attribution [19]. The more salient tokens are highlighted in darker boxes. Cola-FT focuses on the question, choices, and VLMs' plausible answers in (a). While as shown in (b), Cola-Zero pays extra attention to instructions and VLM labels, as a consequence of FLAN-T5 instruction tuning [16].
We finetune Cola-FT to output rationales before answers, by A-OKVQA ground truth rationales. In the modified prompt, we ask the model to provide rationales. For the leftmost example of Figure Cola-FT outputs "Rationale: People might sit here to rest. The umbrellas are on the riverwalk. The answer is:rest ". OFA gives a reasonable answer (but out of choices) the question while BLIP gives an irrelevant answer. In this case, both answers are wrong. However, either Cola-Zero or Cola-FT is able to infer from captions and plausible answers to give the correct answer "to rest". The rationale suggests that the LLM understands the scene that the umbrellas are on the riverwalk and guesses that people might sit here to rest based on commonsense. The final answer correct. For the leftmost example of Figure 11, the rationale output is "The bike is parked in no parking zone. The bike parked next to a pedestrian crossing sign. The answer is:no parking". Both VLMM are wrong in their plausible answers. OFA's answer "boating" is semantically correct the correct answer is "kayaking", though it's not the correct answer because this is a multiple-choice question. Cola-Zero gives wrong answer "OFA" which is obviously wrong because "OFA" is the name of one of the VLMs given in the prompt and it's out of the choices too. However, Cola-FT gives the correct answer "kayaking", recognizing the correct choice based on prompts of captions and plausible answers after being finetuned. Even though the OFA and BLIP captions fail to identify that the people in the water a canoe. The LLM identifies that the people in the water associated with the canoe. The rationale is valid and helpful, though repetitive. The final answer is correct.
To force the LLM to output rationale does not improve the reasoning performance of Cola (w/t rationale 74.3% vs. w/o rationale 77.7%, on A-OKVQA val set). This might be attributed to the low-quality ground truth rationales provided by the A-OKVQA dataset that we use to train the LLM. Such rationales are just short and objective descriptions of the scene, without suggesting the underlying outside knowledge to answer the question. Therefore, training the LLM to output rationale is harmful, though it derives insights into the LLM's 's behaviors during reasoning.
3.8 Does Cola-FT Transfer across Tasks?
Table 5: Cola-FT is generalizable across out-of- distribution tasks. In most cases, the performances surpass Cola-Zero (in-context 2-shot learning re- sults in brackets) on target datasets.
We examine Cola-FT's generalization ability across tasks. From Table 5, we observe the zero-shot performances on target datasets after instruction tuning certain source dataset. Although each dataset varies in question types and prompt templates (see detailed compar- isons in Appendix A.7), we find that Cola- FT maintains competitive performance when zero-shot transferred to new task, outperforming Cola-Zero in-context 2-shot learning and ensemble baselines (see also Table 2).
Finetuned on
A-OKVQA val
e-SNLI-VE
VSR test
A-OKVQA
77.7 (70.4)
58.7
57.6
e-SNLI-VE
71.2
81.6
(57.8)
51.7
VSR
71.4
61.4
66.9 (55.8)
4 Related Work
Visual Reasoning. Beyond unimodal reasoning tasks such question answering (QA) [100, 14, 119, 7], visual reasoning extends high-level cognition to visual domains, requiring an intelligent agent to derive rational solutions [40, 35, 84, 120, 38, 1 12]. Several tasks have been introduced to address

---

visual reasoning, such as VQA [1], in which models are expected to provide answers to questions related to an image, and visual entailment [113], where the model is required determine if a text description is consistent with the visual content provided.
Classic visual reasoning methods employ an image encoder along with reasoning block that utilizes attention mechanisms [102, 121, 122, 105], neuro-symbolic methods [101, 117, 61], or external knowledge [62, 28, 13].
Recent progress in large pretrained models has led to the development of LLMs that capture excep- tional commonsense reasoning capabilities [78, 16, 15]. These LLMs can potentially replace the reasoning module in visual reasoning tasks, and LLMs' lack of perception can be compensated by incorporating multiple VLMs trained on different domains [76, 104, 52]. However, there is still lack of research on how to harness the collective power of these separate VI.Ms for visual reasoning tasks. More related works are in Appendix B.
Model Ensemble. Model ensemble is a powerful machine learning technique that combines the predictions of multiple models to improve the overall performance of given task [20]. The variance and bias of the final predictions decrease, resulting in a more robust and accurate model [83]. To this end, common methods include averaging [[]]], voting [37], interpolation [36], weighting the
predictions based on model performance [22], or stacking the models [10].
Ensemble methods have been challenging for generative tasks like visual reasoning, where simple combination is not applicable to heterogeneous models due to their enormous and varying input/output token spaces. To address the issue, Socratic Models (SMs) [123] use prompt engineering to guide the heterogeneous pretrained multimodal models through natural language discussions. With a similar goal, [54] proposes a closed- loop iterative consensus optimization method to utilize the strengths of individual models. However, previous methods do not fully adapt to the intrinsic patterns different models, particularly in the visual reasoning scenario. Recent studies, such as CICERO [65], have shown that LLMS possess strong social intelligence in coordinating multiple agents, which inspires us reorganize pretrained mixed modal models with a focus adapting LLMs. More recently, Toolformer [86] and HuggingGPT [93] further demonstrate LLMs' abilities to leverage, coordinate, and incorporate the results from external sources such other models or even APIs to solve complex tasks. While the external tools are called in sequential order in existing work, we study coordinating multiple tools (specifically, expert models) in parallel in this work.
5 Discussion
Question Format. Datasets like VQA v2 and OK-VQA contain open-ended questions. while A-OKVQA, e-SNLI-VE, and VSR use multiple-choice. Converting VQA v2 and OK-VQA to classification introduces complexities for traditional ensemble methods, as evident in Table 2. Classic methods struggle with generative models like API-based GPT-4 underscoring Cola's value an end- to-end ensemble strategy for extensive (vision-)lend -language models. Moreover, Cola-Zero's 's efficiency also relies on the question format- it's easier for LLMs to answer when given choices like in A-OKVQA. Conversely, Cola-FT finetunes LLMs to discern answer formats (Figure 7).
Limitations. Visual reasoning is diverse topic. This work demonstrates the first step toward applying end-to-end language models for visual reasoning. While the LLMs perform well on the discussed datasets, there is a large body of visual reasoning tasks to evaluate in future works, such as intention prediction and rationale explanation.
Future Works. First, exploring the use of non- con-parametric tools for visual reasoning would be useful to enhance Cola's performance. Second, Cola's use can be extended to other reasoning and planning tasks, such as image generation and action planning. by coordinating multiple models in parallel. Third, by improving inter-model communications, Cola can be more interpretable and safe for high-stakes applications.
Conclusion. In this paper, we have proposed a novel paradigm for visual reasoning that harnesses the power of multiple VLMs by utilizing a coordination mechanism, where an LLM acts as a coordinator who communicates with VLMs to integrate their respective strengths. Experiments show that reasoning performance is substantially improved by LLM finetuning or in-context learning. Our results provide a promising step towards building multi-comppnent intelligent systems that capture multimodal reasoning capabilities in a human-like way.
10

---

Acknowledgements
This research/project is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISGG-PPDD-2222-1--099)) Besides, this project is sup- ported by NTU NAP, MOE AcRF Tier (MOE-T2EP2021--0122 and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as advise from the industry partner(S).
References
[1] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick. Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision, 123:4-31, 2015. 10, 25
[2] J Alammar. Ecco:
An open source library for the explainability of transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference Natural Language Processing: System Demonstrations, pages 249-257, Online, August 2021. Association for Computational Linguistics. 8
[3] Jean- bean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736. 2022.
[4] Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro-symbolic visual reasoning: Disentangling. In International Conference Machine Learning, pages 279-290. PMLR, 2020. 25
[5] John R Anderson, C Franklin Boyle, and Brian Reiser. Intelligent tutoring systems. Science,
228(4698):456-462 462, 1985. 2
[6] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255: 2021. 22
[7] Alexander Bondarenko, Magdalena Wolska, Stefan Heindorf, Lukas Blübaum, Axel- Cyrille Ngonga Ngomo, Benno Stein, Pavel Braslavski, Matthias Hagen, and Martin Potthast. Causalqa: A benchmark for causal question answering. In ACL., 2022. 9, 25
[8] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020. 2, 4, 22, 24, 25
[9] Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:1 1079-11091, 2022.
[10] Angelos Chatzimparmpas, Rafael M Martins, Kostiantyn Kucher, and Andreas Kerren. Stack- genvis: Alignment of data, algorithms, and models for stacking ensemble learning using perfor- mance metrics. JEEE Transactions on Visualization and Computer Graphics, 1772)1147711557, 2020. 10
[11] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 23
[12] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context win- dow of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595.
2023. 8
[13] Zhuo Chen, Yufen Huang, Jiaoyan Chen, Yuxia Geng, Yin Fang. Jeff Z. Pan, Ningyu Zhang, and Wen Zhang. Lako: Knowledge-driven visual question answering via late knowledge-to-text injection. ArXiv, abs/2207. 12888, 2022. 10, 25
[14] Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. Coarse-to- to-fee question answering for long documents. In ACL, 2017. 9, 25
11

---

[15] Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. 2, 10, 25
[16] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-iiettueed language models. arXiv preprint arXiv:22 /0. 14/6, 2022. 3, 8, 9, 10, 25
[17] Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi qa: A benchmark for information seeking question answering in typologically diverse languages. Transactions of the Association for Computa- tional Linguistics, 2020. 25
[18] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose langunelanguage models with instruction tuning. ArXiv, abs/2305 abx/2305.066000, 2023. 2, 5,5,
[19] Misha Denil, Alban Demiraj, and Nando De Freitas. Extraction of salient sentences from labelled documents. arXiv preprint wrXiv:441.66111 2014. 8,
[20] Thomas G Dietterich. Ensemble methods in machine learning. In International workshop
multiple classifier systems, pages 1-15. Springer, 2000. 10, 25
[21] Virginie Do, Oana-Maria Camburu, Zeynep Akata, and Thomas Lukasiewicz. e-snli-ve: Corrected visual-textual entailment with natural language explanations. arXiv preprint
arXiv:2004.03744, 2020. 2, 19
[22] Alican Dogan and Derya Birant. A weighted majority voting ensemble approach for classifica- tion. In 20/9 4th International Conference on Computer Science and Engineering (UBMK),
pages 1-6, 2019. 2, 10
[23] Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. arXiv preprint arxiv:/907.09//90 2019. 25
[24] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012. 15723, 2020. 24
[25] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 2021. 25
[26] Dan Goldwasser and Dan Roth. Learning from natural instructions. Machine learning,
94(2):205-232, 232200 2014. 24
[27] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017. 2, 19
[28] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Kat: knowledge augmented transformer for saiion-andllanguage arXiv preprint 12Xiv:2/12.0861 (4, 2021. 10, 25
[29] Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.. 10846, 2022. 5
[30] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reason-
ing without training. ArXiv, abs/2211.11559) 2022. 5
[31] Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shum- ing Ma. and Furu Wei. Language models are general-purpor interfaces. arXiv preprint
arXiv:2206.0633 06336, 2022.
[32] Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah Smith, and Jiebo Luo. Promptcap: Prompt-guided aawre image captioning. arXiv preprint arXiv:2211. 1.2211.0060 1.09699, 2022. 5, 20
[33] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lx, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language not all you need: Aligning perception with language models. arXiv preprint
2023. 25
12

---

Lim. [34] Ziqi Huang, Hongyuan Zhu, Ying Sun, Dongkyu Choi, Cheston Tan, and A diagnostic study of visual question answering with analogical reasoning. In 2021 IEEE International Conference on Image Processing (ICIP), pages 2463-2467. IEEE, 2021. 25
[35] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019. 2, 9, 19, 25
[36] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Korblithh, Ali Farhadi, and Ludwig Schmidt. Patching open apen-vocabulary models by interpolating weights. arXiv preprint arXiv:2208, 05592, 2022. 2, 10
[37] Anmol Jain, Aishwary Kumar, and Seba Susan. Evaluating deep neural network ensembles by majority voting cum meta-learning scheme. In Soft Computing and Signal Processing: Proceedings of 3rd ICSCSP 2020, Volume 2, pages 39-37. Springer, 2022. 10
[38] Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong. Robert D Hawkins, and
Yoav Artzi Abstract visual reasoning with tangram shapes. In EMNLP, 2022. 9
[39] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint wxXiv:2310006822 2023. 3
[40] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017. 2, 9, 19, 25
[41] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017. 25
[42] Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. e-vil: A dataset and benchmark for natural lan- guage explanations in vision-language tasks. In Proceedings of the IEEF/CVF International
Conference on Computer Vision, pages 1244-1254, 2021. 5
[43] Daniel Khashabi, Yeganch Kordi, and Hannaneh Hajishirzi. Unifiedqa-v2: Stronger general- ization via broader cross-format training. arXiv preprint arXiv:2202. 12359, 2022.
[44] Wonjac Kim, Bokyung Son, and Ildoo Kim. Vilt: and-language transformer without convolution region supervision. In International Conference on Machine Learning, pages 5583-5594. PMLR, 2021.
[45] Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pages 6265-6274. PMLR, 2021. 25
[46] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising requence-to-sequees pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:1910.13461 2019. 24
[47] Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-exxeee are domain generalizable learners. arXiv preprint arXiv:2206. 04046, 2022. 25
[48] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint
arXiv:2306.0544
2023. 2
[49] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter:
A multi-modal model with in-context instruction tuning, 2023.
[50] Chunyuan Li, Zhe Gan, Zhengyuan Yang. Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao. Multimodal foundation models: From specialists to general-purpose purpose assistants. arXiv preprint arXiv:2309. 10020, 2023. 2
[51] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint
231112230. 2023. 2,
13

---

[52] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language- image pre-traiingg for unified aision-language understanding and generation. arXiv preprint arXiv:2201. 12086, 2022. 3,5, 10, 19, 25
[53] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 3r0iv:990.00355) 2019. 5
[54] Shuang Li, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, and Igor Mordatch. Composing ensembles of pre-trained models via iterative consensus. ArXiv, abs/2210. 1222200.111211 1122,, 2022. 10, 25
[55] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 19
[56] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. arXiv preprint
Stti:2205.003366 2022. 2, 19
[57] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffll Few-shot parameter--effiient fine-tuning is better and cheaper than in-context learning. arXiv preprint arXiv:2205, 05638, 2022. 21, 25
[58] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Rei Chang, Ying Nian Wu, Song- Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023. 2
[59] Mikotaj Malkiński and Jacek Mańdziuk. Deep learning methods for abstract visual reasoning: A survey on raven's 's progressive matrices. arXiv preprint arXiv:2201. 2022. 25
[60] Mikotaj Malkiński and Jacek Mańdziuk. A review of emerging research directions abstract visual reasoning. arXiv preprint arXiv:2202.
2022. 2,
[61] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes words and sentences from natural supervision. ArXiv, abs/1904. 12584, 2019. 2. 10, 25
[62] Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-baie based vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111-14121, 2021. 10, 25
[63] Kenneth Marino, Mohammad Rastegari, Ali Farhadi. and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pages 3195-3204, 2019. 2,
[64] John McCarthy et al. Programs with common sense. RLE and MIT computation center Cambridge, MA, USA, 1960. 24
[65] Fundamental AI Research Diplomacy Team Meta, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 2022. 2, 10, 26
[66] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint 1r. 2021. 24
[67] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsoureing instructions. arXiv preprint arXiv:2104.0877.
2021. 24
[68] Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. arXiv preprint arXiv:2304. 08467, 2023.
[69] Hyacinth S Nwana. Intelligent tutoring systems: an overview. Artificial Intelligence Review, 4(4):251-277, 1990. 2
[70] OpenAI. Gpt-4 technical report. 2023. 2
[71] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint 2203v:2203.000 2022. 22, 24
14

---

[72] Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visu- alcomet:
Reasoning about the dynamic context of still image. In European Conference on Computer Vision 2020. 2, 25
[73]
Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304.05970 2023.
[74] Björn Plister, Jakob Ambsdorf, Lukas Braach, Jae Hee Lee, and Stefan Wermter. Harnessing the power of multi-task pretraining for ground-truth level natural language explanations. arXiv preprint (tti::222.0042333 2022. 5, 20
[75] Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W Bruce Croft, and Mohit Iyyer. Open-retrieval conversational question answering. In ACM SIGIR, 2020. 25
[76] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
pages 11488-763.. PMLR, 2021. 5, 10, 25
[77] Jack W Rae, Sehastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112. 1446. 2021. 25
[78] Colin Raffll, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., (1114)):1-6) 2020. 10, 25
[79] Nazneen Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself!
leveraging language models for commonsense reasoning. In ACL, 2019, 25
[80] Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 2019. 25
[81] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. 3
[82] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583 85955555,555 2021. 25
[83] Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Intendisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4)xe1299 2018. 10, 25
[84] Shailaja Keyur Sampat, Maitreya Patel, Subhasish Das, Yezhou Yang, and Chitta Baral. Reasoning about actions over visual and linguistic modalities: survey. arXiv preprint arXiv:2207.07568, 2022. 9, 25
[85] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2 10.08207, 2021. 24
[86] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04771, 2023. 2, 10
[87] Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676 2020. 24
[88] Benedikt Schmidt, Reuben Borrison, Andrew Cohen, Marcel Dix, Marco Gärtler, Martin Hollender, Benjamin Klöpper, Sylvia Maczey, and Shunmuga Siddharthan. Industrial virtual assistants: Challenges and opportunities. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, pages 994-801, 2018. 2
[89] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv preprint 220.v:2206.01718. 2022. 2, 19
15

---

[90] Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge-base visual question answering. arXiv preprint arXiv:2303.01900 2023. 5
[91] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of- layer. arXiv preprint arXiv: (tt:::700.06533. 2017. 25
[92] Noam Shazeer and Mitchell Stern Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604 PMLR, 2018. 19
[93] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 2023. 2, 10
[94] Derek Sleeman and John Seely Brown. Intelligent tutoring systems. London: Academic Press. 1982.
2
[95] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 13:3008-3022 2020. 22
[96] Md Arafat Sultan, Shubham Chandel, Ramón Fernandez Astudillo, and Vittorio Castelli. On the importance of diversity in question generation for qa. In ACL, 2020. 25
[97] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. urXiv preprint arXiv:2303.0m 00..0000.12... 128, 2023.
[98] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: 07490, 2019. 5
[99] Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training.
arXiv preprint arXiv:2210.0877:: 2022.
[100] Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. In Rep4NLP@ACL, 2016.
9, 25
[101] Aisha Urooj, Hilde Kuehne, Bo Wu, Kim Chheu, Walid Bousselham, Chuang Gan, Niels Lobo, and Mubarak Shah. Learning situation hyper-graphs for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14879-14889, 2023. 10
[102] Mohit Vaishnav and Thomas Serre. Gamr: A guided attention model for (visual) reasoning. In The Eleventh International Conference Learning Representations, 2022. 10
[103] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence, 39(4):652 663, 2016. 2
[104] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-seuueeee to-seuencce learning framework. In International Conference on Machine Learning, pages 23318-23340. PMLR, 2022. 2, 3, 5, 10, 19, 25
[105] Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre- training with mfxture-of-modallee modaatty-eeeppertrt..... ArXiv, abs/211 11 ,02358, 2021. 10, 25
[106] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Rationale- augmented ensembles in language models. arXiv preprint arXiv:2207.00774 2074777777777 2022.
[107] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. //111, 2022. 2
[108] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 2109:2200.0065.. 2021. 8, 24
16

---

[109] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning large language models. arXiv preprint arXiv:2201. /0N0 2022. 2
[110] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowic7, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: -fate-o--the-rr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics.
[111] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 23965-23998, PMLR. 17-23 Jul 2022. 2. 10
[112] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural
Information Processing Systems (NewrIPS), 2021. 9
[113] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. ArXiv, abs/1901.06700 2019. 10, 19, 25
[114] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2 (tt:://1.020800 1.02080, 2021.
[115] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In AAAI, 2022. 2, 25
[116] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. ArXiv, 1oss/990000442, 2019. 2, 25
[117] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenen- baum Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. ArXiv, abs/1810.02338. 2018. 2, 10, 25
[118] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6281-6290, 2019.
[119] Munazza Zaib, Wei Emma Zhang, Quan Z Sheng, Adnan Mahmood, and Yang Zhang. Con- versational question answering: A survey. Knowledge and Information Systems, 2022. 9,
25
[120] Rufai Yusuf Zakari, Jim Wilson Owusu, Hailin Wang, Ke Qin, Zaharaddeen Karami Lawal, and Yuezhou Dong. Vqa and visual reasoning: An overview of recent datasets, methods and challenges. arXiv preprint 2xXiv:2212.13296
2022. 2, 9, 25
[121] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoming. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17244,724, 2018. 2, 10, 25
[122] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In Neural Information
Processing Systems, 2021. 10, 25
[123] Andy Zeng. Adrian S. Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Peter R. Florence. Socratic models: Composing zero-shot multimodal reasoning with language. ArXiv, abs/2204.00598. 2022. 2, 10, 25
[124] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiy preprint arXiv:2205.0100 2022. 4,
17

---

[125] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-hench and chatbot arena. arXiv preprint arXiv:2306.05683 2023. 3
[126] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mfxture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022. 25
[127] Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. Retrieving and reading: A comprehensive survey open-domain question answering. arXiv preprint arXiv:2 00774, 2021. 25
18

---

A Experimental Details
In this section, we elaborate on our training and evaluation details, prompt templates, and more qualitative examples for analysis.
A.1 Datasets
Our experiments are conducted on challenging suite of three diverse visual reasoning tasks, including outside knowledge VQA, visual entailment, and visual spatial reasoning. For each task, we select the following dataset respectively.
Visual Question Answering v2 [27] (VQA v2) is large-scale benchmark containing over 1 million images from the COCO dataset and more than 250,000 human human-generann generated question-answw pairs. The dataset is designed to test the ability of machine learning models to understand both the visual content of image and the meaning behind natural language questions. The questions in VQA v2 cover a wide range of topics and are often open-ended, requiring models to reason and generalize about the world. VQA v2 has been widely used to evaluate the performance of state-of-the-ar models in the field of computer vision and natural language processing.
Augmented Outside Knowledge VQA [89] (A-OKVQA) contains about 25k questions paired with both multiple choice (MC) answer options. Unlike most existing VQA datasets, the questions in A- OKVQA cannot often be answered by querying the knowledge basc, but rather involve some type of commonsense reasoning and outside knowledge about the situation portrayed in the image.
Outside Knowledge VQA [63] (OK- VQA) includes more than 14,000 questions that require external knowledge to answer. The answers provided in free-text direct answer form. Both A-OKVQA and OK- VQA sample images from the COCO dataset, with no overlapping.
e-SNLI-VE [21] dataset is an extended version of SNLI-VE dataset [1 [113], which contains about 190k question pairs and human-annotate natural language explanations for the ground-truth labels.
The text premise provides statement about the contents of the image. The task is to determine whether the statement is true or false based on the image content.
Visual Spatial Reasoning [56] (VSR) consists of 65 spatial relations (e.g., under, in front of, facing, etc.) of instances in images. VSR has more than 10k question pairs, associated with 6940 images from MS COCO [55].
GQA [35] dataset consists of 22M questions about various day-to-day images. The questions are about compositional question answering based on scene graphs. In our evaluation, we only use the text of the question as model input, but not the scene graphs.
Compositional Language and Elementary Visual Reasoning [40] (CLEVR) a synthetic dataset with questions that test various aspects of visual reasoning including attribute identification, counting, comparison, spatial relationships, and logical operations. The dataset contains 700k questions the training set and 150k in the validation set.
A.2 Instruction Tuning Details
We adopt pretrained BLIP [52]' and OFA [104]2 as VLMs unless specified otherwise, and freeze their parameters without updating. The instruction tuning only happens on the language model part. The training set each dataset used for finetuning. We use the whole training set unless otherwise specified in the low-data instruction tuning discussion.
We AdaFactor optimizer [92] at the learning rate of le-4 for all Cola-FT experiments. The batch size is by default set to 16, though we find Cola-FT insensitive to batch size. We finetune and evaluate the models on NVIDIA V100 or A 100 GPUs. The finetuning time is shown in Table
B http://gtthhhccom/sleeerre/.LLL
POFA: hmtp://tuggiiggfac..ooOAAAA.
19

---

30.3 Following the common experiment protocols, we employ a teacher forcing and greedy decoding strategy for fine-tuning.
V100 hours
A-OKVQA
e-SNLI-VE
VSR
GQA
VQA
OK-VQA
CLEVR
Cola-FT
12
8
24
80
12
24
Table Cola-FT training time of FLAN-T3-XXL for each dataset. We finetune subset of GQA.
A.3 Evaluation Details
As specified, we use the validation or test set multiple choice accuracy as the evaluation metric. In A-OKVQA, report ats/t accuracy, and val accuracy in e-SNLI-VE, test (zero-shot split) accuracy in VSR. For simplicity and consistency, we evaluate ablation experiments on A-OK VQA validation set. Following the common experiment protocols [32, 74], we report the single run results for performance comparison.
The exemplars the inference of Cola-Zero randomly sampled from the training set, ie.. supposedly help the LLM learn the input data distribution and output format but do not leak relevant information to the evaluation question.
A.4 A-OKVQA Direct Answer Results
In addition to MC accuracy, we present the direct answer (DA) accuracy models on the A-OKVQA validation set in Tables and 8.
FLAN-T5-Sman
FLAN-TS-Base
FLAN-TS-XL
FLAN-T5-XXL
Cola-FT
56.5
60.6
64.1
Cola-Zero (2-shot)
34.6
57.6
61.0
Cola-Zero (0-shot)
28.6
36.0
55.0
59.3
Table 7: A-OKVQA validation set DA performance. Extension of Figure
1-shot
2-shot
3-shot
4-shot
Cola-Zero
60.2
61.0
60.7
59.2
Table Cola-Zero in-context few-shot learning DA performance on A-OKVQA validation set. Extension of Figure 6.
A.5 Qualitative Examples
In this section, we provide more qualitative examples on A-OKVQA (Figure 8), e-SNLI-VE (Fig- ure 9), and VSR (Figure 10) datasets.
Due to the large span of the three figures, for better visibility, we put the detailed description directly in each figure's caption part. We illustrate how Cola-FT and Cola-Zero process the VLMs answers in each example. Overall, in these examples, we can observe that even if BLIP and OFA provide wrong answers, Cola can still present the correct answer based on the captions provided by OFA and BLIP, as well as the choice set. This may illustrate how Cola amazingly accomplishes visual reasoning tasks via coordinating BLIP and OFA.
A.6 Failure Cases
In Figure 11, we provide a few failed cases to analyze the specific behavior of Cola.
The leftmstt example's correct answer is kayaking, but there are no hints from OFA and BLIP's answers and captions. Therefore Cola-Zero incorrectly provides the answer OFA without sufficient information as hints, while surprisingly Cola-FT answered correctly from OFA's boating answer.
20

---

Question
Why might people sit here?
The room can
what type location are they playing with the body board?
What is in front of the monitor?
described as what?
OFA caption
colorful umbrellas on riverwalk
living room layout and decor medium size how to decorate small living room dining combo mant
person, left, and person look at painting of great white shark.
a desk computer, lamp, a laptop, and plant.
BLIP
colorful umbrella colorful umbrellas
dining room table with a glass
man holding surfboard another standing next to him ['rom', 'beach, 'park', 'store')
a desk with a computer lamp
caption
Choices
I'to testify. "to rest', "to shop', 'get tattoo')
l'tidy,, 'essyy, 'on fire', destroyedd)
l'keyboard; "phone", 'mouse", 'headphones')
OFA answer
living
bedroom
keyboard
BLIP answer
yes
dining room
beach
monitor
Cola-Zero
to rest
tidy
beach
keyboard
answer
Cola-FT
rest
tidy
room
keyboard
answer
Figure 8: A-OKVQA qualitative examples. Leftmost: LLM doesn't 't use BLIP and OFA's answers, but may observe from captions derive the correct final answer. Left: As shown on the left, LLM does not follow the wrong answers from OFA and BLIP but gets the correct answers from captions. Right: With both OFA and BLIP answering incorrectly, LLM derives the correct one from both VLMs' captions and answers. Rightmost: After assessing the questions, answers, and captions, LLM goes with OFA's answer and rewrites it to match the expression the choices. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
The left example again has insufficient information from captions. While BLIP answers and OFA answers yes, Cola-FT chooses to answer maybe, which looks natural but unfortunately picks the wrong choice.
The right example's 's captions contain enough information this time. But both Cola-FT and Cola- Zero are misled by BLIP's wrong answer no parking.
The rightmost example also has insufficient information from captions. In this situation, Cola has choice but to believe either BLIP or OFA's answer, but it mistakenly prefers BLIP's s wrong answer.
A.7 Prompt Templates
Across three datasets, the prompt template is roughly the same, with minor differences mainly in the format of the questions and choices. We list the prompt templates adopted in A-OKVQA and C-SNILIVE/VSR in Table and Table 10, respectively.
A.8 Parametee--fffcieet Finetuning
To further reduce the computation cost in model adaptation, we explored parameter-efficient finetuning (PEFT) techniques to reduce finetuning parameter counts. Specifically, use (IA)² [57], which finetunes an overhead million parameters, equivalent to 0.01% of the full parameters of FLAN- T5-XXL.
Compared full finetuning, (IA)3 requires more iterations to converge. The performance (IA)a finetuned FLAN-TS-XXL model is on par with fully finetuned FLAN-T5-S (80 million parameters) counterpart (Figure 5). Notably, the former is associated with more computation and memory footprint consequence of more parameters in the forward pass.
21

---

Question
Does the image describe A professional daredevil "?
Does the image describe the dog is shitz ??
Does this Image describe "Two twenty seenty-somethngg prepare to catch salmon while other older men catfish"
Does this image describe "A little girl gets hit by woman riding bike"?
OFA caption
person doing flip on mountain bike
dog jumping out the water.
men repairing fishing nets on the beach in zanzibar, tanzania
a man and a woman on tandem bike
BLIP
doing
dog jumping over in the water
man sitting boat with a fishing net net
woman riding
caption
bike in the air
bicycle in a parking lot
Choices
['ess,, 'maybe', 'no']
l'yes', 'maybee, 'no')
['yes,, 'maybe', 'no'] yes
['ess,, 'maybe', 'no')
OFA answer
yes
yes no
BLIP answer
yes
no
yes
Cola-Zero
yes
no
no
no
answer
Cola-FT
maybe
maybe
maybe
answer
Figure 9: e-SNLI-VE qualitative examples. Leftmost: As the connection to daredevil is not obvious in BLIP and OFA's captions, although Cola-Zero is misled, Cola-FT correctly answers maybe. Left: Similar to the left example, Cola-FT answer correctly as no obvious connections are seen from the captions to this question. Right: Similar to the left example, the fact of catch catfish is not reasonable from the captions, Cola-FT picks the correct answer maybe. Rightmost: As girl gets hil is not obvious BLIP and OFA's captions and answers. Cola-Zero and Cola-FT both follow BLIP choose the correct answer no. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
A.9 Extended Ablation Studies
Do caption labels offer useful information to LLM? How would more prompt variations affect the performance of Cola? We tested Cola-Zero with and without caption labels A-OKVQA validation set, observing slight decrease in performance when without them (70.39% w/t vs. 69.97% w/o). More ablative experiments showed that removing the VLM's s answer labels led substantial drop in performance (70.39% w/t vs. 67.62% w/o). Removing the model characteristic descriptions also led decrease (70.39% w/t vs. 68.37% w/o).
Do longer image captions improve reasoning performance? On A-OKVQA validation set, we tested longer image descriptions (>50 tokens) but found gain compared to Cola or single VLMs. Longer captions decreased FLAN-T5+OFA* accuracy by 0.61% % and FLAN-TS with BL.IP by 0.69% on the A-OKVQA validation set. Cola (captions <30 tokens) reached 77.73%. outperforming individual VLMs. Longer captions lacked meaningful visual context, possibly due to short text and image pairs their training datasets. This experiment reaffirms Cola's effectiveness in aggregating individual VLM functionalities.
B Extended Related Works
B.1 Finetuning Large Language Models
Large language models [8, 71, 6] pretrained on massive amounts of unstructured data have gradually demonstrated great performance by finetuning on additional task-specific instances. Finetuning a large language model can considerably more sample efficient than re-training from scratch, although acceptable performance may still require considerable quantity of data [95]. Recent
22

---

Question
maybe Does this image describe
Does image describe "The bed under the handbbag ?
Does this Image describe "The couch behind the hot dog"?
Does this image describe "The bowl contains the banana"
"The truck contains the elephant"
OFA caption
an elephant being transported on truck in lanka
black white tuxedo cat with white nose, yellow eyes, and white
person enjoying meal by the fire
bananas and mangoes in bowl
BLIP
with large elephant in the back of it
laying on
man sitting with a plate of food
fruit
caption
with pillow
this bowl
Choices
[yes', 'no
['yes', Ino']
['yes, 'no]
[yess 'no']
OFA answer
yes
yes
yes
BLIP answer no
no
yes
no
Cola-Zero
no
no
no
yes
answer
Cola-FT
yes
no
yes
answer
Figure 10: VSR qualitative examples. Leftmost: As OFA caption mentioned elephant being transported and OFA provides the correct answer, Cola-FT follows OFA's choice. Left: As OFA and BLIP provide the same answer, Cola-Zero and Cola-IFT follow the choice. Right: As the captions do not provide obvious information, even BLIP and OFA provide the same answer, Cola-Zers Zero and Cola-FT are not misled to the wrong choice. Rightmost: As the captions provide strong clue bananas in bowl, although BLIP's answer incorrect, Cola- and Cola-FT still choose the correct answer. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
Question
What are the people doing water?
Does the image describe making vase"?
What kind of zone is this parked
Does this image describe motorcycle the truck" ?
OFA caption
black and white photo of a man canoe in the river
person on the potter's wheel
city made by people bucharest
men walking past truck kabul, afghanistan.
looking
BLIP
a man and woman on park
man sitting on chair and using wheel
bicycle parked next to pedestrian crossing sign
a man walking down the street city Iyee', 'no')
caption
Choices
('surfing', "ishing", kayaking" 'swimming']
I'yes', 'maybe', 'no')
['temporary' 'pedestrian' parking'. handicap]]
OFA answer
boating
yes
pedestrian
yes
BLIP answer
swimming
no
no parking no parking
no
Cola-Zero
OFA
no
no
answer
Cola-FT answer
kayaking
no parking
no
Figure 11: Failed cases. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
works have finetuned task-specific models that demonstrate amazing capabilities in many real- world applications, such Copilot for program synthesis [1]].
23

---

VQA Prompt Template
Answer the following multiple-choice question by OFA and BLIP's description and their answers to the visual question. OFA and BLIP are two different vision-language models to provide clues.
OFA's description: <CFA caption>
BLIP's description:
<BLIP caption>
Q: <Question>
OFA's answer: <OFA answer>
BLIP's answer: <BL.P answer>
Choices: <Choices to the question>
A:
Table 9: VQA prompt template for the LLM, for VQA v2/ V2/OK-VQAT A-OKVQA, The LLM is instructed to coordinate VLMs. Each question set defines visual context, question with choices, and plausible answers.
C-SNLI-VE/ /VSR Prompt Template
Answer the following multiple-choice question by OFA and BLIP's description and their answers the visual question. OFA and BLIP are different vision-language models provide clues.
OFA's description: <CFA caption>
BLIP's 's description: <BLIP caption
Q: does the image describe <hypothesis>
OFA's answer: <OFA answer>
BLIP's ' answer: <BLIP answer>
e-SNLI-VE Choices: [yes, no, maybe] VSR Choices: [yes, no]
A:
Table 10: e-SNLI-VE/VSR prompt template for the LLM. The LLM is instructed coordinate VLMs. Each question set defines visual context, hypothesis, and plausible answers.
B.2 Instruction-based Learning
Recent advances in the capabilities of language models have piqued researchers' curiosity in the field of instruction-basee learning [26, 64, 87, 24). The core of instruction-based learning is to explore the knowledge of the language model itself. In contrast to prompt learning stimulate the language model's ability to complete blanks, instruction tuning more focuses on activating the language model's comprehension by giving obvious instructions models and expecting correct feedback. Earlier work [67] finetune BART [46] using instructions and few-shot exemplars in question answering, text classification, and text modification. Their findings suggest that few-shot instruction tuning improves performance on unseen tasks. [66] finetunes GPT-2 Large and also observes that few-shot exemplar instruction tuning could improve performance. [85] finetunes T5-11B with more diverse instruction templates and observe similar improvements in sero-shot learning. More recent work [108] performs large-scale experiments with a 137B FLAN-TS model and instruction-tune it on over datasets verbalized via instruction templates. They observe FLAN-T5 substantially improves over zero-shot GPT-3 (175B) on 20 of 25 evaluation datasets. OpenAl also releases InstructGPT [71] based on GPT-3 [8], makes use of human annotations to steer desired model behavior through both instruction

---

Accuracy
# Finetuning Params
Finetuning
77.73
11B (100%)
PEFT, (IA)
63.76
IM (0.01%)
Table 11: (IA) [57] parameter-efficient tuning (PEFT) performance. We finetune a FLAN-T5-
XXL model on the A-OKVQA training set and evaluate it on the A-OKVQA validation set.
tuning and reinforcement learning of human feedback. They discover that InstructGPT is favored by humans over unmodified GPT-3.
B.3
Visual Reasoning
Beyond the uni-modal reasoning tasks such question answering (QA) [100, 41, 14, 80, 79, 23, 75, 17, 96, 25, 127, 119, 7]. visual reasoning requires models to not only understand and interpret visual information but also to apply high-level cognition to derive rational solutions [40, 35, 4, 59, 60, 84, 120, 34]. Several tasks have been introduced to address visual reasoning, such as visual question answering (VQA) []]. in which models are expected to provide answers to questions related to an image and visual entailment (VE) [113], where the model is required to determine the similarity relationship between a given image and description. Classic visual reasoning methods have employed an image encoder and a text encoder, along with a reasoning block that utilizes attention mechanisms [121, 72, 122, 105], neuro-symbolic methods [117, 61, 116], external knowledge [62, 28, 13] to perform reasoning.
Recent progress in large pre-trained models has led to the development of language models (LLMs) that possess exceptional commonsense reasoning capabilities [78, 16, 15, 77]. These models can potentially replace the reasoning block in visual reasoning tasks, and L.MMs' lack of perception
can be compensated by incorporating multiple vision-language models (VLMs) trained on different domains [76, 104, 52]. For example, PICa [1]5] converts the image into captions that GPT-3 [8] can understand, and adapts GPT-3 to solve the VQA task in a few-shot manner by providing a few in-context VQA examples. However, there is still a lack of research on how to harness the collective power of these complementary VLMs for visual reasoning tasks.
B.4 Model Ensembling
Model ensembling is powerful machine learning technique that combines the predictions of multiple models to improve the overall performance of a given task [20]. Classic model ensembling methods include simple averaging, weighting the predictions based on model performance, and stacking the models. By combining the predictions of multiple models, ensembling can reduce the variance and bias of the final predictions, resulting in more robust and accurate model [83]. Ensemble methods have been shown to perform well in wide range of tasks, including image classification, natural language processing, and time series forecasting. However, when it turns multimodal tasks such as visual reasoning, simple combination is not applicable to heterogeneous models as their inputs and outputs vary.
The Mixture-of Eexture-of-Experts (MoE) [91, 82, 126, 45, 47] can be conceptualized as a model ensemble strategy implemented the level of network architecture. MoE-based multi-modal models [33] excel in leveraging the specific strengths of each expert, thereby delivering the performance that often outstrips that of any individual expert. In these networks, the credibility of each expert's output is dynamically weighted, facilitating a comprehensive and nuanced response to multimodal tasks.
However, even within this sophisticated framework, challenges arise, particularly when managing heterogencous pre-trained multimodal models. To address this problem, an innovative approach known as Socratic Models (SMs) [123] has been proposed. SMs employ prompt engineering to guide these diverse models through multimodal discussions, effectively combining their varied knowledge. This method promotes more harmonious and effective integration of different models, enhancing the ensemble's 's ability to handle complex tasks.
With similar goal, [54] proposes a closed-loop iterative consensus optimization method to utilize the strengths of individual models. However, previous methods do not fully explore the potential of a centralized solution adapt to the separate functionalities of different models, particularly in the
25

---

visual reasoning scenario. Recent studies, such as CICERO [65], have shown that language models possess strong capabilities in coordinating multiple agents, which inspires us reorganize pre-trained multimodal models with focus on the language models.
Broader Impact
This study inherits ethical risks of biases from pretrained VLMs and LLMs, depending on their training data. We suggest the users consider the possible biases in reasoning and prompt the model to interpret its predictions in natural languages when necessary.
26


Reviews:
[{"review_id": "tav8jo9FhE", "reviewer": "Reviewer_4gyW", "summary": "The paper introduces a novel approach to ensembling multiple vision-language models (VLMs) for solving visual reasoning tasks. More specifically, the authors propose to use a language model (LM) to coordinate answers from various VLMs, which outperforms traditional ensemble approaches. Multiple experiments demonstrate the effectiveness of the proposed Cola approach.", "questions": "In Table 3, the authors implemented \"Ensemble (average)\" based on Equation (1). Was each $P_i(v, q)$ normalized before averaging them?", "limitations": "The authors discussed the limitations of the work in Section 5. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well written, with great explanations of the proposed Cola approach and various figures and tables. \n\n2. Using the LM to coordinate VLMs for visual reasoning tasks is novel and interesting. ", "weaknesses": "The authors observed that long input (multiple VLMs) does not guarantee higher performance in Section 3.5. Specifically, Figure 4 shows the approach is vulnerable to the number of models. Cola models may also be affected by which VLMs are used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful critique and thoughtful suggestions for improving our work. In response, we have substantially revised the paper to better demonstrate the effectiveness of language models for coordinating multiple vision-language models.\n\nWe humbly ask the reviewer to meticulously check our responses and the revisions. We greatly value your time, effort, and acknowledgment you have given to our work.\n\n**Q: The authors observed that long input (multiple VLMs) does not guarantee higher performance in Section 3.5. Specifically, Figure 4 shows the approach is vulnerable to the number of models.**\n\nRegarding the limitation on long context inputs, we agree this is an inherent challenge for LLMs. Longer text input is a significant challenge for LLM itself. It’s a interesting topic that should be investigated further and we have added references [1,2,3] to recent works that propose techniques to extend LLM ability to model long context, which can potentially enhance Cola in the future.\n\n[1] Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). Extending context window of large language models via positional interpolation. *arXiv preprint arXiv:2306.15595*.\n\n[2] Mu, J., Li, X. L., & Goodman, N. (2023). Learning to compress prompts with gist tokens. *arXiv preprint arXiv:2304.08467*.\n\n[3] Bulatov, A., Kuratov, Y., & Burtsev, M. (2022). Recurrent memory transformer. *Advances in Neural Information Processing Systems*, *35*, 11079-11091.\n\n**Q: In Table 3, the authors implemented \"Ensemble (average)\" based on Equation (1). Was each $P_i(v, q)$ normalized before averaging them?**\n\nWe also appreciate the clarification request on ensemble prediction averaging. To confirm, we did not normalize the per-model scores $P_i(v, q)$ before averaging and majority voting, since they are already on the same 0-1 scale. Normalization is not needed in this circumstance. Thank you for catching this detail - we have updated the text to clearly state that no normalization was applied."}, {"review_id": "EzJTDoE4I0", "reviewer": "Reviewer_3ict", "summary": "The paper introduces a new paradigm called Cola, which aims to coordinate multiple vision-language models (VLMs) for visual reasoning tasks. While several VLMs have demonstrated strong commonsense reasoning abilities in different domains, effectively combining their capabilities remains a challenge. Traditional methods like ensembling struggle to achieve higher-order communications between these models.", "questions": "Refer to the above comments", "limitations": "Refer to the above comments", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Cola proposes a solution by employing a language model (LM) to coordinate the multiple VLMs. The LM facilitates natural language communication, leveraging the distinct and complementary capabilities of each VLM. The authors introduce two variants of Cola: Cola-FT, which involves fine-tuning the models, and Cola-Zero, which performs in-context learning without the need for fine-tuning.\n\nThe authors conduct extensive experiments to evaluate the performance of Cola on various visual reasoning tasks, including visual question answering (VQA), outside knowledge VQA, visual entailment, and visual-spatial reasoning. They demonstrate that Cola-FT achieves state-of-the-art results in these tasks. Additionally, Cola-Zero exhibits competitive performance in zero and few-shot settings, without requiring fine-tuning. The paper further includes ablation studies and visualizations to validate the effectiveness of the coordinator LM. These analyses confirm that the coordinator LM comprehends the instruction prompts and understands the individual functionalities of the VLMs, allowing it to coordinate their efforts and enable visual reasoning capabilities.\n\nUsing a language model as a coordinator for different VLMs is novel.\n\nAdequate experiments show a promising performance over baselines.", "weaknesses": "Does the language coordinator play a role to determine the correctness of comparing multiple VLMs according to the answering language description? What if both VLMs are wrong?\n\nAs is mentioned in the limitation, no rational explanation or logical steps were applied either in\nVLMs or LM.\n\nThe paper missed the recent work in visual reasoning, like Flamingo(https://nips.cc/Conferences/2022/ScheduleMultitrack?event=54165), STAR (http://star.csail.mit.edu/), GAMR(https://openreview.net/pdf?id=iLMgk2IGNyv), SHG-VQA(https://arxiv.org/abs/2304.08682), etc.\n\nOnly two VLMs are applied. It’s interesting to see the margin gains of Cola when compared with the best VLMs in the experiments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our gratitude to reviewer’s insightful critique of our paper. These thoughtful comments have guided revisions that will undoubtedly enhance the quality of our work. We hope that by highlighting the potential of language models (LMs) to coordinate multiple vision-language models (VLMs), our findings can provide new insights into large pretrained models and visual reasoning.\n\n**Q: Does the language coordinator play a role to determine the correctness of comparing multiple VLMs according to the answering language description? What if both VLMs are wrong?**\n\n**Q: As is mentioned in the limitation, no rational explanation or logical steps were applied either in VLMs or LM.**\n\nYes, the language coordinator determines the correctness of VLM plausible answers implicitly, given their captions and the caption and answer labels (see Figure 3 and Sec 3.4). It’s all plausible answers, captions, VLM answer/caption labels, and the world knowledge the LM it encodes in itself, that contribute to the final decision of the language coordinator. Even in the cases where both VLMs give wrong answers, there are chances the language model arrives at the right answer.\n\nIn the paper, we did not apply techniques such as Generating Rationale or Chain-of-Thought to make the LM’s logical process more explicit. However, we attempted in the supplementary experiments, on a small scale, to modify the prompt template of Cola-FT so that the model would output the logical conduction process, allowing us to observe the specific behaviors of the LM during its coordination of VLMs. We finetuning Cola-FT to output rationales before answers, by A-OKVQA ground truth rationales. In the modified prompt, we ask the model to provide rationales.\n\nCola FT's demo output is as follows:\n\n- Case 1: leftmost example of Figure 8\n\n> Rationale: People might sit here to rest. The umbrellas are on the riverwalk. The answer is:rest\n\nOFA gives a reasonable answer (but out of choices) to the question while BLIP gives an irrelevant answer. In this case, both answers are wrong. However, either Cola-Zero or Cola-FT is able to infer from captions and plausible answers to give the correct answer “to rest”. The rationale suggests that the LM understands the scene that the umbrellas are on the riverwalk and guesses that people might sit here to rest based on commonsense. The final answer is correct.\n\n- Case 2: leftmost example of Figure 11\n\n> Rationale: The people are in a canoe. The people are in a canoe. The people are in a canoe. The answer is:kayaking\n\nBoth VLMs are wrong in their plausible answers. OFA’s answer “boating” is semantically correct as the correct answer is “kayaking”, though it’s not the correct answer because this is a multiple-choice question. Cola-Zero gives a wrong answer “OFA” which is obviously wrong because “OFA” is the name of one of the VLMs given in the prompt and it’s out of the choices too. However, Cola-FT gives the correct answer “kayaking”, recognizing the correct choice based on prompts of captions and plausible answers after being finetuned. Even though the OFA and BLIP captions fail to identify that the people in the water are on a canoe. The LM identifies that the people in the water are associated with the canoe. The rationale is valid and helpful, though repetitive. The final answer is correct.\n\n- Case 3: third example of Figure 11\n\n> Rationale: The bike is parked in a no parking zone. The bike is parked next to a pedestrian crossing sign. The answer is:no parking\n\nFrom the rationale, we can tell that the LM understands that the bike is parked next to a pedestrian crossing sign. However, it “overthinks” that this is a no parking zone and therefore gives the wrong answer. This rationale helps us understand why Cola-FT gives a wrong answer.\n\nThe inference results on A-OKVQA validation set are as follows:\n\n|  | Acc. |\n| --- | --- |\n| w/t rationale | 74.3 |\n| w/o rationale | 77.7 |\n\nTo force the LM to output rationale does not improve the reasoning performance of Cola. This might be attributed to the low-quality ground truth rationales provided by the A-OKVQA dataset that we use to train the LM. Such rationales are just short and objective descriptions of the scene, without suggesting the underlying outside knowledge to answer the question. Therefore, training the LM to output rationale is harmful, though it outputs insights into the LM's behaviors during reasoning.\n\n**Q: The paper missed the recent work in visual reasoning, like Flamingo, STAR, GAMR, SHG-VQA, etc.**\n\nWe appreciate the reviewer for pointing out this issue. We believe the related works you mentioned here are indeed relevant papers in the field of visual reasoning. Even though we have encompassed a citation list of total 120+ papers in our revision, it is hard to cover all relevant works in this rapidly and vigorously developing area. We have incorporated the related works mentioned above. This ensures that our audience can better reference more excellent papers by reading our paper.\n\n**Q: Only two VLMs are applied. It’s interesting to see the margin gains of Cola when compared with the best VLMs in the experiments.**\n\nWe benchmark the best VLMs on A-OKVQA, InstructBLIP (FLAN-T5-XXL), and InstructBLIP (FLAN-T5-XL). For InstructBLIP baselines, we use InstructBLIP solely (i.e., without LM) to evaluate 0-shot performance. Interestingly, InstructBLIP (FLAN-T5-XL) slightly outperforms InstructBLIP (FLAN-T5-XXL). Referring to the results below, both Cola-Zero and Cola-FT improved the reasoning performance by substantial margins compared to single VLMs.\n\n|  | Acc. | $\\Delta$ |\n| --- | --- | --- |\n| InstructBLIP (FLAN-T5-XL) | 60.4 |  |\n| InstructBLIP (FLAN-T5-XXL) | 59.8 |  |\n| Cola-Zero (0-shot) | 68.0 | +7.6 |\n| Cola-Zero (2-shot) | 72.3 | +11.9 |\n| Cola-FT | 78.1 | +17.7 |"}, {"review_id": "Oez4fNLgsB", "reviewer": "Reviewer_qcEV", "summary": "The paper proposes an ensemble based approach to solve visual reasoning problems. The paper proposes to use an instruction fine-tuned large language model to integrate answers to visual reasoning problems provided by vision language models. The paper presents two variants of the aggregation model -- using fine-tuning and using in-context learning. The model is evaluated on the VQA v2, A-OKVQA, OK-VQA, e-SNLI-VE and VSR datasets.", "questions": "* The paper should include further details on the computational resources used. L680 in the supplementary material just states that V100 or A100 GPUs were used, but the paper should include further details about the total computational resources used.\n\n* The paper should include further motivational details on why the particular datasets VQA v2, A-OKVQA, OK-VQA, e-SNLI-VE and VSR were used? Why were more datasets which require more complex reasoning abilities such as CLEVR and GQA were not used.", "limitations": "The paper does not include any discussion about its limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper reports promising results on the VQA v2, A-OKVQA, OK-VQA, e-SNLI-VE and VSR datasets.\n* The paper includes a variety of ablations that show the effectiveness of the proposed method -- including model size, scaling, number of video-language models as ensemble members.\n* The paper includes qualitative examples which highlight the effectiveness of the proposed method.\n* The paper is well written and easy to understand.", "weaknesses": "* While the results are very promising, it would be helpful to add results on more complex datasets such as CLEVR (CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning) and GQA (GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering) which study compositional reasoning.\n\n* The paper should also consider comparing to prior work such as Visual Programming: Compositional visual reasoning without training, CVPR 2023, which also uses an external large language model to coordinate vision / language-vision models.\n\n* The paper claims in L291 \"This work demonstrates the first step toward applying language models for visual reasoning\", but Flamingo (Flamingo: a Visual Language Model for Few-Shot Learning, NeurIPS 2022) already shows zero-show visual reasoning on datasets such as Next-QA.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for your feedback on our paper, and we will provide our responses to these concerns in the following. We have also made targeted modifications to the paper, and these issues will help improve our paper. \n\nOur work aims to demonstrate that by using LLM, multiple VLMs can be coordinated to achieve better visual reasoning effects. We hope this work can bring some of our insights to the currently booming fields of LLM and VLM. We hope the reviewer will carefully review our responses and modifications, and we sincerely thank the reviewer for your time and effort, as well as your acknowledgement of our work. \n\n**Q: CLEVR and GQA results (The paper should include further motivational details on why the particular datasets VQA v2, A-OKVQA, OK-VQA, eSNLI-VE and VSR were used? Why were more datasets which require more complex reasoning abilities such as CLEVR and GQA were not used.)**\n\n**Q: The paper should also consider comparing to prior work such as Visual Programming: Compositional visual reasoning without training, CVPR 2023, which also uses an external large language model to coordinate vision / language-vision models.**\n\nWe thank the reviewer for the suggestion of experiments on compositional reasoning datasets and Visual Programming. Below we release the results on GQA and CLEVR datasets. On the GQA validation set, Cola-FT shows marginal improvement over best VLM (OFA), and outperforms Visual Programming [1].\n\n| GQA | Acc. | $\\Delta$ |\n| --- | --- | --- |\n| BLIP | 41.7 |  |\n| OFA | 58.0 |  |\n| Cola-FT | 60.3 | +2.32 |\n| VisProg [1] | 50.5 |  |\n\nOn the CLEVR validation set, Cola-Zero shows marginal performance gain and Cola-FT shows substantial performance gain over single VLMs. It’s interesting to note that InstructBLIP (FLAN-T5-XL) [2] outperforms InstructBLIP (FLAN-T5-XXL) by a large margin in the 0-shot evaluation.\n\n| CLEVR | Acc. | $\\Delta$ |\n| --- | --- | --- |\n| InstructBLIP (FLAN-T5-XL) | 33.7 |  |\n| InstructBLIP (FLAN-T5-XXL) | 16.6 |  |\n| Cola-Zero (2-shot) | 34.4 | +0.7 |\n| Cola-FT | 54.3 | +20.6 |\n\n[1] Gupta, T., & Kembhavi, A. (2023). Visual programming: Compositional visual reasoning without training. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 14953-14962).\n\n[2] Dai, W., Li, J., Li, D., Tiong, A.M., Zhao, J., Wang, W., Li, B., Fung, P., & Hoi, S. (2023). InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. *ArXiv, abs/2305.06500*.\n\n**Q: The paper claims in L291 \"This work demonstrates the first step toward applying language models for visual reasoning\", but Flamingo (Flamingo: a Visual Language Model for Few-Shot Learning, NeurIPS 2022) already shows zero-show visual reasoning on datasets such as Next-QA.**\n\nWe greatly appreciate the reviewer pointing out the oversight regarding the 'Flamingo' work in our article. Our claim, 'This work demonstrates the first step toward applying language models for visual reasoning,' is primarily based on our unique approach, where we employ a standalone and end-to-end LLM  to coordinate outputs from various VLMs and combine them for enhanced results. This, compared to either a single VLM or multiple VLM ensembles, led to significant improvements in multiple visual reasoning datasets as observed with our 'Cola-Zero/FT'. Thus, a better way to quote our claim would be \"This work demonstrates the first step toward applying end-to-end language models for visual reasoning\", which we have revised in the paper.\n\nOn this front, compared to VLM models that incorporate powerful LLMs—such as Flamingo which can integrate with OPT[1] or Chinchilla [2], and OpenFlamingo [3] which can combine with LLaMA [4]—our Cola strategy could potentially benefit from merging with a larger scale LLM to interpret VLM outputs, leading to even better outcomes. For example, Cola-FT obviously applies to end-to-end LM APIs (like Open GPT-4 [5] and Anthropic Claude [6]), which is challenging for conventional ensemble methods or VLMs that need to be finetuned from LLMs.\n\n[1] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., ... & Zettlemoyer, L. (2022). Opt: Open pre-trained transformer language models. *arXiv preprint arXiv:2205.01068*.\n\n[2] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., ... & Sifre, L. (2022). Training compute-optimal large language models. *arXiv preprint arXiv:2203.15556*.\n\n[3] Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., ... & Schmidt, L. (2023). OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models. *arXiv preprint arXiv:2308.01390*.\n\n[4] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. (2023). Llama: Open and efficient foundation language models. *arXiv preprint arXiv:2302.13971*.\n\n[5] OpenAI (2023). GPT-4 Technical Report. *ArXiv, abs/2303.08774*.\n\n[6] Anthropic \\ Introducing Claude https://www.anthropic.com/index/introducing-claude\n\n**Q: the paper should include further details about the total computational resources used.**\n\nWe greatly appreciate the reviewer for pointing out the need for additional detail in this section. To address this, we've incorporated a table illustrating the GPU hours as the computation cost in our training process. These computations are performed with reference to a server with 8 NVIDIA V100 GPUs.\n\n| Cola-FT | V100 hours |\n| --- | --- |\n| A-OKVQA | 12 |\n| e-SNLI-VE | 8 |\n| VSR | 8 |\n| GQA | 24 |\n| VQA v2 | 80 |\n| OK-VQA | 12 |\n| CLEVR | 24 |\n\nFor most datasets, the inference time of Cola-Zero and Cola-FT is ~16 questions / second, with 1 A100 GPU. Each question is composed of 90 to 150 tokens. We have included these details in our revised paper."}, {"review_id": "LLWoSX5HwI", "reviewer": "Reviewer_Yu6H", "summary": "1. The author proposes to utilize a language model as the coordinator between different outputs from different VLMs, leveraging their strengths for visual reasoning.\n2. The proposed method achieves SOTA results on multiple visual reasoning benchmarks. \n3. The analysis shows how zero-shot / fine-tuned language models can coordinate VLM outputs. ", "questions": "1. It is a little unclear that in Table 3, the Cola-FT model is fine-tuned on BLIP+OFA outputs or 3 OFA variance. Why is the Cola-FT performance here much lower than the performance in Table 2 and Figure 3? ", "limitations": "The limitations are discussed and potential solutions are proposed. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method with zero-shot VLMs and zero-shot / fine-tuned LLMs as coordinators are novel. \n2. The performance of the model is significant with sufficient experiment results. ", "weaknesses": "1. Different dataset requires different visual / reasoning capabilities and knowledge. The proposed method performs differently on different datasets. For example, the ZS version performs similarly with 'Ensemble' on VQA v2 and OK-VQA, but outperforms 'Ensemble' significantly on other datasets. It's better to analyze the reason. \n2. Why the proposed method performs better than other methods needs more discussion. For example, comparing with BLIP-2 on VQA. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We kindly request the reviewer to check our detailed responses and revisions in the following. Your time, effort, and affirmation of our research are truly valued.\n\n**Q: Different dataset requires different visual / reasoning capabilities and knowledge….**\n\nWe appreciate the reviewer's insightful question. This is inherently determined by the mechanism of Cola-Zero. It leverages few-shot in-context examples to guide the LLM in harmonizing VLM outputs to obtain the final accurate answer. Its efficiency largely hinges on the quality of these examples.\n\nKey factors to consider include:\n\n1. **Question Format:** Datasets like VQA v2 and OK-VQA contain open-ended questions, while A-OKVQA, e-SNLI-VE, and VSR use multiple-choice. Converting VQA v2 and OK-VQA to classification introduces complexities for traditional ensemble methods, as evident in Table 2. Classic methods struggle with generative models like API-based GPT-4, underscoring Cola's value as an end-to-end ensemble strategy for extensive (vision-)language models. Moreover, Cola-Zero’s efficiency also relies on the question format – it's easier for LLMs to answer when given choices like in A-OKVQA. Conversely, Cola-FT finetunes LLMs to discern answer formats.\n2. **Knowledge Demands and Out-of-Distribution Challenges:** Datasets requiring deep reasoning, such as A-OKVQA, emphasize Cola-Zero and FT’s advantages over Ensemble. With A-OKVQA demanding broad commonsense and world knowledge, pretrained LLMs (like our FLAN-T5) are favored. On such datasets, Cola-Zero outperforms the ensemble baseline, while established models like OFA and BLIP-2, trained on VQA v2, set higher baselines, making significant improvements by Cola challenging.\n3. **In-context Examples:** Cola-Zero's performance is influenced by its in-context examples. Misrepresentative examples can hamper its ability to synchronize VLMs effectively. We use random selection as our baseline for this learning. However, Cola-FT overcomes such limitations. By finetuning, it offers a comprehensive grasp on harmonizing VLM outputs, consistently delivering excellent results, and potentially matching top-tier results.\n\n**Q: Why the proposed method performs better than other methods needs more discussion…** \n\nCompared to individual models like BLIP and OFA, Cola's strength is its integration of responses from multiple models, refined by a language model, to produce superior results. For instance, Figure 2's third example shows that while OFA and BLIP individually fail, Cola-FT uses their captions to deduce the correct answer, “maybe”. While Cola's efficacy may vary across datasets, its LLM can harness comprehensive knowledge to improve VLM outputs. For example, the left side of Figure 8 in the supplementary material showcases instances where both VLMs err. Yet, both Cola-Zero and Cola-FT interpret the data to deliver the right answer, “to rest”. While Cola-Zero's effectiveness is limited by dataset question formats and context examples, Cola-FT's fine-tuning broadens its adaptability across diverse questions. See also our response to Reviewer 3ict.\n\n**Q: It is a little unclear that in Table 3, the Cola-FT model is fine-tuned on BLIP+OFA outputs or 3 OFA variance…**\n\nWe greatly appreciate this constructive question raised by the reviewer, which warrants further clarification. In Table 3, we run OFA-base models under different random seeds, represented as OFA-Base-1, OFA-Base-2, etc. Therefore, the corresponding results of Ensemble and Cola-Zero/FT in Table 3 are based on the outputs of the aforementioned OFA-Base-1/2/3 models. Similarly, in Table 4, the structure of Ensemble and Cola-Zero/FT is based on the outputs of the OFA-tiny/medium/base models.\n\nIn both experiments, the BLIP model is not involved and the prompt is slightly changed to accommodate more VLMs. We have clarified the settings in our revised paper.\n\nAs an ensemble method, Cola demonstrates larger performance gains compared to single VLMs when the VLMs are more complementary in their capabilities. Table 3 presents an example where the three VLMs are nearly identical. As a consequence, the performance gains of Cola-Zero and Cola-FT are limited, and so are Ensemble baselines. In Table 4, OFA-tiny/medium/base possesses different capabilities, which results in a much larger performance gain for Cola-Zero (3.6% on A-OKVQA, 5.0% on e-SNLI-VE) and Cola-FT (8.5% on A-OKVQA, 11.1% on e-SNLI-VE). In Table 2, BLIP and OFA models are quite different in their pre-training data so they show complementary capabilities (see examples in Figure 2, 8-11). Therefore, the performance gains for Cola-Zero and Cola-FT are even larger."}]

Author rebuttal:
We greatly appreciate the reviewers' time and effort in providing thoughtful feedback on our work. We are pleased that the reviewers recognize the novelty of our Cola framework for coordinating multiple VLMs for visual reasoning. We also appreciate the suggestions to strengthen the paper through additional experiments and analysis.

First, we are deeply appreciative of the reviewer’s recognition of our paper by their comments on:

- Introduction of a novel Cola approach using a language model (LM) to effectively coordinate multiple visual language models (VLMs) for visual reasoning tasks;
- Comprehensive experiments that highlight Cola's notable performance across several datasets;
- Detailed analysis, inclusive of ablation studies and visual illustrations, confirming the coordinator LM's adeptness at integrating VLMs.

In response, we have conducted several new ablation studies which provide further insights into Cola:

- Rationales help explain Cola's reasoning steps but do not improve performance, confirming that the coordinator alone captures the necessary reasoning.
- Cola boosts the latest SOTA VLMs like InstructBLIP, showing general effectiveness across model families.
- Cola improves compositional reasoning on GQA and CLEVR, demonstrating broad applicability.

Additionally, we present a detailed analysis differentiating the behaviors of Cola-Zero and Cola-FT. References are provided to specifically address the requests of Reviewers 3ict and Yu6H.

Through these added experiments and analyses, we believe the paper better conveys the strengths of our approach. We are grateful to the reviewers for their feedback pushing us to strengthen the work, and we hope the revisions satisfactorily address the concerns raised. We look forward to continuing the discussion and thank the reviewers for their time and consideration.

Meta-review: The paper combines the use of a LM with two VLMs to address visual question answering. The two VLMs are coordinated through the LM through careful prompting, an additional contribution is demonstrating the benefit of finetuning the LM on the task and demonstrating the effect of LMs at different scales. Reviewers assessed the work mostly positively but there were initial concerns on lack of experiments on other datasets such as GQA and CLEVR. Additional results were provided showing the effectiveness of the method across more diverse scenarios which was viewed positively by reviewers. 

The paper is overall good and authors are encouraged to incorporate reviewer feedback and additional supporting experiments in the final version.
Decision: Accept (poster)