SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In-context Learning & Low-data Instruction Tuning. We conduct experiments on different data scales to verify Cola's performance varying from zero-shot to full-shot under in-context learning and full-finetune paradigm. As shown in Figure 6, with Cola-Zero, few-shot exemplars substantially improve performance compared to zero-shot learning. As [16, 108] revealed, exemplars potentially help the model better understand the output format and understand the instructions in Table 1. Cola-Zero for in-context few-shot learning outperforms zero-shot learning by a large margin, being on par with low-data Cola-FT without instruction tuning. We also observe Cola-FT's substantial performance gain when finetuning shots increase to 1000 and beyond.
3.6 Saliency Visualization
As shown in Figure 7, we visualize the importance of the input prompt tokens by input-gradient saliency feature attribution [19], implementing with Ecco [2]. The input tokens that are more relevant to predict the output token "grass" are highlighted in darker colors. In the given example, both Cola-FT and Cola-Zero predict the correct answer and find the relevant clues from visual context and plausible answers. Figure 7(b) shows that Cola-Zero attributes the output more to the instructions in the prompt template. This explains Cola-Zero's competitive performance, a consequence of FLAN instruction tuning [108]. After instruction tuning, Cola-FT focuses more on the most informative parts of input: the question, choices, as well as VLMs' plausible answers.
3.7 Can Cola-FT Explain its Answers?
We modify the prompt template of Cola so that the model would output the logical conduction process, allowing us to observe the specific behaviors of the LLM during its coordination of VLMs.
Figure 7: Visualization of input token saliency. We visualize the relevancy between input tokens and the output token "grass" by feature attribution [19]. The more salient tokens are highlighted in darker boxes. Cola-FT focuses on the question, choices, and VLMs' plausible answers in (a). While as shown in (b), Cola-Zero pays extra attention to instructions and VLM labels, as a consequence of FLAN-T5 instruction tuning [16].
We finetune Cola-FT to output rationales before answers, by A-OKVQA ground truth rationales. In the modified prompt, we ask the model to provide rationales. For the leftmost example of Figure 8, Cola-FT outputs "Rationale: People might sit here to rest. The umbrellas are on the riverwalk. The answer is:rest". OFA gives a reasonable answer (but out of choices) to the question while BLIP gives an irrelevant answer. In this case, both answers are wrong. However, either Cola-Zero or Cola-FT is able to infer from captions and plausible answers to give the correct answer "to rest". The rationale suggests that the LLM understands the scene that the umbrellas are on the riverwalk and guesses that people might sit here to rest based on commonsense. The final answer is correct. For the leftmost example of Figure 11, the rationale output is "The bike is parked in a no parking zone. The bike is parked next to a pedestrian crossing sign. The answer is:no parking". Both VLMs are wrong in their plausible answers. OFA's answer "boating" is semantically correct as the correct answer is "kayaking", though it's not the correct answer because this is a multiple-choice question. Cola-Zero gives a wrong answer "OFA" which is obviously wrong because "OFA" is the name of one of the VLMs given in the prompt and it's out of the choices too. However, Cola-FT gives the correct answer "kayaking", recognizing the correct choice based on prompts of captions and plausible answers after being finetuned. Even though the OFA and BLIP captions fail to identify that the people in the water are on a canoe. The LLM identifies that the people in the water are associated with the canoe. The rationale is valid and helpful, though repetitive. The final answer is correct.
To force the LLM to output rationale does not improve the reasoning performance of Cola (w/t rationale  $74.3\%$  vs. w/o rationale  $77.7\%$ , on A-OKVQA val set). This might be attributed to the low-quality ground truth rationales provided by the A-OKVQA dataset that we use to train the LLM. Such rationales are just short and objective descriptions of the scene, without suggesting the underlying outside knowledge to answer the question. Therefore, training the LLM to output rationale is harmful, though it derives insights into the LLM's behaviors during reasoning.
3.8 Does Cola-FT Transfer across Tasks?
We examine Cola-FT's generalization ability across tasks. From Table 5, we observe the zero-shot performances on target datasets after instruction tuning on a certain source dataset. Although each dataset varies in question types and prompt templates (see detailed comparisons in Appendix A.7), we find that Cola
Table 5: Cola-FT is generalizable across out-of-distribution tasks. In most cases, the performances surpass Cola-Zero (in-context 2-shot learning results in brackets) on target datasets.
Finetuned on	A-OKVQA val	e-SNLI-VE val	VSR test
A-OKVQA	77.7 (70.4)	58.7	57.6
e-SNLI-VE	71.2	81.6 (57.8)	51.7
VSR	71.4	61.4	66.9 (55.8)
FT maintains competitive performance when zero-shot transferred to a new task, outperforming Cola-Zero in-context 2-shot learning and ensemble baselines (see also Table 2).
Visual Reasoning. Beyond unimodal reasoning tasks such as question answering (QA) [100, 14, 119, 7], visual reasoning extends high-level cognition to visual domains, requiring an intelligent agent to derive rational solutions [40, 35, 84, 120, 38, 112]. Several tasks have been introduced to address
visual reasoning, such as VQA [1], in which models are expected to provide answers to questions related to an image, and visual entailment [113], where the model is required to determine if a text description is consistent with the visual content provided.
Classic visual reasoning methods employ an image encoder along with a reasoning block that utilizes attention mechanisms [102, 121, 122, 105], neuro-symbolic methods [101, 117, 61], or external knowledge [62, 28, 13].
Recent progress in large pretrained models has led to the development of LLMs that capture exceptional commonsense reasoning capabilities [78, 16, 15]. These LLMs can potentially replace the reasoning module in visual reasoning tasks, and LLMs' lack of perception can be compensated by incorporating multiple VLMs trained on different domains [76, 104, 52]. However, there is still a lack of research on how to harness the collective power of these separate VLMs for visual reasoning tasks. More related works are in Appendix B.
Model Ensemble. Model ensemble is a powerful machine learning technique that combines the predictions of multiple models to improve the overall performance of a given task [20]. The variance and bias of the final predictions decrease, resulting in a more robust and accurate model [83]. To this end, common methods include averaging [111], voting [37], interpolation [36], weighting the predictions based on model performance [22], or stacking the models [10].
Ensemble methods have been challenging for generative tasks like visual reasoning, where a simple combination is not applicable to heterogeneous models due to their enormous and varying input/output token spaces. To address the issue, Socratic Models (SMs) [123] use prompt engineering to guide the heterogeneous pretrained multimodal models through natural language discussions. With a similar goal, [54] proposes a closed-loop iterative consensus optimization method to utilize the strengths of individual models. However, previous methods do not fully adapt to the intrinsic patterns of different models, particularly in the visual reasoning scenario. Recent studies, such as CICERO [65], have shown that LLMs possess strong social intelligence in coordinating multiple agents, which inspires us to reorganize pretrained mixed-modal models with a focus on adapting LLMs. More recently, Toolformer [86] and HuggingGPT [93] further demonstrate LLMs' abilities to leverage, coordinate, and incorporate the results from external sources such as other models or even APIs to solve complex tasks. While the external tools are called in sequential order in existing work, we study coordinating multiple tools (specifically, expert models) in parallel in this work.
Question Format. Datasets like VQA v2 and OK-VQA contain open-ended questions, while A-OKVQA, e-SNLI-VE, and VSR use multiple-choice. Converting VQA v2 and OK-VQA to classification introduces complexities for traditional ensemble methods, as evident in Table 2. Classic methods struggle with generative models like API-based GPT-4, underscoring Cola's value as an end-to-end ensemble strategy for extensive (vision-)language models. Moreover, Cola-Zero's efficiency also relies on the question format - it's easier for LLMs to answer when given choices like in A-OKVQA. Conversely, Cola-FT finetunes LLMs to discern answer formats (Figure 7).
Limitations. Visual reasoning is a diverse topic. This work demonstrates the first step toward applying end-to-end language models for visual reasoning. While the LLMs perform well on the discussed datasets, there is a large body of visual reasoning tasks to evaluate in future works, such as intention prediction and rationale explanation.
Future Works. First, exploring the use of non-parametric tools for visual reasoning would be useful to enhance Cola's performance. Second, Cola's use can be extended to other reasoning and planning tasks, such as image generation and action planning, by coordinating multiple models in parallel. Third, by improving inter-model communications, Cola can be more interpretable and safe for high-stakes applications.
Conclusion. In this paper, we have proposed a novel paradigm for visual reasoning that harnesses the power of multiple VLMs by utilizing a coordination mechanism, where an LLM acts as a coordinator who communicates with VLMs to integrate their respective strengths. Experiments show that reasoning performance is substantially improved by LLM finetuning or in-context learning. Our results provide a promising step towards building multi-component intelligent systems that capture multimodal reasoning capabilities in a human-like way.
This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2022-01-029). Besides, this project is supported by NTU NAP, MOE AcRF Tier 2 (MOE-T2EP20221-0012), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as advise from the industry partner(s).
[125] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. 3
[126] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022. 25
[127] Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774, 2021. 25
In this section, we elaborate on our training and evaluation details, prompt templates, and more qualitative examples for analysis.
Our experiments are conducted on a challenging suite of three diverse visual reasoning tasks, including outside knowledge VQA, visual entailment, and visual spatial reasoning. For each task, we select the following dataset respectively.
Visual Question Answering v2 [27] (VQA v2) is a large-scale benchmark containing over 1 million images from the COCO dataset and more than 250,000 human-generated question-answer pairs. The dataset is designed to test the ability of machine learning models to understand both the visual content of an image and the meaning behind natural language questions. The questions in VQA v2 cover a wide range of topics and are often open-ended, requiring models to reason and generalize about the world. VQA v2 has been widely used to evaluate the performance of state-of-the-art models in the field of computer vision and natural language processing.
Augmented Outside Knowledge VQA [89] (A-OKVQA) contains about 25k questions paired with both multiple choice (MC) answer options. Unlike most existing VQA datasets, the questions in A-OKVQA cannot often be answered by querying the knowledge base, but rather involve some type of commonsense reasoning and outside knowledge about the situation portrayed in the image.
Outside Knowledge VQA [63] (OK-VQA) includes more than 14,000 questions that require external knowledge to answer. The answers are provided in free-text direct answer form. Both A-OKVQA and OK-VQA sample images from the COCO dataset, with no overlapping.
e-SNLI-VE [21] dataset is an extended version of SNLI-VE dataset [113], which contains about 190k question pairs and human-annotated natural language explanations for the ground-truth labels. The text premise provides a statement about the contents of the image. The task is to determine whether the statement is true or false based on the image content.
Visual Spatial Reasoning [56] (VSR) consists of 65 spatial relations (e.g., under, in front of, facing, etc.) of instances in images. VSR has more than 10k question pairs, associated with 6940 images from MS COCO [55].
GQA [35] dataset consists of 22M questions about various day-to-day images. The questions are about compositional question answering based on scene graphs. In our evaluation, we only use the text of the question as model input, but not the scene graphs.
Compositional Language and Elementary Visual Reasoning [40] (CLEVR) is a synthetic dataset with questions that test various aspects of visual reasoning including attribute identification, counting, comparison, spatial relationships, and logical operations. The dataset contains 700k questions in the training set and 150k in the validation set.
A.2 Instruction Tuning Details
We adopt pretrained BLIP  $[52]^1$  and OFA  $[104]^2$  as VLMs unless specified otherwise, and freeze their parameters without updating. The instruction tuning only happens on the language model part. The training set of each dataset is used for finetuning. We use the whole training set unless otherwise specified in the low-data instruction tuning discussion.
We use an AdaFactor optimizer [92] at the learning rate of 1e-4 for all Cola-FT experiments. The batch size is by default set to 16, though we find Cola-FT insensitive to batch size. We finetune and evaluate the models on NVIDIA V100 or A100 GPUs. The finetuning time is shown in Table 6.
1BLIP: https://github.com/salesforce/BLIP
$^{2}$ OFA: https://huggingface.co/OFA-Sys
Following the common experiment protocols, we employ a teacher forcing and greedy decoding strategy for fine-tuning.
V100 hours	A-OKVQA	e-SNLI-VE	VSR	GQA	VQA v2	OK-VQA	CLEVR
Cola-FT	12	8	8	24	80	12	24
A.3 Evaluation Details
As specified, we use the validation or test set multiple choice accuracy as the evaluation metric. In A-OKVQA, we report val/test accuracy, and val accuracy in e-SNLI-VE, test (zero-shot split) accuracy in VSR. For simplicity and consistency, we evaluate ablation experiments on A-OKVQA validation set. Following the common experiment protocols [32, 74], we report the single run results for performance comparison.
The exemplars at the inference of Cola-Zero are randomly sampled from the training set, i.e., supposedly help the LLM learn the input data distribution and output format but do not leak relevant information to the evaluation question.
A.4 A-OKVQA Direct Answer Results
In addition to MC accuracy, we present the direct answer (DA) accuracy of models on the A-OKVQA validation set in Tables 7 and 8.
Table 6: Cola-FT training time of FLAN-T5-XXL for each dataset. We finetune a subset of GQA.
FLAN-T5-Small	FLAN-T5-Base	FLAN-T5-XL	FLAN-T5-XXL
Cola-FT	56.5	60.6	64.1	65.4
Cola-Zero (2-shot)	30.3	34.6	57.6	61.0
Cola-Zero (0-shot)	28.6	36.0	55.0	59.3
Table 7: A-OKVQA validation set DA performance. Extension of Figure 5.
1-shot	2-shot	3-shot	4-shot
Cola-Zero	60.2	61.0	60.7	59.2
Table 8: Cola-Zero in-context few-shot learning DA performance on A-OKVQA validation set. Extension of Figure 6.
A.5 Qualitative Examples
In this section, we provide more qualitative examples on A-OKVQA (Figure 8), e-SNLI-VE (Figure 9), and VSR (Figure 10) datasets.
Due to the large span of the three figures, for better visibility, we put the detailed description directly in each figure's caption part. We illustrate how Cola-FT and Cola-Zero process the VLMs answers in each example. Overall, in these examples, we can observe that even if BLIP and OFA provide wrong answers, Cola can still present the correct answer based on the captions provided by OFA and BLIP, as well as the choice set. This may illustrate how Cola amazingly accomplishes visual reasoning tasks via coordinating BLIP and OFA.
A.6 Failure Cases
In Figure 11, we provide a few failed cases to analyze the specific behavior of Cola.
The leftmost example's correct answer is kayaking, but there are no hints from OFA and BLIP's answers and captions. Therefore Cola-Zero incorrectly provides the answer OFA without sufficient information as hints, while surprisingly Cola-FT answered correctly from OFA's boating answer.
Figure 8: A-OKVQA qualitative examples. Leftmost: LLM doesn't use BLIP and OFA's answers, but may observe from captions to derive the correct final answer. Left: As shown on the left, LLM does not follow the wrong answers from OFA and BLIP but gets the correct answers from captions. Right: With both OFA and BLIP answering incorrectly, LLM derives the correct one from both VLMs' captions and answers. Rightmost: After assessing the questions, answers, and captions, LLM goes with OFA's answer and rewrites it to match the expression in the choices. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
Question	Why might people sit here?	The room can be described as what?	In what type of location are they playing with the body board?	What is in front of the monitor?
OFA caption	colorful umbrellas on the riverwalk	living room layout and decor medium size how to decorate a small living room dining combo mant	person, left, and person look at a painting of a great white shark.	a desk with a computer, a lamp, a laptop, and a plant.
BLIP caption	a colorful umbrella umbrella with colorful umbrellas	a dining room table with a glass table and chairs	a man holding a surfboard while another man is standing next to him	a desk with a computer and a lamp
Choices	['to testify', 'to rest', 'to shop', 'get tattoo']	['tidy', 'messy', 'on fire', 'destroyed']	['room', 'beach', 'park', 'store']	['keyboard', 'phone', 'mouse', 'headphones']
OFA answer	to eat	living room	bedroom	a keyboard
BLIP answer	yes	dining room	beach	monitor
Cola-Zero answer	to rest	tidy	beach	keyboard
Cola-FT answer	to rest	tidy	room	keyboard
The left example again has insufficient information from captions. While BLIP answers no and OFA answers yes, Cola-FT chooses to answer maybe, which looks natural but unfortunately picks the wrong choice.
The right example's captions contain enough information this time. But both Cola-FT and Cola-Zero are misled by BLIP's wrong answer no parking.
The rightmost example also has insufficient information from captions. In this situation, Cola has no choice but to believe either BLIP or OFA's answer, but it mistakenly prefers BLIP's wrong answer.
A.7 Prompt Templates
Across three datasets, the prompt template is roughly the same, with minor differences mainly in the format of the questions and choices. We list the prompt templates adopted in A-OKVQA and e-SNLI-VE/VSR in Table 9 and Table 10, respectively.
A.8 Parameter-efficient Finetuning
To further reduce the computation cost in model adaptation, we explored parameter-efficient finetuning (PEFT) techniques to reduce finetuning parameter counts. Specifically, we use  $(\mathbb{I}\mathbb{A})^3$  [57], which finetunes an overhead of 1 million parameters, equivalent to  $0.01\%$  of the full parameters of FLAN-T5-XXL.
Compared to full finetuning,  $(\mathbb{I}\mathbb{A})^3$  requires more iterations to converge. The performance of a  $(\mathbb{I}\mathbb{A})^3$  finetuned FLAN-T5-XXL model is on par with a fully finetuned FLAN-T5-Small (80 million parameters) counterpart (Figure 5). Notably, the former is associated with more computation and memory footprint as a consequence of more parameters in the forward pass.
Figure 9: e-SNLI-VE qualitative examples. Leftmost: As the connection to daredevil is not obvious in BLIP and OFA's captions, although Cola-Zero is misled, Cola-FT correctly answers maybe. Left: Similar to the left example, Cola-FT answer correctly as no obvious connections are seen from the captions to this question. Right: Similar to the left example, the fact of catch catfish is not reasonable from the captions, Cola-FT picks the correct answer maybe. Rightmost: As girl gets hit is not obvious in BLIP and OFA's captions and answers, Cola-Zero and Cola-FT both follow BLIP to choose the correct answer no. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
Question	Does the image describe " A professional daredevil"?	Does the image describe " the dog is a shitz " ?	Does this image describe "Two twenty-sometimes prepare to catch salmon while other older men catch catfish" ?	Does this image describe "A little girl gets hit by a woman riding a bike" ?
OFA caption	person doing a flip on a mountain bike	a dog jumping out of the water.	men repairing fishing nets on the beach in zanzibar, Tanzania	a man and a woman on a tandem bike
BLIP caption	a man doing a trick on a bike in the air	a dog jumping over rocks in the water	a man sitting on a boat with a fishing net net	a man and woman riding a bicycle in a parking lot
Choices	['yes', 'maybe', 'no']	['yes', 'maybe', 'no']	['yes', 'maybe', 'no']	['yes', 'maybe', 'no']
OFA answer	yes	no	yes	yes
BLIP answer	yes	no	yes	no
Cola-Zero answer	yes	no	no	no
Cola-FT answer	maybe	maybe	maybe	no
A.9 Extended Ablation Studies
Do caption labels offer useful information to LLM? How would more prompt variations affect the performance of Cola? We tested Cola-Zero with and without caption labels on A-OKVQA validation set, observing a slight decrease in performance when without them (70.39% w/t vs. 69.97% w/o). More ablative experiments showed that removing the VLM's answer labels led to a substantial drop in performance (70.39% w/t vs. 67.62% w/o). Removing the model characteristic descriptions also led to a decrease (70.39% w/t vs. 68.37% w/o).
Do longer image captions improve reasoning performance? On A-OKVQA validation set, we tested longer image descriptions ( $>50$  tokens) but found no gain compared to Cola or single VLMs. Longer captions decreased FLAN-T5+OFA's accuracy by  $0.61\%$  and FLAN-T5 with BLIP by  $0.69\%$  on the A-OKVQA validation set. Cola (captions  $<30$  tokens) reached  $77.73\%$ , outperforming individual VLMs. Longer captions lacked meaningful visual context, possibly due to short text and image pairs in their training datasets. This experiment reaffirms Cola's effectiveness in aggregating individual VLM functionalities.
B.1 Finetuning Large Language Models
Large language models [8, 71, 6] pretrained on massive amounts of unstructured data have gradually demonstrated great performance by finetuning on additional task-specific instances. Finetuning a large language model can be considerably more sample efficient than re-training from scratch, although acceptable performance may still require a considerable quantity of data [95]. Recent