SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liangyu Chen*,†, Bo Li*, Sheng Shen‡ Jingkang Yang
Chunyuan Li\* Kurt Keutzer\* Trevor Darrell\* Ziwei Liu\*,
S-Lab, Nanyang Technological University
$\clubsuit$  University of California, Berkeley  $\clubsuit$  Microsoft Research, Redmond
{lchen025，libo0013，ziwei.liu}@ntu.edu.sg
https://github.com/cliangyu/Cola
Visual reasoning requires multimodal perception and commonsense cognition of the world. Recently, multiple vision-language models (VLMs) have been proposed with excellent commonsense reasoning ability in various domains. However, how to harness the collective power of these complementary VLMs is rarely explored. Existing methods like ensemble still struggle to aggregate these models with the desired higher-order communications. In this work, we propose  $\sqrt{}$  Cola, a novel paradigm that coordinates multiple VLMs for visual reasoning. Our key insight is that a large language model (LLM) can efficiently coordinate multiple VLMs by facilitating natural language communication that leverages their distinct and complementary capabilities. Extensive experiments demonstrate that our instruction tuning variant,  $\sqrt{}$  Cola-FT, achieves state-of-the-art performance on visual question answering (VQA), outside knowledge VQA, visual entailment, and visual spatial reasoning tasks. Moreover, we show that our in-context learning variant,  $\sqrt{}$  Cola-Zero, exhibits competitive performance in zero and few-shot settings, without finetuning. Through systematic ablation studies and visualizations, we validate that a coordinator LLM indeed comprehends the instruction prompts as well as the separate functionalities of VLMs; it then coordinates them to enable impressive visual reasoning capabilities.
(a) Comparison between Paradigms
Action ① - VLMs describe the visual context.
How many vehicles have headlights on?
Action ③ - LLM coordinates and answers.
(c) Cola Solution to the Example VQA
v many vehicles have headlights on?
(b) Example Visual Reasoning Task: VQA
Figure 1: We propose, Cola, using a coordinative language model for visual reasoning. Cola coordinates multiple pretrained VLMs based on the visual context and plausible answers they provide.
(A) One
(B) None
(C) Two
(D) Three
†Project Lead.
Corresponding Author.
Visual reasoning is a crucial task that demands models to not only comprehend and interpret visual information but also to apply high-level cognition to derive logical solutions [40, 120, 60]. The field has received significant attention from the machine learning community because of its potential to enable a wide range of intelligent applications, such as intelligent tutoring systems [5, 94, 69], automated image captioning [103], and virtual assistants [88, 50]. To perform visual reasoning effectively, a model must possess both visual perception capabilities and strong logic reasoning abilities.
While classic visual reasoners typically rely on complex architectures [117, 61, 116] or are unable to generalize beyond the training dataset [121, 72], recent advancements in large pretrained models have shown that vision-language models (VLMs) can achieve impressive performance on visual reasoning tasks even under zero-shot settings [104, 52, 51, 49, 48, 3]. Meanwhile, large language models (LLMs) have also demonstrated robust zero-shot commonsense reasoning abilities on the natural language processing (NLP) applications [8, 15, 109]. Several recent studies have attempted to combine such complementary VLMs and LLMs for visual reasoning. For example, PICa [115] utilizes image captioning models to generate textual prompts for GPT-3 [8], and adapts GPT-3 to solve the visual question answering (VQA) tasks in an in-context few-shot learning manner. Socratic Models [123] allow VLMs and LLMs to communicate through prompt engineering to unlock zero-shot multimodal reasoning capabilities.
On the premise that current studies have focused on the interactions among heterogeneous models (specifically, among VLM and LLMs), in this work, we shift to examine how to reconcile homogeneous expert models (e.g., multiple VLMs) with an LLM in a coordinative paradigm. Inspired by the findings in CICERO [65] that LLMs capture strong strategic planning and negotiation abilities in coordinating multiple agents, we propose  $\nabla$  Cola, a novel model ensemble approach that utilizes an LLM as the coordinator in between multiple VLMs. A key finding of this study is that given multiple VLMs with different preferred patterns in describing the visual context and predicting plausible answers in natural languages, an LLM can coordinate and integrate their respective strengths efficiently and effectively. We present two variants of Cola, namely  $\nabla$  Cola-FT and  $\nabla$  Cola-Zero, where FT corresponds to an instruction finetuning approach and Zero stands for an in-context learning approach to adapt the coordinator LLM for visual reasoning. Figure 1 provides an overview of Cola and conventional model ensemble approaches.
Existing work on model ensembles usually focuses on manipulating model weights [36] or predictions [111, 22], while remaining cumbersome, if possible, to implement on prevalent end-to-end black box model APIs, like GPT-4 [70], Google Bard, Anthropic Claude, etc. To address this issue, prompt ensembles [107, 106, 73] sample model outputs (e.g., rationales) in natural languages to boost chain-of-thought reasoning [109]. Recent studies on augmented LLM such as [86, 93, 58] have delved into developing a comprehensive strategy that enables LLMs to utilize external tools. These tools comprise multiple off-the-shelf models, web search engines, Python functions [97], and rule-based modules, which are instrumental in performing complex tasks. Despite these efforts, the power of prompt ensembles to aggregate multiple models remains untouched. In contrast, we show that Cola leverages language prompts generated from multiple expert models to make model ensembles.
Systematic experiments demonstrate that Cola performs at the pinnacle of ability on VQA, outside knowledge VQA, visual entailment, and visual spatial reasoning tasks. Specifically, Cola-FT achieves state-of-the-art performance on A-OKVQA [89], OK-VQA [63], e-SNLI-VE [21], and VSR datasets [56], even when compared with methods that adopt larger models or require more training computations. Cola-FT also demonstrates competitive capabilities on VQA v2 [27], and compositional reasoning tasks (GQA [35] and CLEVR [40]). Perhaps surprisingly, we find that Cola-Zero demonstrates comparable performance without finetuning, as an emerging ability of larger language models. Compared to a single VLM and ensemble modeling, both Cola-FT and Cola-Zero improve the performance substantially across most datasets. They are even effective with the recent large multimodal models like InstructBLIP [18] which embeds a pretrained LLM inside itself. Besides, we conduct a thorough analysis of perturbed VLM caption or plausible answer labels and saliency visualization to investigate how Cola recognizes each VLM's individual functionalities and then performs coordination behaviors. We conjecture that, in principle, any language-expressing reasoning task can be usefully augmented with coordinative language models that learn to aggregate multiple expert models, even via in-context learning.
- Cola: a novel paradigm that utilizes a language model as a coordinator between multiple vision-language models to integrate their respective strengths for visual reasoning (§2).
- State-of-the-art performance: Cola achieves the pinnacles on a challenging suite of diverse visual reasoning tasks and datasets(§3.2).
- Systematic analysis: our experiments reveal how Cola comprehends the instruction prompts, then coordinates them to capture impressive visual reasoning capabilities (§3.3, §3.4, §3.6).
2 Cola
We formulate various visual reasoning tasks as a multi-class classification problem for simplicity. Given an image  $v \in \mathcal{V}$  and a question-like prompt  $q \in \mathcal{Q}$ , the reasoner is required to select an answer  $a$  from the candidate set  $\mathcal{A} = \{a\}$ . In the case that the reasoner outputs a text sequence  $s_{v,q}$ , we map  $s$  to a prediction  $P(v, q) = \mathrm{sim}(T(s_{v,q}), T(\{a\}))$  where  $T$  transforms text sequences into text embeddings (we use a all-mptnet-base-v2 model [81] here), and sim denotes cosine similarity.
Ensemble Modeling aggregates multiple models' predictions in order to improve the overall performance (Figure 1a). For instance, one common practice is averaging over  $n$  models:
$$
P (v, q) = \frac {1}{n} \sum_ {i = 1} ^ {n} P _ {i} (v, q), \tag {1}
$$
where  $P_{i}(v,q)$  denotes the prediction of the  $i^{th}$  model on input  $(v,q)$ .
2.1 Cola & Templates
An overview of Cola is shown in Figure 1c. We use OFA [104] and BLIP [52] as the VLMs. LLMs include encoder-decoder (FLAN-T5 [16]) and decoder-only (Vicuna-1.5 [125], Mistral [39]) transformers. We first prompt each VLM to output captions and plausible answers independently. We then concatenate the instruction prompt, the question with choices, captions, and plausible answers to fuse all contexts for the LLM to reason, coordinate, and answer.
Image captioning gives important visual context to reason from. We first employ  $i^{th}$
VLM to describe each image respectively to get visual descriptions  $c_{i}(v)$ . We use ofa-large for OFA and blip-image-captioning-large for BLIP, both implemented by the Hugging Face Transformers library [110].
General Prompt Template
Answer the following multiple-choice question by OFA and BLIP's description and their answers to the visual question. OFA and BLIP are two different vision-language models to provide clues. OFA's description: <OFA caption> BLIP's description: <BLIP caption> Q: <Question> OFA's answer: <OFA answer> BLIP's answer: <BLIP answer> Choices: <Choices to the question> A:
Table 1: LM prompt template. The LM is instructed to coordinate VLMs. Each question set defines visual context, question (and choices), and plausible answers.
Plausible answers by the VLMs to the question provide clues and patterns of VLMs for the LM to consider and coordinate. Similar to captioning, we prompt each  $i^{th}$  VLM using the image-question pair to get a plausible answer  $\hat{a}_i(v,q)$ . We use ofa-large for OFA and blip-vqa-base for BLIP. Following OFA, our prompt template varies by task category. For the VQA tasks, we leave the original question unchanged. For the visual entailment and visual spatial reasoning tasks, our prompt template is "does the image describe "<text premise>"?"
Prompt template is shown in Table 1. First, we designed an instruction prompt for LM to understand the requirement to coordinate VLMs to answer the visual reasoning question. We then concatenate the captions from each VLM model, with the VLM identification labels in natural languages (referred to as VLM caption labels in Figure 3), such as "OFA's description: <OFA caption>". Next, the question and its plausible answers provided by VLMs (with similar identification labels referred to as VLM answer labels in Figure 3) are concatenated. We follow [16] to
include the choices of question (for multiple-choice questions in A-OKVQA, e-SNLI-VE, and VSR) and "A:" to prompt for answers. Overall, the prompt for LLM input is given by:
$$
\operatorname {P r o m p t} (v, q) = \operatorname {T e m p l a t e} \left(\left\{\left(c _ {i} (v), \hat {a} _ {i} (v, q)\right) \mid i = 1, \dots , n \right\}\right). \tag {2}
$$
More specific prompt templates on each dataset are provided in Appendix A.7.
2.2 Cola-FT
Instruction Tuning of Cola is initialized with pretrained checkpoints. Given the question  $q$  based on the image  $v$ , the LM predicts the answer in the form of sequence
$$
s _ {v, q} = \operatorname {L L M} (\operatorname {P r o m p t} (v, q)). \tag {3}
$$
To optimize the LLM, we use the language modeling loss for next-token prediction, with the teacher forcing strategy. We only finetune the LLM (while not the VLMs) to follow the common paradigm of ensemble modeling and simplify the method (Figure 1).
Inference deploys the same prompt as Table 1 to align with instruction tuning. We resort to the greedy decoding strategy for conditional sequence generation at both instruction tuning and inference.
2.3 Cola-Zero
In-context learning is an emerging ability of the LLM models pretrained on documents of long-range coherence. By learning input and output format from demonstration, in-context learners learn to perform a downstream task simply by conditioning on a prompt consisting of input-output examples [114]. The coordinator LLM, finetuned on instruction prompts with examples, is capable of in-context few-shot learning and zero-shot learning (see Figures 6 and 7).
Cola-Zero is the in-context few-shot/zero-shot learning variant of Cola, without instruction tuning. For in-context  $k$ -shot learning, we modify the prompt (Table 1) to include  $k$  input-output examples sampled from the training set. For zero-shot learning, the prompt remains the same as Table 1.
First, the experimental setups and basic methods are described in this section. The main quantitative results are then presented in Table 2. Next, we analyze qualitative visualizations and scaling to verify the effectiveness of the Cola paradigm in different settings. Further details on datasets, training, evaluation, and experimental analysis can be found in Appendix A.
3.1 Baseline Methods
State-of-the-art Methods are in two broad categories, VLM alone, and VLM combined with LLM. In Table 2, for a fair comparison, we detail the techniques (whether finetuning or in-context learning is required) used for training VLMs and LLMs, and the number of training epochs.
Ensemble Modeling can be considered the most basic baseline for aggregating VLMs. It represents the base performance that the combination of VLMs can achieve on the target task when not trained. We implement an averaging ensemble (Equation (1)) of cosine similarity between VLM output and each choice of a question as our ensemble baseline.
3.2 Overall Performance
In Table 2, we first observe that Cola-FT achieves state-of-the-art (SOTA) performance on four datasets (A-OKVQA, OK-VQA, e-SNLI-VE, VSR), with merely 1 epoch of instruction tuning and a medium-sized language model. In contrast, many previous SOTA methods require finetuning more epochs than Cola-FT (e.g., VLC-BERT, PromptCap on A-OKVQA). Some also use much larger language models, such as GPT-3 (175B) [8] and OPT (175B) [124]. Cola-FT outperforms OFA-X on e-SNLI-VE, although the latter is finetuned on much more related tasks and data (c.f.
Figure 2: Qualitative examples. The correct choices are underlined. Leftmost: a commonsense question example of A-OKVQA; LLM follows the answer of BLIP. Left: a visual question example of A-OKVQA; LLM follows the answer of OFA. Right: an example of e-SNLI-VE; LLM chooses another option after coordination. Rightmost: an example of VSR; LLM predicts based on the caption of OFA and the answer of BLIP. Cola-Zero answers are inferred in zero-shot settings. The bottom row, Cola-FT ( swapped VLM answer labels), indicates that the LLM follows the answer of certain VLMs based on their separate functionalities. LLM answers are associated with the distribution of VLM answer labels.
Question	What type of shot is the man hitting?	What appliance is next to an appliance that is highly decorated?	Does this image describe "puppy running after a stick in grass"?	Does this image describe "The truck is away from the elephant" ?
OFA caption	tennis player hits a return to tennis player during their men's singles second round match at	a refrigerator covered in a variety of stickers.	a coyote is seen in this undated file photo. (credit: ktla	an elephant is loaded onto a truck in yangon. photo: afp
BLIP caption	a man in a blue shirt is playing tennis	a refrigerator with many pictures on it	a dog running through the grass in a field	a man riding a motorcycle with a truck behind him
Choices	['forehand', 'backhand', 'serve', 'dropshot']	['mixer', 'stove/oven', 'refrigerator', 'microwave']	['yes', 'maybe', 'no']	['yes', 'no']
OFA answer	backhand	stove/oven	yes	yes
BLIP answer	forehand	microwave	no	no
Cola-Zero answer	forehand	stove/oven	no	no
Cola-FT answer	forehand	stove/oven	maybe	no
Cola-FT answer (swapped VLM answer labels)	backhand	microwave	maybe	yes
Cola-FT is trained on each one dataset only in Table 2). In addition, the lighter variant Cola-Zero also achieves comparable performance to most baseline methods through in-context few-shot and zero-shot learning, without training any model parameter. To evaluate the performance of Cola with large multimodal models that embed large language models inside, we also assembled InstructBLIP [18] models based on FLAN-T5 XL and XXL and tested on A-OKVQA dataset. In Table 2, we report the multiple-choice accuracies on A-OKVQA. See Appendix A.4 for direct answer results.
3.3 Qualitative Examples
In Figure 2, we exhibit several qualitative examples. The language coordinator determines the correctness of VLM plausible answers implicitly, given their captions and the caption and answer labels. The leftmost example (a tennis player playing) demonstrates a case when captions are not informative to guide the LLM for predictions. Between OFA and BLIP's plausible answers, the LLM follows the answer of BLIP. In contrast, in the left example (an oven next to a fridge), again with trivial captions, the LLM follows OFA's plausible answer instead.
It's all plausible answers, captions, VLM answer/caption labels, and the world knowledge the LLM encodes in itself, that contribute to the final decision of the language coordinator. The rightmost example presents the scenario of inconsistency between captions and answers. OFA describes the image as "an elephant is loaded onto a truck in yangon." Though, it agrees that "the truck is away from the elephant". With Cola-FT, The LLM coordinates OFA's correct caption and BLIP's correct answer to make a reasonable prediction.
Notably, we observe a scenario in which captions can be more informative than plausible answers to guide LLM. The right example (a puppy running) presents an uninformative image. Though neither OFA nor BLIP succeeds in answering the question, the LLM chooses to answer with "maybe" based on the given visual context. See Appendix A.5 and Appendix A.6 for more analysis on qualitative examples and failure cases.
3.4 Coordination Analysis
Overall, Figure 3 validates the efficacy of Cola to coordinate VLMs. All the experiments use the same prompt template as in Table 1 unless otherwise stated. On A-OKVQA validation set, the performance of a single VLM (w/o FLANT5) is  $50.83\%$  for BLIP, or  $54.75\%$  for OFA. To validate the effectiveness of multi-VLM collaboration, we first ablate single-VLM variants of Cola-FT, shown as #1 (OFA only, without BLIP) and #2 (OFA only, without OFA) from the top. As expected, both fall behind Cola-FT (#8) by a large margin. With both VLMs, we ablate VLMs' captions (#3) and VLMs' plausible answers (#4), which reveal that plausible answers are much more significant in helping the LLM answer visual reasoning questions. Next, we perturb caption labels by swapping the VLM caption labels at instruction tuning and evaluation (#5), specifically "OFA's description: " and "BLIP's description: ", by a chance of  $50\%$ . Under such settings, the LLM fails to acquire the preferred patterns of VLM for
captioning, though the overall visual context is preserved. The results underperform Cola-FT, which verifies that VLM caption labels improve Cola-FT performance. Notably, the VLM (plausible) answer labels are more important to the LLM's decision: a considerable gap exists between (#6) and Cola-FT. In #6, the LLM fails to learn the separate functionalities of VLM when answer labels are perturbed. This highlights that the performance gained from the coordination between BLIP and OFA, but not the strong reasoning capabilities of the LLM, FLAN-T5.
Naturally, we ask what if the LLM can learn the patterns each VLM answers, but they cannot apply it at inference? We input correct VLM answer labels at instruction tuning and swap labels at evaluation (#7). Consequently, #7 falls behind Cola-FT with a smaller but still considerable margin. The results suggest that learning and applying the separate functionalities of VLMs is important for the coordinator LLM to make predictions. See Appendix A.9 for more ablation studies.
OFA only + FLAN-T5
BLIP only + FLAN-T5
VLMs' plausible answers only
VLMs' captions only
FT: perturb VLM caption labels  
Eval: perturb VLM caption labels
FT: perturb VLM answer labels  
Eval: perturb VLM answer labels
FT: correct VLM answer labels  
Eval: swap VLM answer labels
FT: correct VLM answer labels  
Eval: correct VLM answer labels
Figure 3: Ablation study results using a single VLM (#1, #2 from top), VLMs' plausible answers only (w/o captions, #3), VLMs' captions (w/o plausible answers, #4), perturbed VLM caption/answer labels at instruction tuning (#5, #6), and swapped answer labels at evaluation (#7). In #6, the coordination prior cannot be learned by the LLM. In #7, the coordination prior can be learned by the LLM, but cannot be properly applied at evaluation. FT: instruction tuning; Eval: evaluation.
3.5 Scaling Cola
Methods	A-OKVQA	e-SNLI-VE
OFA-base (1)	45.76	52.60
OFA-base (2)	46.07	51.70
OFA-base (3)	45.73	52.33
Ensemble (majority voting)	44.79	52.71
Ensemble (average)	46.04	52.25
Cola-Zero (2-shot)	47.71	54.42
Cola-FT	48.85	56.92
Table 3: Performance of ensemble methods based on three identical models.
Methods	A-OKVQA	e-SNLI-VE
OFA-tiny	39.03	50.20
OFA-medium	42.45	51.04
OFA-base	45.76	52.60
Ensemble (majority voting)	46.71	53.94
Ensemble (average)	46.62	54.41
Cola-Zero (2-shot)	49.37	57.63
Cola-FT	54.26	63.68
Table 4: Performance of ensemble methods based on three different models.
Scaling Cola with More VLMs. By decoding the top-k ( $k = 5$ ) results from three identical (OFA-base) models on the A-OKVQA validation set, both the answers and captions may exhibit slight variations. Cola demonstrated significant performance improvements compared to a single VLM or ensemble, as shown in Table 3. Furthermore, the performance gap between the ensemble baselines and Cola based on three different models (OFA-tiny/medium/base) is even more substantial, as depicted in Table 4.
We further scale the number of OFA-base and find that Cola-FT saturates at 5 VLMs  $(49.77\%)$  and Cola-Zero (2-shot in-context learning) saturates at 3 VLMs  $(47.71\%)$ . We observe that long input harms the performance of Cola-Zero, which is negative for scaling with more VLMs (Figure 4). LMs with a larger context window size [12, 68, 9] are promising to further improve the performance of Cola, which we leave for future works.
Scaling Model Size. We conduct experiments on scaling the coordinator LLM size to see if there are ramifications when operating at a larger scale. Figure 5 reveals that Cola-FT performance increases as the LLM (FLAN-T5) model size increases. Notably, Cola-FT/small, with only 80M parameters, could achieve  $65\%$  MC accuracy on A-OKVQA validation set, which is far beyond our baseline methods  $(55\%)$ . Cola-Zero, under
Figure 4: Scaling of # Models.
the in-context learning paradigm, achieves competitive performance when the model grows to a billion-parameter scale. This observation on Cola-Zero can be regarded as a proof-of-concept that potentially reveals Cola-Zero's emerging abilities (inherited from FLAN-T5 [16]) on visual reasoning tasks at a relatively large scale. Cola-FT is effective with small models, but Cola-Zero is an emerging ability on larger models only.
Figure 5: Cola performances versus the LLM (FLAN-T5) sizes.
Figure 6: Low-data Cola-FT and Cola-Zero performances.  $x$ -axis is distorted for optimal display.