SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Question	Does this image describe "The truck contains the elephant"?	Does this image describe "The bed is under the handbag"?	Does this image describe "The couch is behind the hot dog"?	Does this image describe "The bowl contains the banana"?
OFA caption	an elephant being transported on a truck in sri lanka	a black and white tuxedo cat with a white nose, yellow eyes, and white	person enjoying a meal by the fire	bananas and mangoes in a bowl
BLIP caption	a truck with a large elephant in the back of it	a black cat laying on a bed with a pillow	a man sitting on a couch with a plate of food	a bowl of fruit is shown in this bowl
Choices	['yes', 'no']	['yes', 'no']	['yes', 'no']	['yes', 'no']
OFA answer	yes	no	yes	yes
BLIP answer	no	no	yes	no
Cola-Zero answer	no	no	no	yes
Cola-FT answer	yes	no	no	yes
Figure 11: Failed cases. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
Figure 10: VSR qualitative examples. Leftmost: As OFA caption mentioned elephant being transported and OFA provides the correct answer, Cola-FT follows OFA's choice. Left: As OFA and BLIP provide the same answer, Cola-Zero and Cola-FT follow the choice. Right: As the captions do not provide obvious information, even BLIP and OFA provide the same answer, Cola-Zero and Cola-FT are not misled to the wrong choice. Rightmost: As the captions provide strong clue bananas in a bowl, although BLIP's answer is incorrect, Cola-Zero and Cola-FT still choose the correct answer. The correct choices are underlined. Cola-Zero answers are given in zero-shot settings.
Question	What are the people doing in the water?	Does the image describe " The man is making a vase"?	What kind of zone is this bike parked in?	Does this image describe "The motorcycle is beside the truck" ?
OFA caption	black and white photo of a man on a bike looking at a canoe in the river	person on the potter's wheel	a city made by peoplebucharest	men walking past a truck in kabul, afghanistan.
BLIP caption	a man and woman on a bike in a park	a man is sitting on a chair and is using a wheel	a bicycle parked next to a pedestrian crossing sign	a man walking down the street in a city
Choices	['surfing', 'fishing', 'kayaking', 'swimming']	['yes', 'maybe', 'no']	['temporary', 'pedestrian', 'no parking', 'handicap']	['yes', 'no']
OFA answer	boating	yes	pedestrian	yes
BLIP answer	swimming	no	no parking	no
Cola-Zero answer	OFA	no	no parking	no
Cola-FT answer	kayaking	maybe	no parking	no
works have finetuned task-specific models that demonstrate amazing capabilities in many real-world applications, such as Copilot for program synthesis [11].
VQA Prompt Template
Answer the following multiple-choice question by OFA and BLIP's description and their answers to the visual question. OFA and BLIP are two different vision-language models to provide clues.
OFA's description: <OFA caption>  
BLIP's description: <BLIP caption>
Q: <Question>
OFA's answer: <OFA answer>  
BLIP's answer: <BLIP answer>
Choices: <Choices to the question>
Table 9: VQA prompt template for the LLM, for VQA v2 / OK-VQA / A-OKVQA. The LLM is instructed to coordinate VLMs. Each question set defines visual context, question with choices, and plausible answers.
e-SNLI-VE / VSR Prompt Template
OFA's description: <OFA caption> BLIP's description: <BLIP caption>
Q: does the image describe <hypothesis>?
e-SNLI-VE Choices: [yes, no, maybe]  
VSR Choices: [yes, no]
Table 10: e-SNLI-VE/VSR prompt template for the LLM. The LLM is instructed to coordinate VLMs. Each question set defines visual context, hypothesis, and plausible answers.
B.2 Instruction-based Learning
Recent advances in the capabilities of language models have piqued researchers' curiosity in the field of instruction-based learning [26, 64, 87, 24]. The core of instruction-based learning is to explore the knowledge of the language model itself. In contrast to prompt learning to stimulate the language model's ability to complete blanks, instruction tuning more focuses on activating the language model's comprehension by giving obvious instructions to models and expecting correct feedback. Earlier work [67] finetune BART [46] using instructions and few-shot exemplars in question answering, text classification, and text modification. Their findings suggest that few-shot instruction tuning improves performance on unseen tasks. [66] finetunes GPT-2 Large and also observes that few-shot exemplar instruction tuning could improve performance. [85] finetunes T5-11B with more diverse instruction templates and observe similar improvements in zero-shot learning. More recent work [108] performs large-scale experiments with a 137B FLAN-T5 model and instruction-tune it on over 60 datasets verbalized via instruction templates. They observe FLAN-T5 substantially improves over zero-shot GPT-3 (175B) on 20 of 25 evaluation datasets. OpenAI also releases InstructGPT [71] based on GPT-3 [8], it makes use of human annotations to steer desired model behavior through both instruction
Accuracy	# Finetuning Params
Finetuning	77.73	11B (100%)
PEFT, (IA)3	63.76	1M (0.01%)
Table 11: (IA)3 [57] parameter-efficient tuning (PEFT) performance. We finetune a FLAN-T5-XXL model on the A-OKVQA training set and evaluate it on the A-OKVQA validation set.
tuning and reinforcement learning of human feedback. They discover that InstructGPT is favored by humans over unmodified GPT-3.
B.3 Visual Reasoning
Beyond the uni-modal reasoning tasks such as question answering (QA) [100, 41, 14, 80, 79, 23, 75, 17, 96, 25, 127, 119, 7], visual reasoning requires models to not only understand and interpret visual information but also to apply high-level cognition to derive rational solutions [40, 35, 4, 59, 60, 84, 120, 34]. Several tasks have been introduced to address visual reasoning, such as visual question answering (VQA) [1], in which models are expected to provide answers to questions related to an image and visual entailment (VE) [113], where the model is required to determine the similarity or relationship between a given image and a description. Classic visual reasoning methods have employed an image encoder and a text encoder, along with a reasoning block that utilizes attention mechanisms [121, 72, 122, 105], neuro-symbolic methods [117, 61, 116], or external knowledge [62, 28, 13] to perform reasoning.
Recent progress in large pre-trained models has led to the development of language models (LLMs) that possess exceptional commonsense reasoning capabilities [78, 16, 15, 77]. These models can potentially replace the reasoning block in visual reasoning tasks, and LLMs' lack of perception can be compensated by incorporating multiple vision-language models (VLMs) trained on different domains [76, 104, 52]. For example, PICa [115] converts the image into captions that GPT-3 [8] can understand, and adapts GPT-3 to solve the VQA task in a few-shot manner by providing a few in-context VQA examples. However, there is still a lack of research on how to harness the collective power of these complementary VLMs for visual reasoning tasks.
B.4 Model Ensembling
Model ensembling is a powerful machine learning technique that combines the predictions of multiple models to improve the overall performance of a given task [20]. Classic model ensembling methods include simple averaging, weighting the predictions based on model performance, and stacking the models. By combining the predictions of multiple models, ensembling can reduce the variance and bias of the final predictions, resulting in a more robust and accurate model [83]. Ensemble methods have been shown to perform well in a wide range of tasks, including image classification, natural language processing, and time series forecasting. However, when it turns to multimodal tasks such as visual reasoning, a simple combination is not applicable to heterogeneous models as their inputs and outputs vary.
The Mixture-of-Experts (MoE) [91, 82, 126, 45, 47] can be conceptualized as a model ensemble strategy implemented at the level of network architecture. MoE-based multi-modal models [33] excel in leveraging the specific strengths of each expert, thereby delivering the performance that often outstrips that of any individual expert. In these networks, the credibility of each expert's output is dynamically weighted, facilitating a comprehensive and nuanced response to multimodal tasks.
However, even within this sophisticated framework, challenges can arise, particularly when managing heterogeneous pre-trained multimodal models. To address this problem, an innovative approach known as Socratic Models (SMs) [123] has been proposed. SMs employ prompt engineering to guide these diverse models through multimodal discussions, effectively combining their varied knowledge. This method promotes a more harmonious and effective integration of different models, enhancing the ensemble's ability to handle complex tasks.
With a similar goal, [54] proposes a closed-loop iterative consensus optimization method to utilize the strengths of individual models. However, previous methods do not fully explore the potential of a centralized solution or adapt to the separate functionalities of different models, particularly in the
visual reasoning scenario. Recent studies, such as CICERO [65], have shown that language models possess strong capabilities in coordinating multiple agents, which inspires us to reorganize pre-trained multimodal models with a focus on the language models.
This study inherits ethical risks of biases from pretrained VLMs and LLMs, depending on their training data. We suggest the users consider the possible biases in reasoning and prompt the model to interpret its predictions in natural languages when necessary.