Table 2: Overall performance. Model Spec. denotes specification where we summarize the detailed VLMs and LMs adopted in each method and their parameters. FT and ICT denote finetuning and in-context learning, respectively. Downward arrows indicate that fewer FT and ICT are more efficient. The accuracy metric varies slightly in different datasets. In A-OKVQA, we report both val/test accuracies, and val accuracy in VQA v2, OK-VQA, e-SNLI-VE, GQA, and CLEVR; test (zero-shot split) accuracy in VSR. Upward arrows indicate higher accuracy is better. We mark the best performance on each dataset with bold font and second-best with underlines.
Method	Vision-language Model	Large Language Model	Accuracy ↑
Model Spec.	FT ↓	Model Spec.	ICL ↓	FT ↓
Visual Question Answering (VQA v2)
MetaLM [31]	Pretrained Encoder	350k steps	MetaLM (1.3B)	-	350k steps	41.1
PNP-VQA [99]	BLIP-Caption (446M)	-	UnifiedQAv2 [43] (11B)	0-shot	-	63.3
BLIP-2 [51]	CLIP [76] (1.2B trainable)	5 epochs	FLAN-T5 (3B)	-	-	81.6
BLIP-2 [51]	CLIP [76] (1.2B trainable)	5 epochs	OPT [124] (6.7B)	-	-	82.2
Ensemble	-	-	-	-	-	68.0
Cola-Zero	BLIP+OFA (384M+472M)	-	FLAN-T5 (11B)	2-shot	-	69.1
Cola-FT	-	FLAN-T5 (11B)	-	1 epoch	83.7
Outside Knowledge Visual Question Answering, Multiple Choice (A-OKVQA)
PromptCap [32]	OFA (472M)	2 epochs	GPT-3 (175B)	0-shot	-	-/73.2
Img2Prompt [29]	BLIP (384M)	-	OPT (175B)	0-shot	-	42.9 / 40.7
Prophet-MC [90]	MCAN-large [118] (56M)	6 epochs	GPT-3 (175B)	16-shot	-	76.4 / 73.6
Ensemble	-	-	-	-	-	56.6 / 54.9
Cola-Zero	-	-	FLAN-T5 (11B)	0-shot	-	65.4 / 61.6
Cola-Zero	-	-	FLAN-T5 (11B)	2-shot	-	70.4 / 66.5
Cola-FT	-	FLAN-T5 (11B)	-	1 epoch	77.7 / 74.0
Cola-Zero	InstructBLIP [18]	-	FLAN-T5 (11B)	0-shot	-	68.0 / 66.5
Cola-Zero	XL+XXL	-	FLAN-T5 (11B)	2-shot	-	72.3 / 72.3
Cola-FT	(3B+11B)	-	FLAN-T5 (11B)	-	1 epoch	78.1 / 76.7
Cola-Zero	-	Vicuna (7B)	2-shot	-	63.9 / 63.0
Cola-Zero	-	Vicuna (7B)	-	1 epoch	68.6 / 66.9
Cola-Zero	-	Mistral (7B)	2-shot	-	69.3 / 66.2
Cola-FT	-	Mistral (7B)	-	1 epoch	74.3 / 71.8
Outside Knowledge Visual Question Answering, Direct Answer (OK-VQA)
PromptCap [32]	OFA (472M)	2 epochs	GPT-3 (175B)	0-shot	-	58.8
Prophet [90]	MCAN-large [118] (56M)	6 epochs	GPT-3 (175B)	16-shot	-	61.1
Ensemble	-	-	-	-	-	39.2
Cola-Zero	BLIP+OFA (384M+472M)	-	FLAN-T5 (11B)	0-shot	-	39.4
Cola-Zero	-	FLAN-T5 (11B)	2-shot	-	39.4
Cola-FT	-	FLAN-T5 (11B)	-	1 epoch	62.4
Visual Entailment (e-SNLI-VE)
e-UG [42]	UNITE (86M)	400 epochs	GPT-2 (117M)	-	400 epochs	79.5
OFA-X [74]	OFA (472M)	10 epochs	-	-	-	80.9
Ensemble	-	-	-	-	-	48.8
Cola-Zero	-	-	FLAN-T5 (11B)	0-shot	-	56.2
Cola-Zero	-	FLAN-T5 (11B)	2-shot	-	57.8
Cola-FT	-	FLAN-T5 (11B)	-	1 epoch	81.6
Visual Spatial Reasoning (VSR)
VisualBERT [53]	VisualBERT (110M)	100 epochs	-	-	-	54.0
LXMERT [98]	LXMERT (110M)	100 epochs	-	-	-	63.2
ViLT [44]	ViLT (88M)	30 epochs	-	-	-	62.4
Ensemble	-	-	-	-	-	51.4
Cola-Zero	BLIP+OFA (384M+472M)	-	FLAN-T5 (11B)	0-shot	-	55.8
Cola-Zero	-	FLAN-T5 (11B)	2-shot	-	54.9
Cola-FT	-	FLAN-T5 (11B)	-	1 epoch	67.0
Compositional Question Answering, Real Images (GQA)
BLIP [52]	BLIP (384M)	-	-	-	-	41.7
OFA [104]	OFA (472M)	-	-	-	-	58.0
VisProg [30]	ViLT (88M)	-	GPT-3 (175B)	8-shot	-	50.5
Cola-FT	BLIP+OFA (384M+472M)	-	FLAN-T5 (11B)	-	1 epoch	60.3
Compositional Question Answering, Synthetic Images (CLEVR)
InstructBLIP [18]	XL (3B)	-	-	-	-	33.7
XXL (11B)	-	-	-	-	16.6
Cola-Zero	InstructBLIP	-	FLAN-T5 (11B)	2-shot	-	34.4
Cola-FT	XL+XXL (3B+11B)	-	FLAN-T5 (11B)	-	1 epoch	54.3