SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6 presents the in-context learning performance of language tasks. KOSMOS-1 achieves comparable or even better performance in cloze completion and commonsense reasoning tasks when compared to LLM. In terms of the average result across all these datasets, LLM performs better in zero-shot and one-shot settings, whereas our model performs better in few-shot  $(k = 4)$  settings. In addition, Section 3.9.2 shows that MLLMs learn better visual commonsense knowledge compared with LLMs.
Task	Zero-shot	One-shot	Few-shot (k=4)
LLM	KOSMOS-1	LLM	KOSMOS-1	LLM	KOSMOS-1
StoryCloze	72.9	72.1	72.9	72.2	73.1	72.3
HellaSwag	50.4	50.0	50.2	50.0	50.4	50.3
Winograd	71.6	69.8	71.2	68.4	70.9	69.8
Winogrande	56.7	54.8	56.7	54.5	57.0	55.7
PIQA	73.2	72.9	73.0	72.5	72.6	72.3
BoolQ	56.4	56.4	55.1	57.2	58.7	59.2
CB	39.3	44.6	41.1	48.2	42.9	53.6
COPA	68.0	63.0	69.0	64.0	69.0	64.0
Average	61.1	60.5	61.2	60.9	61.8	62.2
3.9 Cross-modal Transfer
Cross-modal transferability allows a model to learn from one modality (such as text, image, audio, etc.) and transfer the knowledge to the other modalities. This skill can enable a model to perform various tasks across different modalities. In this part, we evaluate the cross-model transferability of KOSMOS-1 on several benchmarks.
3.9.1 Transfer from Language to Multimodal: Language-Only Instruction Tuning
To evaluate the effect of language-only instruction tuning, we conduct an ablation study using four datasets: COCO, Flickr30k, VQAv2, and VizWiz. These datasets consist of image captioning and visual questions answering. The evaluation metrics are: CIDEr scores for COCO/Flickr30k and VQA accuracy for VQAv2/VizWiz.
Table 7 shows the experimental results. Language-only instruction tuning boosts our model's performance by 1.9 points on Flickr30k, 4.3 points on VQAv2, and 1.3 points on VizWiz. Our experiments show that language-only instruction tuning can significantly improve the model's instruction-following capabilities across modalities. The results also indicate that our model can transfer the instruction-following capability from language to other modalities.
Table 6: Performance comparisons of language tasks between KOSMOS-1 and LLM. We use the same textual data and training setup to reimplement a language model. Both models do not use instruction tuning for fair comparisons.
Model	COCO	Flickr30k	VQAv2	VizWiz
KOSMOS-1	84.7	67.1	51.0	29.2
w/o language-only instruction tuning	87.6	65.2	46.7	27.9
Table 7: Ablation study on language-only instruction tuning. We report CIDEr scores for COCO and Flickr30k, and VQA accuracy scores for VQAv2 and VizWiz.
3.9.2 Transfer from Multimodal to Language: Visual Commonsense Reasoning
Visual commonsense reasoning tasks require an understanding of the properties of everyday objects in the real world, such as color, size, and shape. These tasks are challenging for language models because they may require more information about object properties than what is available in texts. To investigate the visual commonsense capabilities, we compare the zero-shot performance of KOSMOS-1 and LLM on three object commonsense reasoning datasets, RELATIVESIZE [36], MEMORYCOLOR [37] and COLORTERMS [38] datasets. RELATIVESIZE contains 486 object pairs from 41 physical objects. The model is required to predict the size relation between two objects in a binary question-answering format with "Yes"/"No" answers. MEMORYCOLOR and COLORTERMS require the model to predict the color of objects from a set of 11 color labels in a multiple-choice format.
Table 8 presents the zero-shot performance of KOSMOS-1 and LLM on visual commonsense reasoning tasks. KOSMOS-1 significantly outperforms LLM by  $1.5\%$  on RELATIVESIZE,  $14.7\%$  on MEMORYCOLOR, and  $9.7\%$  on COLORTERMS dataset. The consistent improvements indicate that KOSMOS-1 benefits from the visual knowledge to complete the corresponding visual commonsense reasoning. The reason for KOSMOS-1's superior performance is that it has modality transferability, which enables the model to transfer visual knowledge to language tasks. On the contrary, LLM has to rely on textual knowledge and clues to answer visual commonsense questions, which limits its ability to reason about object properties.
Model	Size Reasoning RELATIVESIZE	Color Reasoning MEMORYCOLOR	COLORTERMS
Using retrieved images
VALM [49]	85.0	58.6	52.7
Language-only zero-shot evaluation
LLM	92.7	61.4	63.4
KOSMOS-1	94.2	76.1	73.1
Table 8: Zero-shot visual commonsense reasoning on RELATIVESIZE, MEMORYCOLOR, and COLORTERMS datasets. Accuracy scores are reported.
In recent years, vision-language learning and representation models has garnered significant attention [2, 3, 4, 34, 50, 51, 52, 53, 54]. Previous vision-language models still exhibit limitations in instruction following, in-context abilities, and generalization capabilities for unseen tasks. Researchers have begun exploring more powerful multimodal large language models. Flamingo [5] trained its model from scratch and made it possible to generate text tokens conditioned on both visual and text inputs. Another category of research focuses on learning multimodality abilities based on LLMs [7, 55, 56]. Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.
In this work, we introduce KOSMOS-1, a multimodal large language model that can perceive general modalities, follow instructions, and perform in-context learning. The models trained on web-scale multimodal corpora achieve promising results across a wide range of language tasks and multimodal tasks. We show that going from LLMs to MLLMs enables new capabilities and opportunities. In the future, we would like to scale up KOSMOS-1 in terms of model size [13, 14, 60], and integrate the speech [12] capability into KOSMOS-1. In addition, KOSMOS-1 can be used as a unified interface for multimodal learning, e.g., enabling using instructions and examples to control text-to-image generation. We further discuss the limitations and broader societal impacts of KOSMOS-1 in the supplemental material.
[58] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
[59] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructclip: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.06500, 2023.
[60] Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts. In Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=mWaYC6CZf5.
[61] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Retrieval-augmented multimodal language modeling. ArXiv, abs/2211.12561, 2022.
[62] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823, 2023.