Language Is Not All You Need: Aligning Perception with Language Models
Authors: Shaohan Huang
Li Dong
Wenhui Wang
Yaru Hao
Saksham Singhal
Shuming Ma
Tengchao Lv
Lei Cui
Owais Khan Mohammed
Barun Patra
Qiang Liu
Kriti Aggarwal
Zewen Chi
Johan Bjorck
Vishrav Chaudhary
Subhojit Som
Xia Song
Furu Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodal large language model
TL;DR: KOSMOS-1 is a multimodal large language model that is capable of perceiving multimodal input, following instructions, and performing in-context learning for not only language tasks but also multimodal tasks

Abstract: A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce KOSMOS-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train KOSMOS-1 from scratch on web-scale multi-modal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that KOSMOS-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.

Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang! Li Dong. Wenhui Wang, Yaru Hao? Saksham Singhal," Shuming Ma" Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Weit Microsoft
Abstract
A big convergence language, multimodal perception, action, and world model- ing is key step toward artificial general intelligence. In this work, introduce KosMos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train KOSMOs-1 from scratch on web-scale multi- modal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-though prompting, on wide range of tasks without any gradient updates or finetuning. Experimental results show that KOSMOS- I achieves impressive performance (i) language understanding. generation, and even OCR-free NI.P (directly fed with document images), (ii) perception-lange languugge tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMS can benefit from cross cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal language. In addition, we introduce dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.
Introduction: From LLMs to MILMs
Large language models (LLMs) have successfully served as general-purpose interface across various natural language tasks [1]. The LLM-based interface can be adapted to a task as long as we are able to transform the input and output into texts. For example, the input of the summarization task is document and the output its summary. So we can feed the input document into the language model and then produce the generated summary.
Despite the successful applications in natural language processing, it is still struggling natively LLMs for multimodal data, such as image, and audio. Being a basic part of intelligence, multimodal perception is necessity achieve artificial general intelligence, in terms of knowledge acquisition and grounding to the real world. More importantly, unlocking multimodal input [2, 3, 4, 5, 6, 7] greatly widens the applications language models to more high-value areas, such as multimodal machine learning, document intelligence, and robotics.
In this work, introduce KosMos-I, 1, Multimodal Large Language Model (MLLM) that can perceive general modalities, follow instructions (i.e., zero-shot learning), and learn in context (i.e., few-shot learning). The goal is align perception with LLMs, that the models are able to see and talk. To be specific, we follow METALM [3] to train the KOsMOS- model from scratch. As shown in Figure 1, a Transformer-baat based language model is regarded as the general-purpose interface, and perception modules are docked with the language model. We train the model on web-scale multimodal
Equal contribution. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

output
Multimodal Large Language Model (MLLM)
Kosmos-1 can perceive both language and
learn in context, reason, and generate
E Embedding
Recess
Figure 1: KOSMOS-1 is multimodal large language model (MLLM) that is capable of perceiving multimodal input, following instructions, and performing in-context learning for not only language tasks but also multimodal tasks. In this work, align vision with large language models (LLMss), advancing the trend of going from LLMs to MLLMs.
corpora, i.e., text data, arbitrarily interleaved images and texts, and image-caption pairs. In addition, we calibrate the instruction- instructionfoolloiing capability across modalities by transferring language-only data.
The KOsMos-1 I model natively supports language, perception language, and vision tasks. In addition to various natural language tasks, KosMos-1 models natively handle wide range of perception- intensive tasks, spanning visual dialogue, visual explanation, visual question answering, image captioning. simple math equation, OCR, and zero-shot image classification with descriptions, We also build an IQ test benchmark following Raven's Progressive Matrices [8, 9], which evaluates the capability nonverbal reasoning for MLLMs. The examples show that the native support of multimodal perception enables new opportunities apply LLMs new tasks. Moreover, we show that MLLMs achieve better commonsense reasoning performance compared with LLMs, which indicates cross-moda transfer helps knowledge acquisition.
The key takeaways are as follows:
Properly handling perception is a necessary step toward artificial general intelligence. The capability of perceiving multimodal input is critical LLMs. First, multimodal perception enables LLMs to acquire commonsense knowledge beyond text descriptions. Second, aligning perception with LLMs opens the door to new tasks, such as robotics, and document intelli- gence. Third, the capability of perception unifies various APIs, as graphical user interfaces are the most natural and unified way to interact with. For example, MLLMs can directly read the screen or extract numbers from receipts. We train the KOSMOS-1 models on web-scale multimodal corpora, which ensures that the model robustly learns from diverse sources. We not only use large- scale text corpus but also mine high-quality image-caption pairs and arbitrarily interleaved image and text documents from the web.
From LLMs to MILMs.
Language models as penerol-purpose interfaces. Following the philosophy proposed in METALM [3], we regard language models as universal task layer. Because of the open-ended output space, we are able to unify various task predictions texts. Moreover, natural-languse instructions and action sequences (such as programming language) can be well handled by language models. LLMS also serve as basic reasoners [10], which is complementary to perception modules on complex tasks. So is natural to align world, action, and multimodal perception with the interface, i.c., language models.
New capabilities MILMs. Apart from the capabilities found previous LLMs [1, 11], MLLMS enable new usages and possibilities. First, we can conduct zero- and few-shot multimodal learning by using natural language instructions and demonstration examples. Second, we observe promising signals of nonverbal reasoning by evaluating the Raven IQ test, which measures the fluid reasoning

---

ability of humans. Third, MLLMs naturally support multi-turn interactions for general modalities, such as multimodal dialogue.
2 Kosmos-1: A Multimodal Large Language Model
KosMos-1 is multimodal language model that can perceive general modalities, follow instructions, learn in context, and generate outputs. Given the previous context, the model learns to generate texts an auto-regressive manner. Specifically, the backbone of KosMos- 1 is a Transformer-basee causal language model. Apart from text, other modalities are embedded and fed into the language model. The Transformer decoder serves as a general-purpos interface to multimodal input. We train KOSMOS-I on multimodal corpora, including monomodal data, cross-modal paired data, and interleaved multimodal data. Once the models are trained, we can directly evaluate the models in zero-shot and few-shot settings on both language tasks and multimodal tasks.
2.1 Input Representation
The Transformer decoder perceives general modalities in a unified way. For input format, we flatten input as a sequence decorated with special tokens. Specifically, we use and 
 to denote start- and -equuenc The special tokens <image> and < </image> indicate the beginning and end of encoded image embeddings. For example, "
 document 
" is text input, and "
 paragraph <image> Image Embedding </ </image> paragraph 
" is interleaved image-text input.
An embedding module used to encode both text tokens and other input modalities into vectors. Then the embeddings are fed into the decoder. For text tokens, we use lookup table to map them into embeddings. For the modalities of continuous signals (e.g., image, and audio), it is also feasible to represent inputs as discrete code and then regard them as "foreign languages" [4, 12]. In this work, following [3], we employ a vision encoder as the embedding module for input images. In addition, Resampler [5] is used as an attentive pooling mechanism to reduce the number of image embeddings.
2.2 Multimodal Large Language Models (MLLMs)
After obtaining the embeddings of an input sequence, we feed them into the Transformer- based decoder. The left-to-right causal model processes the sequence in auto-regressive manner, which produces the next token by conditioning on past timesteps. The causal masking is used to mask out future information. A softmax classifier upon Transformer used generate tokens over the vocabulary.
MILMMs serve as general-purpose interfaces [3] that can perform interactions with both natural language and multimodal input. The framework is flexible to handle various data types, as long as we can represent input as vectors. MLLMs combine the best of two worlds. First, the language models naturally inherit the capabilities of in-context learning and instruction following. Second, perception is aligned with language models by training multimodal corpora.
The implementation based on the library TorchScale [13], which is designed for large- model training. Compared with the standard Transformer architecture, we include the following modifications: We use MAGNETO [14], a Transformer variant, as the backbone architecture and xPos [15] relative position encoding for better long-contess modeling.
2.3 Multimodal Training Data
The models are trained on web-scale multimodal corpora. The training datasets consist of text corpora, image-caption pairs, and interleaved data of images and texts.
Text Corpora We train our model with The Pile [16] and Common Crawl (CC). The Pile is massive English text dataset built for training large-scale language models. We exclude data splits from GitHub, arXiv, Stack Exchange, and PubMed Central. We also include the Common Crawl snapshots (2020-50 and 2021-04) datasets, CC-Stories, and RealNews datasets [17, 18).
Image-Caption Pairs The image-caption pairs are constructed from several datasets, including English LAION-2B [19], LAION-400M [20], COYO-700M [21], and Conceptual Captions [22, 23].
3

---

English LAION-2B. 4010N-400M, and COY0-700M are collected from web pages of the Common Crawl web data by extracting image sources and the corresponding alt-text. Conceptual Captions are also from internet web pages.
Interleaved Image- Text Data We collect interleaved multimodal data from the Common Crawl snapshot, which is publicly available archive of web pages. We use a filtering process to select about 71 millions web pages from the original 2 billions web pages in the snapshot. We then extract the text and images from the HTML of each selected web page.
2.4 Training Objective
The KOsMos-1 I training is conducted web-scale multimodal corpora, including monomodal data (e.g., text corpus), cross-modal paired data (c.g., image-caption pairs), and interleaved multimodal data (e.g., documents of arbitrarily interleaved images and texts). To be specific, data for representation learning. For example, language modeling with text data pretrains instruction following, in-context learning, and various language tasks. Moreover, cross-modal pairs and inter- leaved data learn to align the perception of general modalities with language models. Interleaved data also naturally fit in the multimodal language modeling task. We present more details of training data collection in the supplemental material.
monomodal
The models are trained with the next-token prediction task, i.e., learning to generate the next token depending on the previous context. The training objective is to maximize the log- likelihood of tokens in examples. Notice that only discrete tokens, such as text tokens, loss. Multimodal language modeling is scalable way to train the models. More importantly, the emergence of various capabilities makes the training task favorable for downstream applications.
accounted for in the training
Experiments
3.1 Training Setup
We train KOSMos-1 1 with 1.6 billion parameters using mix of text corpora, image-caption pairs, and interleaved data. We use Magneto's 's initialization for optimization stability and pretrained CLIP VIT-L/14 model for image representation. The model is trained for 300k steps using batch size of 1.2 million tokens and the Adam W optimizer. We adopt a learning rate warm-up and decay schedule, and use SentencePiece for tokenization.
To improve cnntruction-folloiinn capabilities, we perform language-only instruction tuning using Unnatural Instructions [24] and FLANv2 [25] datasets. This tuning process is conducted as language modeling, and improvements transfer across modalities. More details about hyperparameters can be found in the supplemental material.
Table summarizes the corresponding datasets and what capabilities we would like to evaluate. We evaluate different capabilities related to language, perception-language and vision.
3.2 Perception- Languuge Tasks
Image Captioning Table 2a shows the captioning performance on COCO [39] Karpathy test split and Flickr30k [40] test set. KOSMOS-1 achieves remarkable results in zero-shot setting on two image captioning datasets. Specifically, our model achieves a CIDEr score of 67.1 on the Flickr30k dataset, compared to 60.6 and 61.5 for the Flamingo-3B and Flamingo-9B models, respectively. Notably, our model is able to accomplish this feat with smaller size of 1.6B, compared to Flamingo models. This demonstrates our model's superiority in zero-shot image captioning.
Visual Question Answering Table 2b reports the visual question answering results on VQAv2 [41] and VizWiz [42]. We show that KOsMOs-1 can better handle the diversity and complexity of the VizWiz dataset. KOsMOS-1 achieves higher accuracy and robustness than Flamingo-31 and Flamingo-98 models on zero-shot settings. addition, our model competitive with Flamingo on the VQAv2 dataset.

---

Dataset
Task description
Metric
Zero-shot
Few-shot
Language tasks
StoryCloze [26]
Commonsense reasoning
Accuracy
HellaSwag [27]
Commonsense NLI
Accuracy
Winograd [28]
Word ambiguity
Accuracy
Winogrande [29]
Word ambiguity
Accuracy
PIQA [30]
Physical commonsense
Accuracy
BoolQ [31]
Question answering
Accuracy
CB [32]
Textual entailment
Accuracy
COPA [33]
Causal reasoning
Accuracy
Rendered SST-2 [34]
OCR-free sentiment classification
Accuracy
HatefulMemes [35]
OCR-free meme classification
ROC AUC
Cross-modal transfer
RelativeSize [36]
Commonsense reasoning (object size) Commonsense reasoning (ohject color)
Accuracy
MemoryColor [37]
Accuracy
ColorTerms [38]
Commonsense reasoning (object color)
Accuracy
Nomverbal reasoning tasks
IQ Test
Raven's s Progressive Matrices
Accuracy
Perception-uue -aaggugg lasks
COCO Caption [39]
Image captioning
CIDEr, etc.
Flicker30k [40]
Image captioning
CIDEr, etc.
VQAv2 [41]
Visual question answering
VQA acc.
VizWiz [42]
Visual question answering
VQA acc.
WebSRC [43]
Web page question answering
Fl score
Vision tasks
CUB [44]
Zero-shot image classification with descriptions
Accuracy
Table I: We evaluate the capabilities of KosMoS-1 on language, perception perception-language, language, and vision
tasks under both zero- and few-shot learning settings.
3.3 IQ Test: Nonverbal Reasoning
Raven's Progressive Matrices [9, 8] is one of the most common tests to evaluate nonverbal reasoning. The capability of nonverbal reasoning typically a reflection of an individual's intelligence quotient (IQ). Figure 2 shows example. Given eight images, the task is to identify the following element from six similar candidates.
Here
The following
correct?
conect?
inconett?
cooeect
Vin
Yeu
Nei Prompt))
Method
Accuracy
Random Choice
17%
8
KosMOS-1
22%
Figure 2: We append each candidate image to the prompt Table 3: Zero-shot generalization on separately and query the model if it is correct.
Raven IQ test.
The models need to conduct zero-shot nonverbal reasoning without explicitly fine-tuning. The Raven IQ test is analogous to in-context learning of language models, where the difference is whether the context is nonverbal or verbal. In order to infer the answers, the models have to recognize abstract concepts and identify the underlying patterns of given images. So the IQ task is good testbed to benchmark the nonverbal in-context learning capability.
Table 3 shows the evaluation results on the IQ test dataset. KOSMOS- achieves 5.3% improvement respectively over random baseline. The results indicate that KOSMOS- able to perceive abstract conceptual patterns in nonverbal context, and then deduce the following element across multiple choices. To the best of our knowledge, it is the first time that a model can perform such zero-shot Raven IQ tests. Although there is still large performance gap between the current model and the

---

Shot
Model
COCO
Flickr30k
Shot
Model
VQA+2
VizWiz
ZeroCap [45]
14.6
Frozen
29.5
VLKD [46]
58.3
VLKDViT-B/16
38.6
FewVLM [47]
METALM
41.1
METALM [3]
43.4
0
Flamingo-3B*
49.2
28.9
Flamingo-3B [5]
60.6
Flamingo
51.8
28.8
Flamingo-9B" [5]
79.4
61.5
KosMos- 1 (1.6B)
51.0
29.2
KosMos-1 (1.6B)
67.1
Flamingo-33 [5]
Flamingo-33 [5]
2
Flamingo-98* [5]
Flamingo-9B5
KOsMoS- I (1.6B)
51.4
31.4
KosMos- (I.6B)
99.6
70.0
Flamingo-31" [5]
85.0
72.0
Flamingo-3B [5]
53.2
34.4
Flamingo-9B" [5]
93.1
72.6
4
Flamingo 9B" [5]
56.3
34,9
KosMos- (1.6B)
101.7
75.3
KosMOs-1 I (1.6B)
51.8
35.3
Flamingo-3B [5]
90.6
71.7
Flamingo-3B [5]
55.4
38.4
Flamingo-9B" [5]
99.0
73.4
8
Flamingo-9B [5]
58.0
39.4
KosMos-1 (1.6B)
96.7
68.0
KosMos-1 (1.6B)
51.4
39.0
(a) Image captioning results on COCO caption Karpa- thy test and Flickr30k test. We present CIDEr scores.
(b) Visual question answering results on VQAv2 and VizWiz. We present VQA accuracy scores.
Table 2: "*": Flamingo [5] builds the zero-shot prompt with examples from the downstream tasks where their corresponding images are removed (i.e., similar to few-shot text prompts) while the others evaluate true zero-shot learning.
average level of adults, KosMos-1 1 demonstrates the potential of MLLMs to perform zero-shot nonverbal reasoning by aligning perception with language models.
3.4 OCR-Free Language Understanding
OCR-free language understanding is task that focuses understanding text and images without relying on Optical Character Recognition (OCR). For example, during the Rendered SST-2 task [34], sentences from the Stanford Sentiment Treebank [48] dataset are rendered images. The model is asked to predict the sentiment of the text within the images. The task evaluates a model's ability to read and comprehend the meaning of words and sentences directly from the images.
As shown in Table 4a, KosMos-1 achieves ROC AUC of 63.9% for the HatefulMemes validation set and a test accuracy of 67.1% for the Rendered SST-2 test set. It outperforms CLIP VIT-L and Flamingo-9B, which achieve AUCs of 63.3% and 57.0% on the HatefulMemes task. Note that Flamingo explicitly provides OCR text into the prompt, while KOSMoS- does not access any external tools or resources. This indicates that KOSMOS-1 has built-in abilities to read and comprehend the text in the rendered images.
Model
HatefulMemes
Rendered SST-2
Model
EM
F1
CLIP ViT-B/32
57.6
59.6
Using extracted text
CLIP ViT-B/16
61.7
59.8
LLM
7.6
17.9
CLIP ViT-L/14
63.3
64.0
KOSMOS-1
15.8
31.3
Flamingo-38
53.7
Flamingo-9B
57.0
Without using extracted text
K (1.6B)
63,9
67.1
KOsMOS-1
10.6
(a) Zero-shot generalization on OCR-free language understand- ing. We report accuracy scores.
(b) Zero-shot performance on WebSRC task. We report exact match (EM) and Fl scores.
3.5 Web Page Question Answering
Web page question answering aims finding answers to questions from web pages. It requires the model to comprehend both the semantics and the structure of texts (such tables, lists, and

---

Hiput Prompt
Inst Pronpt
Questonnwhet name the weodpecker or downy? Answer:
wnodperken black white throughout the bodl and yellow crewn. Description dowry woudnecken has spois on wings and some red crown.
Descrittion
Duestonoweat is name woodpacker in picture? Anrser
Completion
Completion
downy woodoecker
X
free lce woodpecker
her Classficatonn
Classiitooo Descriptions
Figure 3: In-context verbal descriptions can help KOSMOs- recognize visual categories better.
HTML layout). We compare the performance the Web-based Structural Reading Comprehension (WebSRC) dataset [43]. For comparisons, we train a language model (LL.) on the same text corpora with the same training setup as in KosMos-1.
The experimental results are summarized Table 4b, We observe that KOSMOS- I outperforms the LLM, indicating that KosMos-1 1 can benefit from the layout and style information of web pages in images. In addition, we evaluate the performance of KosMos-1 without the extracted text in the prompt. It shows that extracted text has contribution of + 12.0/20..7 EM/F1 KosMos-1, indicating that the benefit from modeling images does not sacrifice its language abilities.
3.6 Multimodal Chain-of- -hhouhtf Prompting
Thought
Chain-of-thoughh prompting [10] allows large language models to generate a series of reasoning steps and decompose multi-step problem into intermediate steps, which can significantly improve the performance in complex tasks. Motivated by chain-of-thouuht prompting, we investigate multimodal chain-of-thought prompting using KOsMos-1. We break down perception-languaat tasks into two steps. In the first stage, given image, we use a prompt guide the model to generate a rationale. The model then fed the rationale and task-aware prompt to produce the final results.
We conduct experiments to evaluate the performance of the multimodal chain-or oor---of-th-e of-thhught prompting. Table 5a shows that multimodal chain-of- ththughffthought thought prompting achieves score of 72.9, which 5.8 points higher than the standard prompting. By generating intermediate content, the model can recognize the text the images and infer the sentiment of the sentences more correctly.
Model
Accuracy
CLIP VIT-B/32
59.6
Settings
Accuracy
CLIP ViT-B/16
59,8
Without Descriptions
61.7
CLIP VIT-L/14
64.0
With Descriptions
90.0
KOSMOS-1
67.1
w/ multimodal CoT prompting
72.9
(b) Results of zero-shot image classification without and with verbal descriptions.
(a) Multimodal chain-of-thought (CoT) prompting on Rendered SST-2 task.
3.7 Zero-Shot Image Classification with Descriptions
The standard approach of image classification as above is to prompt the model for the specific name of the object depicted in the image. However, there are also some classification rules customized for different users and scenarios, such the refined classification of complex animal subspecies. We can utilize natural language descriptions to guide KOsMOs-1 to distinguish images the zero-shot setting, which makes the decision process more interpretable. Following CUB [44], construct bird classification dataset that contains images and natural- -anural-language descriptions categories. The evaluation procedure is illustrated in Figure 3.
The evaluation results are shown Table 5b. We observe that providing descriptions context can significantly improve the accuracy of image classification. The consistent improvements indicate that KOsMOs-1 can perceive the intentions of instructions and well align the concepts language modality with visual features in vision modality.
7

---

3.8
Language Tasks
The models are evaluated on the language tasks given task instructions (i.e., zero-shot) or several demonstration examples (i.e., few-shot). Text inputs are directly fed into the models as in vanilla language models. We train language model (LLM) baseline with same text corpora and training setup. We evaluate KOSMOS-1 and the LLM baseline eight language tasks.
Table 6 presents the in-context learning performance of language tasks. KOSMOs-1 achieves comparable or even better performance in cloze completion and commonsense reasoning tasks when compared LLM. terms of the average result across all these datasets, LLM performs better in zero-shot and one-shot settings, whereas our model performs better in few-shot (k 4) settings. In addition, Section 3.9.2 shows that MLLMs learn better visual commonsense knowledge compared with LLMs.
Zero-shot
One-shot
Few-shot (A: = 4)
Task
LLM
KosMos-1
LLM
KosMos-1
LLM
KosMOs- 1
StoryCloze
72.9
72.1
72.9
72.2
73.1
72.3
HellaSwag
50.4
50.0
50.2
50.0
50.4
50.3
Winograd
71.6
69.8
71.2
68.4
70.9
69.8
Winogrande
56.7
54.8
56.7
54.5
57.0
55.7
PIQA
73.2
72.9
73.0
72.5
72.6
72.3
BoolQ
56.4
56.4
55.1
57.2
58.7
59.2
CB
39.3
44.6
41.1
48.2
42.9
53.6
COPA
68.0
63.0
69.0
64.0
69.0
64.0
Average
61.1
60.5
61.2
60.9
61.8
62.2
Table 6: Performance comparisons of language tasks between KOSMOS- and LLM. We the
same textual data and training setup to reimplement a language model. Both models do not use instruction tuning for fair comparisons.
3.9 Cross-modal Transfer
Cross-modal transferability allows a model to learn from one modality (such as text, image, audio, etc.) and transfer the knowledge to the other modalities. This skill can enable model to perform various tasks across different modalities. In this part, we evaluate the cross-model transferability of KOSMOS-1 on several benchmarks.
3.9.1 Transfer from Language to Multimodal: Language-Oni Instruction Tuning
To evaluate the effect of language-only instruction tuning, we conduct an ablation study using four datasets: COCO, Flickr30k, VQAv2, and VizWiz. These datasets consist of image captioning and visual questions anwsering. The evaluation metrics are: CIDEr scores for COCO/Flickr30k and VQA accuracy for VQAv2/VizWiz.
Table shows the experimental results. Language-only instruction tuning boosts our model's performance by 1.9 points Flickr30k, 4.3 points on VQAv2, and 1.3 points VizWiz. Our experi- ments show that language-only instruction tuning can significantly improve the model's instruction- following capabilities across modalities. The results also indicate that our model can transfer the instruction-following capability from language to other modalities.
Model
COCO
Flickr30k
VQAv2
VizWiz
KosMOS-1
84.7
67.1
51.0
29.2
w/o language-only instruction tuning
87.6
65.2
46.7
27.9
Table 7: Ablation study on language-only instruction tuning. We report CIDEr scores for COCO and Flickr30k, and VQA accuracy scores for VQAv2 and VizWiz.

---

3.9.2 Transfer from Multimodal to Language: Visual Commonsense Reasoning
Visual commonsense reasoning tasks require an understanding of the properties of everyday objects in the real world, such color, size, and shape. These tasks are challenging for language mod- els because they may require more information about object properties than what is available in texts. To investigate the visual commonsense capabilities, compare the zero-shot performance of KosMos-1 and LLM on three object commonsense reasoning datasets, RELATIVESIZE [36], MEMORYCOLOW [37] and COLORTERMS [38] datasets. RELATIVESIZE contains 486 object pairs from 41 physical objects. The model is required to predict the size relation between two objects in binary question-answering format with "Yes"/"No" answers. MEMORYCOLOR and COLORTERMS require the model to predict the color of objects from a set of 11 color labels in multiple-choice format.
Table 8 presents the zero-shot performance of KOSMOS-1 I and LLM on visual commonsense rea- soning tasks. KOsMos-1 significantly outperforms LLM by 1.5% RELATIVESIZE, 14.7% MEMORYCOL.OR and 9.7% on COLORTTERM dataset. The consistent improvements indicate that KosMos-1 benefits from the visual knowledge complete the corresponding visual commonsense reasoning. The reason for KosMos-I I 's superior performance is that it has modality transferability. which enables the model transfer visual knowledge language tasks. On the contrary, LLM has to rely on textual knowledge and clues to answer visual commonsense questions, which limits its ability reason about object properties.
Size Reasoning RELATIVESIZE
Color Reasoning
Model
MEMORYCOLOR COLORTERMS
Using retrieved images
VALM [49]
85.0
58.6
52.7
Language-only zero-shot evaluation
LLM
92.7
61.4
63.4
KOSMOs-1
94.2
76.1
73.1
Table 8: Zero-shot visual commonsense reasoning RELATIVESIIE, MEMORYCOLOOR, and COLORTERMS datasets. Accuracy scores are reported.
4 Related Work
In recent years, laiguaee learning and representation models has garnered significant at- tention [2, 3, 4, 34, 50, 51, 52, 53, 54]. Previous vision- models still exhibit limitations in instruction following, in-context abilities, and generalization capabilities for unseen tasks. Re- searchers have begun exploring more powerful multimodal large language models. Flamingo [5] trained its model from scratch and made it possible to generate text tokens conditioned on both visual and text inputs. Another category of research focus learning multimodality abilities based on LLMs [7, 55, 56]. Meanwhile, some work [57, 58, 59] introduce visual instruction tuning to enhance instruction following capabilities.
5 Conclusion
In this work, we introduce KOsMOS- 1, a multimodal large language model that can perceive general modalities, follow instructions, and perform in-context learning. The models trained on web-scale multimodal corpora achieve promising results across a wide range of language tasks and multimodal tasks. We show that going from LLMs MILMs enables new capabilities and opportunities. In the future, we would like to scale up KOSMOS-I 1 in terms of model size [13, 14, 60]. and integrate the speech [12] capability into KOSMOS-1 1. addition, KOSMOS-1 I can be used as unified interface for multimodal learning, e.g., enabling using instructions and examples to control text-to-image generation. We further discuss the limitations and broader societal impacts of KOSMOS-1 1 in the supplemental material.

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[2] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In Neural Information Processing Systems, 2021.
[3] Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma. and Furu Wei. Language models are general-purpose interfaces. ArXiv, abs/2200.00633,. 2022.
[4] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: BEiT pretraining for all vision and vision-language tasks. ArXiv, abs/2208. 10442, 2022.
[5] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022. URI JRRLhtttp::/opperrrrve wwtpeeewwwwwweeeeeeeeeeeeeeeeeee www.:eettoouuee ddd.dddimbbbbb............
[6] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the Internet. ArXiv, abs/2201.07520, 2022.
[7] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12297, 2023.
[8] John and Jean Raven. Raven Progressive Matrices, pages 223-237. Springer US, Boston, MA, 2003. ISBN 978-1--461--013--4 doi: 10. 10.1007/978-144615-005 11, URL https:
/ //doi. org. 170000999888-11111111-------11-1
[9] Patricia A Carpenter, Marcel A Just, and Peter Shell. What one intelligence test measures: a theoretical account of the processing in the raven progressive matrices test. Psychological review, 97(3):404, 1990.
[10] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. / 1903, 2022.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek B Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski. Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eek, Jeff Dean, Slav Petrov, and Noah Fiedel. Pa Scaling language modeling with pathways. ArXiv, abs/2204.02311 2022.
10

---

[12] Chengyi Wang, Sanyuan Chen, Yu Wu, Zi-Hua Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. ArXiv, abs/23011022 102111,211,,1,111111111111111111 2023.
[13] Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang, Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song, and Furu Wei. TorchScale: Transformers at scale. CoRR, abs/2211.1311 2022.
[14] Hongyu Wang, Shuming Ma. Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj. Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, and Furu Wei. Foundation transformers. CoRR, abs/2210.06423. 2022.
[15] Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrrapllatbbe transformer. arXiv preprint WwXi::2212.10554, 2022.
[16] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
[17] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint (90V::1999000000 2019.
[18] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using DeepSpeed and Megatron train Megatron- NLG 530B, large-scale generative language model, 2022.
[19] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint urXiv:2210.084/1 2022.
[20] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
[21] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.
[22] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Papers, pages 2556-2565. Association for Computational Linguistics, 2018
[23] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1355-3568,, 2021.
[24] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor, 2022. URL https://arxiv.ott 09689.
[25) Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 200.22301.1688 2023.
[26] Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-lever Semantics, pages 46-51, 2017.
11

---

[27] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[28] Hector Levesque, Emest Davis, and Leora Morgenstemn. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012.
[29] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732-8740, 2020.
[30] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
[31] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924 2936, Minneapolis, Minnesota, June 2019. Association for Computational URL https: // /aaacalaattttt aclanthology..... org wrg/N19/ 1100
Linguisties. doi: 10.18653/v1/N19-11
[32] Catre-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung. 23(2):107-12,, Jul. 2019.
[33] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium, 2011.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[35] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems, volume 33, pages 2611-2624, 2020.
[36] Hessam Bagherinezhad, Hannaneh Hajishirzi, Yejin Choi, and Ali Farhadi. Are elephants bigger than butterflies? reasoning about sizes of objects. ArXiv, abs/1602.00053, 2016.
[37] Tobias Norlund, Lovisa Hagström, and Richard Johansson. Transferring knowledge from vision to language: How to achieve it and how to measure it? ArXiv, abs/2 100/21099113211. 2021.
[38] Elia Bruni, Gemma Boleda, Marco Baroni, and Nam Khanh Tran. Distributional semantics in technicolor. In ACL, 2012.
[39] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[40] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. TACL, 2:67-78, 2014.
[41] Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6325-6334 2017.
[42] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 3608-3617, 2018
12

---

[43] Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: dataset for web-based structural reading comprehension. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173-4185, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.186531 httpw/12221emmmmmm.. mem.lp-main.343...... URL
org/
2021 ewnlp-main....... 343.
[44] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 2irds-200-2011 dataset. 2011.
[45] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-ter generation for semantic arithmetic. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17897-17907, 2021.
[46] Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang. Qun Liu, and Pascale Fung. Enabling multimodal generation on CLIP via vision-language knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2383-221 2095, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 http://...222....ss-/oeoee wwt.iiinns-acl....... URL https: / anthooologg org 2022. findings--iiii acl.a111 187.
[47] Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt worth millions of parameters: Low-resource prompt-based learning for vision-language models. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 2763-2775 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1/2022.acl-oongg....e 197. URL https://acllntthll..g .ereggg...rr//ee org/2022...... acl- long.1997
[48] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631 1631-1642, 1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: /www. aclweb. org/ http///hhhoogy//-111111
[49] Weizhi Wang. Li Dong, Hao Cheng, Haoyu Song, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Representations, 2023.
language modeling. In International Conference on Learning
[50] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022.
[51] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
[52] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointl-ssalled multilingual language-image model. (itii::2000069940 2022.
[53] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sehastian Goodman, Xiao Wang. Yi Tay, al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint arXiv:2305.18566 2023.
[54] OpenAl. Gpt-4 technical report, 2023.
[55] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model, 2023.
[56] Danny Driess, Fei Xia, Mchdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.033 033778 2023.
[57] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint 28X44:2304.08485, 2023.
13

---

[58] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing languale understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[59] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXhv:2305.0650 06500, 2023.
[60] Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts. In Advances in Neural Information Processing Systems, 2022. URL fttttp:://operevvieer net/ www.mmmddmmaaYCCCC..
[61] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen Yih. Retrieval -uggeeeted multimodal language modeling. ArXiv, abs/2211. 2022.
[62] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXiv:2301.13823 2023.
14


Reviews:
[{"review_id": "8yelv1cyd6", "reviewer": "Reviewer_tesr", "summary": "This paper proposed KOSMOS-1, a Multimodal Large Language Model (MLLM) that can take image embeddings as additional input to auto-regressive LLM. Trained on both text-only and image-text web-crawled data, KOSMOS-1 can get strong performance across a variety of tasks including language understanding, perception-language tasks, and vision tasks, demonstrating the utility of cross-modal knowledge transfer. ", "questions": "See weaknesses.", "limitations": "Discussed in Supp.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Although there are no specific novelties in terms of loss and module design, the introduced model is simple, unified, versatile, and effective.\n2. The experiments cover both language-focused, vision-language, and vision-focused evaluations and demonstrate the strong performance of the proposed models.\n3. The study about whether one modality can benefit the other (cross-modal transfer) is interesting and showcase some insightful observations.", "weaknesses": "1. Comparison with recent MLLM works either in related work or experiments was missing. For example, MiniGPT4, LLaVA, etc. \n2. Although in ablation about cross-modal transfer, authors show that Language-Only Instruction Tuning can help several visual-language tasks. However, that experiment is only about language instruction tuning data. I wonder if the general Text Corpora (not just language instruction tuning data) in pre-training help general vision-language downstream tasks? \n3. Can Visual Instruction Tuning data (LLAVA's or MiniGPT4's) help NLP tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments.\n\n**Question 1**\n> Comparison with recent MLLM works either in related work or experiments was missing. For example, MiniGPT4, LLaVA, etc.\n\nWe will add the comparison with recent MLLM works. The table below presents the comparison with MiniGPT4 and LLaVA on zero-shot image captioning (COCO and Flickr30k) and visual question answering (VQAv2). Since MiniGPT4 and LLaVA do not report results on these benchmarks, we test their released model on them ourselves. We use the MiniGPT4 version 'Vicuna-7B' and the LLaVA version 'LLaVA-Lightning-MPT-7B-preview'.\n\n| Model          | Model Size  | COCO (CIDEr) | Flickr30k (CIDEr) | VQAv2 (VQA-acc) |  \n|----------        |-------  |------- |--------------|-------|  \n| KOSMOS-1 | 1.6B | 84.7  | 67.1            | 51.0  |  \n| MiniGPT4   | 7.3B | 86.3  | 54.4            | 28.9  |  \n| LLaVA          | 7.3B | 74.3  | 45.9           | 34.9  |   \n\nExperimental results show that KOSMOS-1 outperforms both MiniGPT4 and LLaVA in terms of Flickr30k and VQAv2 tasks. Our model also achieves competitive performance on COCO captioning.\n\n**Question 2**\n> Although in ablation about cross-modal transfer, authors show that Language-Only Instruction Tuning can help several visual-language tasks. However, that experiment is only about language instruction tuning data. I wonder if the general Text Corpora (not just language instruction tuning data) in pre-training help general vision-language downstream tasks?\n\nWe conducted an ablation study on the general text corpora in a smaller setting, where the text decoder contains 300 million parameters, and the image encoder is initialized from CLIP ViT-B/16. The training step is 100k. \nThe results are presented in the accompanying table. The general text corpora improve the performance on visual question answering (VQAv2) tasks, while resulting in a decline in the model's performance on image captioning (COCO, Flickr30k). The downstream COCO and Flickr30k image captioning data are similar to the image-caption pairs used in training (LAION-2B and COYO-700M). Adding general text corpora into training prevents the model from converging towards captioning tasks and results in a slight drop. But training on general text corpora improves the model to follow instructions. Learning from text question answering data improves the model to achieve a better performance on VQAv2. \n\n| Dataset                                                          | COCO (CIDEr) | Flickr30k (CIDEr) | VQAv2 (VQA-acc) |\n|------------------------------------------------------------------|------|-----------|-------|\n| Text Corpora + Image-Capton Pairs  + Interleaved Image-Text Data | 74.9 | 52.5      | 33.4  |\n| Image-Capton Pairs  + Interleaved Image-Text Data                | 77.8 | 54.1      | 28.1  |\n\n**Question 3**\n> Can Visual Instruction Tuning data (LLAVA's or MiniGPT4's) help NLP tasks?\n\nWe perform instruction tuning with LLaVA’s visual instruction tuning data and evaluate the model on NLP tasks. As shown in the table, performing visual instruction tuning improves the zero-shot performance on HellaSwag, Winograd, Winogrande, PIQA and COPA. But we also observe a decline on StoryCloze and a large drop on BoolQ. Overall, introducing visual instruction tuning data does not significantly help NLP tasks, and we will explore more in the future.\nIn addition, we evaluate the model on vision-language tasks and find that adding visual instruction data significantly improves the zero-shot performance on Flickr30k and VQAv2.\n\n| Model             | StoryCloze | HellaSwag | Winograd | Winogrande | PIQA | BoolQ | COPA |\n|-------------------|------------|-----------|----------|------------|------|-------|------|\n| KOSMOS-1          | 72.1       | 50.0      | 69.8     | 54.8       | 72.9 | 56.4  | 63.0 |\n| + Visual instruction tunning | 71.6       | 50.6      | 70.5     | 55.7       | 73.0 | 50.8  | 69.0 |\n\n"}, {"review_id": "zqwZ4DLhnV", "reviewer": "Reviewer_oWBZ", "summary": "This paper presents a vision/language model trained on text and interleaved image/text data. It uses a ViT to encode the image into tokens and a transformer predict output tokens from the previous ones. It is trained on web-scale data and then fine-tuned on NLP instruction tuning data.", "questions": "Will the model or the interleaved data corpus be released?\n\nIs there any evidence the model \"forgets\" any of its multi-modal knowledge during instruction fine-tuning, which only has NLP data? Is this a concern?\n\nDid the authors consider initializing the model somehow with a pre-trained LLM? With so much work following that approach recently it would be interesting to hear why everything was trained from scratch.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Despite a lot of interest in instruction following for V/L models, training them from scratch is a somewhat less well explored area so I think contributions to this direction are useful. \n- Many evaluations that help understand how effective the model is and what it has learned in different settings\n- Showing NLP instruction tuning benefits multi-modal tasks is very interesting, although I would have liked see the ablation consider tasks with more complex instruction then just captioning/VQA kind of tasks. I also found the the chain-of-thought and visual common sense results interesting.\n", "weaknesses": "- The models appears to be limited in its ability to take advantage of few-shot data, which I think should be one of the main theoretical advantage of supporting interleaved V/L data and pre-training on such data. Captioning seems to be the only task where we really see a clear benefit.\n- The instruction following shown in the paper is a bit simple, just for tasks like captioning, QA or classification which are tasks that are pretty close to the pre-training tasks. While it is interesting that this data helps the model. it has not really been shown the model can follow instruction for very different tasks in the same way LLM can.\n- Related work is poorly discussed. There is no related work section, and as far I can see there is not much discussion about similar works such as other V/L foundation models or models like BLIP that adapt a pre-trained LLM in other parts of the paper or appendix either.\n\nOverall I feel like the model and many experiments have scientific value despite the relatively smaller scale compared to other recent models, but I also feel like the lack related work is a non-trivial issue and I am not really sure how to balance those two points. I have recommend accepting the paper for now since I still think the paper would a benefit for the conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions on our paper.\n\n**About related work**\n> Related work is poorly discussed. There is no related work section, and as far I can see there is not much discussion about similar works such as other V/L foundation models or models like BLIP that adapt a pre-trained LLM in other parts of the paper or appendix either.\n\nWe will incorporate a discussion of related work in the final version.\n\n**Question 1**\n> Will the model or the interleaved data corpus be released?\n\nWe will indeed release our model for the benefit of the research community. \nWithout violating copyright restrictions and compliance, we will release the interleaved data corpus.\n\n**Question 2**\n> Is there any evidence the model “forgets” any of its multi-modal knowledge during instruction fine-tuning, which only has NLP data? Is this a concern?\n\nAn ablation study was conducted to assess the impact of language-only instruction tuning on the model's performance (Table 7). The results indicate that language-only instruction tuning leads to improved performance on VL tasks.\n\n**Question 3**\n>Did the authors consider initializing the model somehow with a pre-trained LLM? With so much work following that approach recently it would be interesting to hear why everything was trained from scratch.\n\nHere are some key factors to this decision:\n1) **Cross-modality transfer**: One of our goals is to achieve cross-modality transfer and enable language models to learn multimodal commonsense. By training from scratch, the model can be designed specifically to handle such transfers.\n2) **Generalization through large-scale training**: Large-scale training tends to result in better generalization, as the model is exposed to more diverse data. Starting from scratch allows for the incorporation of more diverse data during the training process.\n3) **Native multimodal LLM**: The model is a native multimodal LLM, meaning it has been designed from the ground up with multimodal learning in mind. \n4) **Results drop in small size model**: We conducted a comprehensive comparison between training from scratch and initializing with a pre-trained LLM (text decoder contains 110 million parameters, and the image encoder is initialized from CLIP ViT-B/16. The training step is 300k). For a fair comparison, we trained these two settings with the same number of training steps. The results, as shown in the table below, indicate a decline in performance when initializing with a small-sized pre-trained LLM.\n\n | Model           | COCO (CIDEr) | Flickr30k (CIDEr) | VQAv2 (VQA-acc) |  \n|----------        |------- |--------------|-------|  \n| From scratch     | 75.2  | 59.1            | 37.0  |  \n| Cont. training   | 67.6  | 51.3            | 35.1  |  \n\nFurther research by PaLM-E [1] corroborated these findings (Figure 6), revealing that concatenating two pre-trained models could lead to a substantial performance degradation on NLP tasks, when the model size is small. To avoid this issue, we chose to train the model from scratch.  \n\n[1] Driess, Danny, et al. \"PaLM-E: An embodied multimodal language model.\" arXiv preprint arXiv:2303.03378 (2023).\n\n\n"}, {"review_id": "ASuD0EF6aR", "reviewer": "Reviewer_XnP9", "summary": "This work introduces KOSMOS-1, a multimodal large language model trained on large-scale text corpora, image-caption pairs, and interleaved image-text data. KOSMOS-1 can perform classic captioning/VQA tasks in a zero-shot or few-shot in-context prompting fashion. It can also perform OCR from visual document, and answer questions based on a webpage. It consists of a pre-trained CLIP-L/14 model for image representation and MAGNETO for language decoding. KOSMOS achieves competitive results on a wide suite of vision-language tasks with 1.6B parameters compared against Flamingo-9B. Furthermore, it introduces a Raven’s IQ benchmark to evaluate nonverbal intelligence.", "questions": "- How can one improve KOSMOS-1 for better performance on Raven-IQ test?\n- Why is the zero-shot VQAv2 performance of KOSMOS-1 worse than BLIP-2?\n- Could authors report KOSMOS-1 performance on commonsense reasoning tasks while using multimodal inputs?\n- What is the intuition behind language-only instruction-tuning? Why could this benefit vision-language tasks?\n- Will the code and model be released to the public?\n\nThe writing can be further improved. For example:\nL24: “it is still struggling to natively use LLMs for multimodal data, such as image, and audio.” -> “it still struggles to natively use LLMs for multimodal data, such as image and audio.”", "limitations": "Yes, the authors discuss limitation in appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents comprehensive technical details about the proposed system, including pre-trained models, loss design, and datasets. It also shows impressive qualitative sample usage of the proposed system on a wide range of tasks in supplemental, including multi-turn multimodal dialog, image classification with description, few-shot multimodal in-context learning, multimodal chain-of-thought prompting, and reasoning with webpage screenshots. ", "weaknesses": "I do not find the proposed system particularly novel, because its architecture design, training losses, and pre-training datasets are widely adopted in prior art such as Flamingo [1] and BLIP-2 [2]. Furthermore, BLIP-2 [2] achieves stronger zero-shot performance on VQAv2 (Table 2b) 65% compared to KOSMOS-1's 51.0%, even though it is trained on much fewer data (LAION114M) with much fewer tunable parameters.\n\nThe small-scale Raven IQ test with 50 samples is undoubtedly challenging and interesting, however, it is hard to say KOSMOS-1 is better than random chance because it is only marginally better by 5.3%. Could authors discuss what are some promising directions to improve KOSMOS-1 on this benchmark?\n\nLanguage-only instruction-following paradigm shows worse performance on COCO but leads to better results on Flickr30K, VQAv2, and VizWiz (Table 7). Could the author explain why language-only instruction-tuning might lead to better or worse results on different vision-language tasks? \n\nTo demonstrate that the KOSMOS-1 has better commonsense reasoning capabilities, the authors show language-only zero-shot evaluation results on RelativeSize, MemoryColor, and ColorTerms even though KOSMOS-1 is a multimodal model. Is there a reason not to report the multimodal zero-shot performance?\n\nThe authors do not promise a model release.\n\n[1] Flamingo: a Visual Language Model for Few-Shot Learning. 2022.\n\n[2] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions on our paper. \n\n**Improving Raven IQ test**\n\n> The small-scale Raven IQ test with 50 samples is undoubtedly challenging and interesting, however, it is hard to say KOSMOS-1 is better than random chance because it is only marginally better by 5.3%. Could authors discuss what are some promising directions to improve KOSMOS-1 on this benchmark?\n\nThe performance gain looks marginal because the baseline is relatively low at 16.7%. The relative gain of KOSMOS-1 over the random chance is 31.8%, which looks more substantial. \nThese are a few promising directions to explore for improving KOSMOS-1 on Raven IQ test:\n1) **Increasing model capacity**: Scaling up the model size can potentially improve performance. This allows the model to learn more complex representations and better capture the relationships between different elements in the Raven IQ test task.\n2) **Training data**: Enhancing the training data by incorporating more diverse and complex Raven-style problems can lead to better generalization.\n3) **More fine-grained image representations**: Raven IQ test often requires a more fine-grained representation of images, which allows the model to understand the global relationships between multiple input images. \n\n**About Language-only Instruction-tuning**\n> Language-only instruction-following paradigm shows worse performance on COCO but leads to better results on Flickr30K, VQAv2, and VizWiz (Table 7). Could the author explain why language-only instruction-tuning might lead to better or worse results on different vision-language tasks?\n\nFor image captioning tasks, we conduct some statistics on the COCO and Flickr30k datasets, and we find that the Flickr30k dataset has a higher average number of caption tokens. The model tends to generate longer outputs after instruction tuning, which might lead to some improvement in the Flickr30k dataset, but some decline in the COCO dataset.\n\nVQAv2 and VizWiz are question-answering tasks, where the model needs to understand both the question and the image content to provide an accurate answer. Language-only instruction-tuning tends to obtain better results on these tasks as the text-based instructions can help the model better follow instructions and improve its ability to answer questions.\n\n**Question 1**\n> Why is the zero-shot VQAv2 performance of KOSMOS-1 worse than BLIP-2? \n\nThe zero-shot VQAv2 performance of KOSMOS-1 is not worse than BLIP-2. \nAs shown in Table 2 of the BLIP-2 paper, the VQAv2 (test-dev) performance of the KOSMOS-1 model (1.3B text decoder + 300M image decoder) is 51.0 in the zero-shot setting, while the performance of BLIP-2 (2.7B text decoder + 300M image decoder) is 49.7. Given that the text decoder of KOSMOS-1 is smaller than that of BLIP-2, this demonstrates the superior performance of KOSMOS-1 in this context.\nBLIP-2 achieves a VQAv2 score of 65.0 in the zero-shot setting when using the CLIP ViT-g (1.2B) image encoder and FlanT5XXL (11B) text decoder. We think that the significant improvement in performance can be attributed to the substantially larger model size. \n\nAdditionally, the training dataset may have played a role in the performance disparity. BLIP-2 included the COCO dataset in their training, which is the source of images for VQAv2. Consequently, the model can learn in-domain knowledge from COCO dataset, further enhancing its performance on VQAv2 tasks.\n\n**Question 2**\n> Could authors report KOSMOS-1 performance on commonsense reasoning tasks while using multimodal inputs?\nTo demonstrate that the KOSMOS-1 has better commonsense reasoning capabilities, the authors show language-only zero-shot evaluation results on RelativeSize, MemoryColor, and ColorTerms even though KOSMOS-1 is a multimodal model. Is there a reason not to report the multimodal zero-shot performance?\n\nWe have conducted experiments on commonsense reasoning tasks while using multimodal inputs. As the original dataset does not provide images of objects, we obtained relevant images using Google Image Search. These images were then prepended to the text descriptions and fed into the model. The results of these experiments are presented in the following table:\n\n| Model                                                 | RelativeSize | MemoryColor | ColorTerms |\n|------------------------------------------|--------------|-------------|------------|\n| KOSMOS-1                                          | 94.2         | 76.1        | 73.1       |\n| KOSMOS-1 + Google searched image | 80.7         | 82.6        | 86.5       |\n\nThe table reveals that KOSMOS-1's performance on object color tasks (MemoryColor and ColorTerms) is improved when using multimodal inputs. \nWe found that Google searched images are relatively noisy, which can affect the predictions. As a result, the performance of the RelativeSize task may decline.  In this case, it is better to use text for prediction directly, which also demonstrates the importance of our model's cross-modality.\n\n**Question 3**\n> What is the intuition behind language-only instruction-tuning? Why could this benefit vision-language tasks?\n\nThe motivation of language-only instruction-tuning is to explore cross-modal transfer in KOSMOS-1 whether a model uses information learned in one modality (e.g., language) to enhance its performance in another modality (e.g., vision). As shown in Table 7, language-only instruction-tuning helps the model follow instructions. It helps the model to better understand the questions and generate answers in an appropriate format.\n\n**Question 4**\n> Will the code and model be released to the public?\n\nYes, we will indeed release our model and code to the public.\n\n\n\n\n\n"}, {"review_id": "JpJTEOBAll", "reviewer": "Reviewer_BRko", "summary": "This paper propose a pretrained multimodal large language model, which can achieve impressive zero-shot performance on many downstream tasks. \n\nThe experimental results verify the transfer knowledge from language to multimodal or from multimodal to language. \n\nHowever, compared to the very similar model FROMAGe and BLIP-2, this model structure seems to be of little difference. The most advantage of this model is that it uses more multimodal data. \n\nIt seems that the performance on visual question answering is less than BLIP-2.", "questions": "See the weaknesses part.", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation is clear and the writing can be easy to be followed. \n\nThe multimodal COT reasoning ability is analyzed which is interesting. \n\nThe pretrained multimodal large language model achieve impressive zero-shot performance on many downstream tasks.", "weaknesses": "This paper presents a lot of experiments to demonstrate how good their model is, but it does not compare GPT-4 or BLIP-2 to show how much room for improvement their model has. Directly adopting a good image captioning model to convert images into diverse text and feeding them into a large language model (gpt-3.5 or llama) will also achieve good results (please see the multimodal instruction-following data construction way shown in the paper ``visual instruction tuning’’ ). The technique novelty advantage of this model compared to the common approach should be elaborated.\n\nThis paper claims that they proposed a multi-modal large language model that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot), but the experimental baseline always lacks some key models that are relatively strong in the field, such as the experiment results (Table 6) shown in Section 3.8: language tasks. They only compared the language model trained on their own corpus. In fact, the multimodal version of KOSMOS-1 sees more text data and image description, leading to the comparison a little unfair. On the one hand, the comparison is not sufficient because there is no comparison with other large language models. What are the overall advantages and core contribution of KOSMOS-1 compared to LLMs (such as LLaMA, BLOOM, OPT) and previous multimodal models (BLIP-2, Flamingo)?\n\nIn addition, Sec 3.6 Multimodal Chain-of-Thought Prompting analyzed the multimodal COT reasoning ability, but it is only evaluated on the classification task SST-2. There is no demonstration of corresponding examples and no evaluation on complex reasoning tasks (e.g., ScienceQA), so it is difficult to judge whether this KOSMOS-1 really has the multimodal COT reasoning ability. Because prior work has shown that the ability of the chain of thought may emerge when the number of parameters of large language model is greater than 10B. The conclusion drawn by this paper is hard to convince. \n\nFinally, Sec 2.4:Training Objective does not specify in detail how the visual representation model and language model are trained. If you train separately, how do you perform multimodal alignment and fusion? From the description, what is the difference between this whole training process and Oscar's training method? If following your description to train KOSMOS-1, will the model have the multi-modal COT reasoning ability?  \n\nIn sec.2.1, should the input textual token embedding and visual token embedding be pre-aligned and then feed to the decoder? Or is it entirely dependent on the learning ability of the following decoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions on our paper. \n\n**About compared to LLM**\n> They only compared the language model trained on their own corpus. In fact, the multimodal version of KOSMOS-1 sees more text data and image description, leading to the comparison a little unfair. On the one hand, the comparison is not sufficient because there is no comparison with other large language models.\n\nFor fair comparisons, we train the language model baseline with the same text corpora and training setup. They have the same number of training tokens.\n\nOther LLMs use different architectures, data, hyperparameters, steps, etc., resulting in too many incomparable factors. Therefore, we specifically align the settings for a fair comparison, allowing for more scientific comparisons.\n\n**About core contribution compared to LLMs and previous multimodal models**\n> What are the overall advantages and core contribution of KOSMOS-1 compared to LLMs (such as LLaMA, BLOOM, OPT) and previous multimodal models (BLIP-2, Flamingo)?\n\nCompared to LLMs: \n1) We are a Multimodal Large Language Model (MLLM), supporting multimodal input and cross-modality transfer. \n2) Our model brings more possibilities and application scenarios, such as embodied AI. \n3) Moreover, our training data is different from LLMs, which use pure text corpora. We employ datasets like text corpora, image-caption pairs, and interleaved image-text data.\n\nCompared to previous multimodal models: \n1) We train our model from scratch, enabling it to learn visual commonsense through cross-modality transfer. \n2) We can perform in-context learning, while BLIP-2 lacks the ability to perform in-context learning (as stated in their Limitation section)[1].\n3) Although our model size is relatively small, the results are impressive, indicating great potential.\n4) In terms of the methodology, our model implementation is remarkably simple, following a minimalist approach. \n5) In terms of evaluation, we explore OCR-free language understanding, text instruction tuning, Raven IQ tests, customized image classifiers, and cross-modality transfer, which have not been analyzed in previous works.\n\n[1] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" arXiv preprint arXiv:2301.12597 (2023).\n\n**No evaluation on complex reasoning tasks (e.g., ScienceQA)**\n> There is no demonstration of corresponding examples and no evaluation on complex reasoning tasks (e.g., ScienceQA), so it is difficult to judge whether this KOSMOS-1 really has the multimodal COT reasoning ability. Because prior work has shown that the ability of the chain of thought may emerge when the number of parameters of large language model is greater than 10B. \n\nWe evaluate KOSMOS-1 on ScienceQA benchmark under zero-shot and few-shot settings with and without explanations. The table below presents the results on ScienceQA test image set. We find that chain-of-thought prompting via introducing explanations performs slightly better than standard prompting (Few-shot (k=1) in the Table), which demonstrates the chain-of-thought ability of KOSMOS-1 on complex reasoning tasks. We believe that chain-of-thought prompting can bring greater improvements as the model size increases, as the text part improves.\n\n| Setting   | ScienceQA |\n|-----------|----------------|\n| Zero-shot                      | 56.1    |\n| Few-shot (k=1)                 | 56.8   |\n| Few-shot (k=1) w/ explanations | 57.2  |\n\n**About training objective**\n> Finally, Sec 2.4:Training Objective does not specify in detail how the visual representation model and language model are trained. If you train separately, how do you perform multimodal alignment and fusion? From the description, what is the difference between this whole training process and Oscar’s training method? \n\nWe train the language decoder and visual encoder together via the next-token prediction task on text corpora, image-text pairs and interleaved image-text data. \n\nGiven the input containing images and texts, we obtain image embeddings and text embeddings via the visual encoder and lookup table of word embeddings. Then we feed these image and text embeddings into the Transformer-based decoder. The self-attention module in the decoder can fuse the images and texts. \nFor the training on image-text pairs and interleaved data, the model is trained to generate captions/texts based on images, which helps the model to learn its alignment.\n\nCompared to Oscar:\n1) The main component of KOSMOS-1 is a Transformer-based decoder, which is unidirectional, processing the input sequence only from left to right. While Oscar is a bidirectional Transformer-based encoder. \n2) KOSMOS-1 shows zero-shot and in-context learning capabilities on vision-language tasks, Oscar is mainly used for fine-tuning on specific vision-language tasks. \n3) KOSMOS-1 is trained using next-token prediction task, while Oscar is trained using masked token predication since it is a bidirectional model.\n\n**About CoT reasoning ability**\n> If following your description to train KOSMOS-1, will the model have the multi-modal COT reasoning ability?\n\nCoT (or step-by-step generation) ability can be inherited from the capabilities of text language models, and we will also conduct more experiments on a larger scale.\n\n**About pre-align textual token embedding and visual token embedding**\n> In sec.2.1, should the input textual token embedding and visual token embedding be pre-aligned and then feed to the decoder? Or is it entirely dependent on the learning ability of the following decoder.\n\nIt is not necessary to pre-align the input textual token embeddings and visual token embeddings before feeding them to the decoder. \n\nWe feed the visual and textual embeddings into the decoder, and train the model to generate the next token depending on the previous context (images and texts). The training helps the model to align image and text representations.\n\n"}]

Author rebuttal:

Meta-review: All reviewers were in favor of this paper pre-rebuttal, however there were serious concerns on: (i) the lack of novelty and insights (XnP9), (iii) the lack of a related works section (oWBZ), and (iii) missing comparisons to other LLMs and MLLMs (tesr, BRKo). That said, the reviewers liked the simplicity of the model and the notion of training the MLLM from scratch while demonstrating reasonable performance improvements against chosen baselines even with smaller model sizes. The authors presented a strong rebuttal with new ablation results and comparisons to other models. Given the authors have promised to publicly release their model, the reviewers have unanimously agreed to accept the paper. 

The AC thinks that the empirical results in the paper are interesting and the public release of the pre-trained model will be valuable to the MLLM community; thus has decided to accept this paper. However, the AC recommends the authors to revise the camera-ready for: (i) better presentation of the paper, especially the Introduction, (ii) including a related works section, and (iii) improving the technical quality, e.g., via including relevant details of prior works that the paper builds on, such as MetaLM, Megneto, etc.
Decision: Accept (poster)