Grounding Multimodal Large Language Models to the World
Authors: Zhiliang Peng
Wenhui Wang
Li Dong
Yaru Hao
Shaohan Huang
Shuming Ma
Qixiang Ye
Furu Wei
Venue: ICLR 2024 poster (2024)
Keywords: large language model
multimodal large language model
grounding
referring
Primary area: generative models

Abstract: We introduce Kosmos-2, a Multimodal Large Language Model (MLLM), enabling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent text spans (i.e., referring expressions and noun phrases) as links in Markdown, i.e., [text span](bounding boxes), where object descriptions are sequences of location tokens. To train the model, we construct a large-scale dataset about grounded image-text pairs (GrIT) together with multimodal corpora. In addition to the existing capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning), Kosmos-2 integrates the grounding capability to downstream applications, while maintaining the conventional capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning). Kosmos-2 is evaluated on a wide range of tasks, including (i) multimodal grounding, such as referring expression comprehension and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This study sheds a light on the big convergence of language, multimodal perception, and world modeling, which is a key step toward artificial general intelligence. Code can be found in [https://aka.ms/kosmos-2](https://aka.ms/kosmos-2).

Published as a conference paper at ICLR 2024
GROUNDING MULTIMODAL LARGE LANGUAGE MODELS TO THE WORLD
Zhiliang Pengl* Wenhui Wang2* Li Dong
Yaru Hao Shaohan Huang* Shuming Ma² Universsity of Chinese Academy of Sciences
Qixiang Ye
Furu Weift
Microsoft Research https//aka.mm/Gennrall
ABSTRACT
We introduce KOSMOS-2, a Multimodal Large Language Model (MLLM), en- abling new capabilities of perceiving object descriptions (e.g., bounding boxes) and grounding text to the visual world. Specifically, we represent text spans (i.e., referring expressions and noun phrases) as links in Markdown, i.e., "[text span] (bounding boxes)" ", where object descriptions are sequences of loca- tion tokens. To train the model, we construct a large-scale dataset about grounded image-text pairs (GRIT) together with multimodal corpora. KOsMOS-2 integrates the grounding capability to downstream applications, while maintaining the con- ventional capabilities of MLLMs (e.g., perceiving general modalities, following instructions, and performing in-context learning). KosMos-2 is evaluated on a wide range of tasks, including (i) multimodal grounding, such as referring ex- pression comprehension and phrase grounding, (ii) multimodal referring, such as referring expression generation, (iii) perception-language tasks, and (iv) language understanding and generation. This study sheds a light on the big convergence language, multimodal perception, and world modeling, which is key step toward artificial general intelligence. Code can be found in https://aka. https://aka.ms/kssmtt ms/kosmos-2.
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Hao al., 2022: Alayrac al., 2022; Huang et al., 2023; Driess et al., 2023; OpenAI, 2023) have been a general-purpose purpose interface across language, vision, and vision-language tasks. They are able to perceive general modalities, including texts, images, and audio, and generate responses using free-form texts under zero-shot and few-shot settings.
In this study, we unlock the referring and grounding capabilities of multimodal large language models, with the aim to construct a more flexible and general human human-computer computer interface about vision- vision-languagge tasks, Figure 1. With such model, users can directly point to objects or image regions without requiring detailed text descriptions referring to them. It also enables the model to respond with visual answers (i.e., bounding boxes), supporting more vision-language tasks such as referring expression comprehension while resolving their co-reference ambiguity.
Our proposed model, referred to as KOsMOs-2, is a Transformer-bast causal language model built upon KosMos-1 (Huang et al., 2023), but has the major differences of grounding and referring capabilities. To unlock the grounding capability, we first construct a web-scale dataset of grounded image-text pairs (GRIT), which are built upon subset of image-text pairs from LAION-2B (Schuh- mann et al., 2022) and COYO-700M (Byeon et al., 2022). GRIT is combined with the multimodal corpora (i.e., text corpora, image-text pairs and interleaved image-text data) train the model. To construct GRIT. we propose an approach to extract and link text spans (i.e., noun phrases and referring expressions) in image captions to spatial coordinates (e.g., bounding boxes) of the corresponding objects or image regions. Spatial coordinates of each object are converted to sequence of location tokens, which are appended atop text span of the object as an expansion. The expanded text span serves as a "hyperlink" ([text span] (locaticn inn tokens)) ) to connect the objects or regions of the image to the caption, as shown in Figure 1. Given the "hyperlink" data and trained in the
Equal contribution. f Corresponding authors.

---

Published as a conference paper at ICLR 2024
Grounding
[a campfire](sloc,² <|oc1007>)
Kosmos-2:
Multimodal Large Language Model
[It](<loc,,> <loCg63>) sits next to
Referring
Figure KosMos-2 multimodal large language model which has new capabilities of multimodal grounding and referring. KosMos-2 2 can understand multimodal input, follow instructions, perceive object descriptions (e.g., bounding boxes), and ground language to the visual world.
causal language modeling fashion, KoSMOS-2 links text spans in the generated free-form text re- sponse to image regions, which facilities generating more accurate, informational, and comprehensive vision-language predictions. Utilizing a pronoun the text span in conjunction with location tokens, KosMos-2 can perceive the referring object and incorporate into the reasoning process. This simple transform enables the ability of referring, providing more flexible interaction interface.
Experimental results show that KOsMOs-2 achieves not only competitive performance on language and rision-language tasks, but also leading performance grounding tasks (phrase grounding and referring expression comprehension) and referring tasks (referring expression generation). The grounding capability born with KosMos-2 enables it be applied to more downstream tasks, such as grounded image captioning, and grounded visual question answering.
GRIT:WEB-SCAN GROUNDED IMAGE-TEXT PAIRS
To learn the grounding capability, we first construct a large-scale dataset of Grounded Image-Texi pairs (GRIT), based image-text pairs from subset of COY0-700M (Byeon et al., 2022) and LAION-2B (Schuhmann et al., 2022). To this end, pipeline is designed to extract and link text spans (i.e., noun phrases and referring expressions) regions, Figure 2. The pipeline consists of two steps: generating noun-chunk-ae bhbunk-bounding-aa box pairs and producing
the caption to their corresponding image
pairs, which are detailed in what follows.
Generating noun-chunk-bee Hoatunk-ooundig-beee pairs Given an image-text pair, we first extract noun chunks from the caption and associate them with image regions through pretrained object detector. In specific, spaCy (Honnibal et al., 2020) is employed to parse the caption ("a dog in field of flowers") and extract all noun chunks ("a dog". "a field" and "flowers"). To reduce potential noise, we eliminate certain abstract noun phrases that are challenging to recognize in the image, such as "time", "love", and "freedom"". Subsequently, the image and extracted noun chunks are fed to pretrained grounding model (e.g., GLIP (Li et al., 2022b)) to obtain the associated bounding boxes. The non-man con-maximu suppression algorithm is applied to remove bounding boxes that have a high overlap with others, regardless of whether they are associated with the same chunk not. The noun-chume waan--hunk-bouudiin bounding box pairs with predicted confidence scores higher than 0.65 are kept. If no bounding boxes retained, discard the corresponding image-caption pair.
Step-1:
2

---

Published as a conference paper at ICLR 2024
Sentence dependency relations
a dog field of flowers
Identify noun chunks
Expand noun chunks
[. in field of flowers field of flowers flowers
dog a field flowers
Keep dog field flowers' field flowers"
substrings
Drop "flweer"
Detection & Post post-pracess
Compose
dog in
6290,37,,05,7500
flewers: 2990,77.605.7501
a field: [0.264.919,221]
Step-1: Creating churk bounding box pairs
Step-2: Producing refeniing expression bounding box pairs
Figure 2: The pipeline to construct web-scale grounded image-text pairs.
pairs endow model with the ability to ground diverse and complex linguistic descriptions, we expand noun chunks to referring expressions. In specific, spaCy is employed again to obtain dependency relations the caption. A noun chunk then expanded referring expression by recursively traversing its children in the dependency tree and concatenating children tokens with the noun chunk. We do not expand noun chunks with conjuncts. The illustrated in Figure 2, the noun chunk 'a dog' is expanded to "a dog in a field of flowers", the noun chunk 'a field' is expanded "a field of flowers", and the noun chunk 'flowers' remains unchanged as it cannot be expanded.
Step-2: Producing wwarrinngexppeessoo-boodddiiiiii
chunks without children tokens retained for the subsequent process. As
Furthermore, only retain text spans that we keep the referring expression "a dog in a field of flowers" and drop "a field of flowers" and 'flowers' (as they are contained by "a dog in a field of flowers"). The bounding box of the noun chunk ('a dog') is assigned to the corresponding generated referring expression ("a dog in a field of flowers").
contained by others. As demonstrated in Figure
So far, we have more than 91M images, 115M spans, and 137M associated bounding boxes. In comparison with publicly available visual grounding datasets, as shown in Table 8 in Appendix C.I. GRIT significantly improves the data seale. More data samples of GRIT are shown in Appendix C.3.
KosMos-2 2: GROUNDED MULTIMODAL LARGE LANGUAGE MODEL
KOsMOS-2 is grounded MILM, which follows the model architecture and training objective of KosMos- 11 but integrates grounding and referring capabilities. For example, KosMos-2 model can accept image regions drawn by users, provide visual answers (i.e., bounding boxes), and ground the output the visual world. To endow the model with grounding and referring capabilities, we add grounded image-text pairs to the training data. For text span (e.g., noun phrase and referring expression) and its corresponding bounding boxes in grounded image-text pair, we discretize continuous coordinates of bounding boxes to sequence of location tokens which are uniformly encoded alongside text tokens. We then link the location tokens and their corresponding text span via "hyperlink" data format. Based grounded image-text pairs, KosMOs-2 2 model is trained to establish a mapping between image regions and their corresponding location tokens and connect the image regions with their associated text spans.
3.1 GROUNDED INPUT REPRESENTATION
Given text span and its associated bounding boxes in grounded image-text pair, we first convert the continuous coordinates of bounding boxes to sequence of discrete location tokens (Chen et al., 2021). For image with width W and height H. respectively divide both the width and height into P segments. A total of P P bins are calculated, with each bin comprising (W/P) x (H/P) pixels. For each bin, we location token represent the coordinates within that bin. When
3

---

Published as a conference paper at ICLR 2024
mapping location tokens back to the continuous coordinates in the image, utilize the center pixel coordinates each bin.
Each bounding box be represented using its top-left point (zu. Yu) and bottom-right point (Ebe. ybr), which are discretized to top-left and bottom-right location tokens, respectively. A top-left used with special boundary tokens location token (<locı>> and bottom-right one (<1 <box><oo.g><<> <ooooc))>>>><<<<</>_>>>>>>>>>>>> ></bo>>. (<box> and < </box>) to form the bounding box representation: In instances where the text span is linked to multiple bounding boxes, special token, <deling mellim, is employed to concatenate the location tokens of these bounding boxes. The resulting representation can be expressed as: <box> <ooxt>>>oof//>>>> <<lccj <ddlim>.....mmm.m....
Accordingly, we arrange the text span and its associated location tokens in format resembling "hyperlink" in Markdown. For the text span with single bounding box, the resultant sequence is "<p> text span </p><<bo>><1 1oop:=<ooooooooooo>>>>>
where <p> and </p> are special tokens indicating the beginning and end of the text span. Such format conveys the model that image regions within <locm><<ocm> <locr>>>>>> are associated with 'text span'.
Take Figure as an exmaple, the input representation is:

<image>Imin Image httppiiww......c.ooooooooooooooa </box> sits <p>a oot.mpiieeggooooooxooooooooooooa
where 
 and 
 indicate the start- and end-of-sequence, and <image> and </ immaee>>> represent the begimning and end of encoded image embeddings. <grounding> is special token employed to signal the model that the subsequent sequence includes text spans and their associated location tokens. We map input text tokens and location tokens to embeddings via lookup table. A vision encoder, accompanied by a re-sampler module, is utilized to acquire image embeddings.
For language-only data, cross-modal paired data (i.e., image-text pairs), and interleaved multi- modal data, we use the same input representations as of KosMos-1. Therefore, in these cases, the agru token is not required to be prepended.
3.2 GROUNDED MULTIMODAL LARGE LANGUAGE MODEL.
KOSMOS-2 2 uses a Transformer- based causal language model as the foundation architecture and is trained through the autoregressive language modeling task. In addition multimodal corpora used in KosMos-1 1 (including text corpora, image-caption pairs, and interleaved image-text data), we add GRIT into training. The training loss only considers discrete tokens, such as text and location tokens. The model learns to locate and understand image regions through the location tokens, associate text spans to image regions, and output bounding boxes the image region using location tokens.
KOSMOS-2 enhances MLLMs (Huang et al., 2023) by incorporating grounding and referring capabil- ities. Specifically, we can use "<grounding>... <p>tet text span</p>" as imput to prompt KOsMos-2 to generate location tokens for the 'text span' in multimodal grounding tasks. We can also employ a pronoun as a text span in conjunction with location tokens, such as wwwroonniigg...... enable KosMos-2 to perceive the referring objects
< </p>bbx>><lcc>>>>> (ooo>>>>oooooo/ooooo////// ></bo>>", or regions, providing flexible human-compute interaction fashion. Furthermore, we can simply prepend the '<grounding>" token in conventional vision-language tasks (like image captioning) to facilitate new applications, resulting in more accurate, informative, and comprehensive responses.
3.3 MODEL TRAINING
Setup KosMOS-2 is trained upon GRIT, text corpora, image-caption pairs, and interleaved image- text data. The training procedure involves a batch size of 419K tokens, consisting of 185K tokens from text corpora, 215K tokens from original and grounded image-caption pairs, and 19K tokens from interleaved data. The model is trained for 60K steps, utilizing approximately 25 billion tokens, using an Adam optimizer with 3 = (0.9, 0.98). a weight decay of 0.01, and dropout rate of 0.1. The learning rate increases to 2e-4 during the first 375 warm-up steps and linearly decays zero. The model is trained on 256 V100 GPUs for 24 hours. To tell the model when to ground text output to the visual world, we prepend the '<groundi ng>" token the grounded caption during training.
4

---

Published as a conference paper at ICLR 2024
<box-- <locyes" <100ges" </box>
<baic <loces"
<locuzs"
</box>>
Grounded MLLM
Grounded MLLM
A man blue hard safety vest </p>
cpo A man in a blue
<p> orange
hard orange
safety vest <ip>
(1) Phrase grounding
(2) Referring expression comprehension
Figure 3: Input format of evaluations on phrase grounding and referring expression comprehension.
The vision encoder has 24 layers with 1,024 hidden size and 4,096 FFN intermediate size. The multimodal large language model component is 24-layer MAGNETO Transformer (Wang et al., 2022a; Ma et al., 2022) with 2,048 hidden dimensions, 32 attention heads, and 8, 192 FFN intermediate size. The total number of trainable parameters amounts to approximately 1.6B. The image resolution set to x24 and the patch size is 14x 14. To discretize the continuous coordinates, we divide the width and height of the image into 32 equally sized bins, with each bin encompassing area of 7x7 pixels. A total of 32x32 location tokens are added to the vocabulary. KOSMOS-2 uses the weights of KOSMOs- for initialization, the newly added x2 word embeddings of location tokens are initialized randomly. We update all the parameters during training and instruction tuning.
Instruction Tuning After model training, instruct tuning is used to better align KosMos-2 with human instructions. We combine rision-language instruction dataset (i.e. --I-trructt (Liu et al., 2023a)) and language-only instruction datasets (i.e.. Unnatural Instructions (Honovich et al., 2022) and FLANv2 (Longpre al., 2023)) with the training data to tune the model. In addition, we construct grounded instruction data by utilizing the pairs of bounding boxes and text spans in GRIT. Given an expression-bee wowriiioooounnnin--bb pair, we use "<p> expression < </->" as the input instruction, and prompt the model to generate the corresponding location tokens of the bounding boxes. We also use the prompt like "<p> It </p><box><<>>/// ocu>< oct,>< / /box> is" to ask the model to generate expressions according to its bounding boxes. More templates are included in Appendix C.2.
EVALUATION
KosMOS-2 is initially assessed on multimodal grounding (Sec. 4.1) and multimodal referring (Sec. 4.2) tasks evaluate its capabilities, while also being tested perception-languaee
(Sec. 4.3) and language tasks (Sec. 4.4) to examine conventional MLLM abilities. As mentioned in Sec. 3.2, the grounding capability allows new applications to emerge for KosMos-2. More details can be found in Appendix A.
4.1 MULTIMODAL GROUNDING
To evaluate the ability of multimodal grounding, we use generative fashion to test KOSMOS-2 on grounding tasks including phrase grounding and referring expression comprehension, The former requires the model to predict set of bounding boxes based on or more given phrases that maybe interrelated within a single caption. The latter encourages the model to locate the object described in text referring expression within given image.
For both phrase grounding and referring expression comprehension tasks, KosMos-2 generates location tokens which then converted to bounding boxes for evaluation. The input format is where "<groundings" is used
"
<<i 
<immae>>>> Image Embedding < to prompt the model to generate locations tokens in its response.
5

---

Published as a conference paper at ICLR 2024
Val Split
Test Split
Model
Zero-shot
R@D
R@5
@ 10
R @
R@5
R @ 10
VisualBert (Li et al., 2019)
x
70.4
84.5
86.3
71.3
85.0
86.5
MDETR (Kamath et al., 2021)
X
83.6
93.4
95.1
84.3
93.9
95.8
GLIP (Li et al., 2022b)
X
86.7
96.4
97.9
87.1
96.9
98.1
FIBER (Dou et al., 2022)
X
87.1
96.1
97.4
87.4
96.4
97.6
GRILL (Jin et al., 2023)
J
18.9
53.4
70.3
KosMOS-2
77.8
79.2
79.3
78.7
80.1
80.1
Table I:
Phrase grounding results on Flickr30k Entities. We report the R@1, R@5, and R 10 metrics, where R@ 1/5/10 means calculating the recall using the top 1/5/10 generated bounding boxes.
4.1.1 PHRASE GROUNDING
We evaluate phrase grounding task Flickr30k Entities (Plummer et al., 2015) val and test splits. To reduce ambiguity, we do not prompt the model with individual phrases; instead, we use the current phrase along with the preceding words as input where preceding words serve context: " <p> (phrase) </p>". For the example shown in Figure 3(1), the model needs to predict the locations of phrases "A man", "a blue hard hat", "orange safety vest" and "an intersection" in the caption man in blue hard hat and orange safety vest stands in intersection.". ". To generate location tokens for the phrase "A man" that the beginning of the caption, the prompt is "<p>A man</p>". For phrase "orange safety vesf", the prompt is "A When there are multiple persons in the image, the context "A man in a blue hard hat and" explicitly helps the model locate the object to reduce ambiguity.
in blue hard hat and <p>orange safety vest< p>>
We convert location tokens in "<box>.. </box>" from the model's response into bounding boxes. A generated bounding box is correct if its intersection bounding box is greater than 0.5. If KOsMOs-2 generates a location sequence that can not be converted correctly (e.g.,
union (IoU) with the ground-truth
</bx>>">/>>/>>>////////>>>>>>>>> "), we treat it as a negative sample. We use the ANY-BOX protocol in MDETR (Kamath et al., 2021) and report the R@) 1, R@5, and R 10 metrics, where @@ 1/5/10 means calculating the recall using the top 1/5/10 generated bounding boxes. If there are fewer than 5 or 10 bounding boxes generated by KOSMOS-2, we use all available bounding boxes.
Results Table 1 presents results on Flickr30k Entities (Plummer et al., 2015) val and test splits. KosMos-2 outperforms GRILL (Jin et al., 2023), which relies margin under zero-shot setting. Moreover, our model outperforms the finetuned VisualBert (Li et al., 2019) model by 7.4% R @1 on both val and test splits. Compared with other models, KosMos-2 does not involve prior designs (e.g., object queries or proposals), leading to similar results among R R@) 1, R@5, and R @ 10. These results demonstrate that KosMos-2 generate high-quality locations without the need for post-processing redundant locations.
attached detector, by large
4.1.2 REFERRING EXPRESSION COMPREHENSION
The model is tested using three well-established datasets: RefCOCO (Yu et al., 2016), RefCOCO+ (Yu al., 2016) and RefCOCOg (Mao et al., 2015). Both RefCOCO and RefCOCO+ were generated through a two-player game while RefCOCO+ is designed to exclude spatial relations. RefCOCOg incorporates spatial relations and features longer expressions. Different from phrase grounding on Flickr30k entities, we measure this task by using referring expression as the input: "
< <issi>imae>>>> Image Embedding </ <ntagge><groundinne
<p> referring expression </p>". For the example in Figure 3(2), the input sequence is "<p>A A man in blue hard hat and orange safery vest< ppst"/p>> The predicted bounding box is correct its IoU with the ground -ruth bounding box is greater than 0.5. The failed decoded sequence is treated as negative sample. Regardless of whether the model's response contains one or multiple bounding boxes, we only use the first generated bounding box to measure the accuracy.
Results Table 2 reports referring comprehension results on RefCOCO, RefCOCO+ and RefCOCOg KosMos-2 also obtains promising zero-shot performance on the comprehension task, significantly

---

Published as a conference paper at ICLR 2024
Zero-
RefCOCO
RefCOCO+
ReICOCOg
Model
shot
val
testA
testB
val
testA
testB
val
test
UNITER (Chen al., 2019)
X
81.41
87.04
74.17
75.90
81.45
66.70
74.86
75.77
MIETRR (Kamath et al., 2021)
X
87.51
90.40
82.67
81.13
85.52
72.96
83.35
83.31
OFA (Wang et al., 2022c)
X
90.05
92.93
85.26
84.49
90.10
77.77
84.54
85,20
FIBER (Dou el al., 2022)
X
90.68
92.59
87.26
85.74
90.13
79.38
87.11
87.32
LM (Wang et al., 2023)
X
86.70
GRILL (Jin et al., 2023)
47.50
KosMos-2
52.32
57.42
47.26
45.48
50.73
42.24
60.57
61.65
Table 2: Accuracy of referring expression comprehension.
front right of other cows.
right of other cows.
Grounded MLLM
Grounded MLLM
</p> <boxc- <l0Gg>?> <locas?? <box>> is
<p> </p> <bax>> 4lows?
<p> </p> <boxc <losss? <logas" </bok> is
<loCson?
<lbox> the giraffe middle.
(1) Zero-shot evaluation
(2) Few-shot evaluation
Figure 4: The input format of referring expression generation evaluation under (1) zero-shot and (2)
fow-shot settings. The bounding boxes shown the image are for visualization purposes.
outperforming previous zero-shot models on RefCOCOg benehmark. However, compared to previous well-finetuned works, KOSMOS-2 achieves slightly lower performance on RefCOCO and RefCOCO+ than on RefCOCOg. This discrepancy can be attributed the data distribution present in RefCOCO and RefCOCO+, where they tend to use shorter referring expression (e.g.. "left bottom") during the two-player game. Hence, one of our future goals is to enhance MI.LMs' ability to accurately understand more types of human expressions.
4.2 MULTIMODAL REFERRING
In addition to multimodal grounding tasks, we evaluate the model's ability to understand image regions or objects users refer to via inputting bounding boxes. Compared with previous MLLMs that can only refer image regions or objects to the model via detailed text descriptions, directly referring image regions using its bounding boxes more effective and reduces ambiguity.
We evaluate the model on the referring expression generation task, which aims to generate unambigu- ous text descriptions of specific objects or regions within the bounding box. We employ the widely used RefCOCOg dataset (Mao et al., 2015) to evaluate the model's 's performance under both zero-shot and few-shot settings, showcasing its adaptability in different scenarios.
Evaluation Setup The model is tasked with generating associated text description for object or region given its location tokens of the bounding boxes (e.g., Benefiting from the unified input format, we use "<p> It //pp><oooxxooccxxxlloooo
is" as prompt to encourage the model to predict its text description. Figure 4 (1) and (2) demonstrate the input format for zero-shot and few-shot referring expression generation, respectively. Following previous works, we report results using METEOR and CIDEr metrics. The image resolution is 224x224. Greedy search is used for decoding.
Results Table 3 presents the zero-shot and few-shot results of on RefCOCOg. We compare KosMos-2 with a finetuned listener-speaker model, which introduces an added reward- based module (SLR). Our model obtains impressive zero-shot performance and even outperforms finetuned SLR by
7

---

Published as a conference paper at ICLR 2024
RefCOCOg
Model
Setting
Meteor
CIDEr
SLR (Yu al., 2017)
Finetuning Finetuning
15.4
59.2
SLR+Rerank (Yu et al., 2017)
15.9
66.2
Zero-shot
12.2
60.3
Kosmos-2
Few-shot (A: = 2)
13.8
62.2
Few-shot (Ac 4)
14.1
62.3
Table 3: Results of referring expression generation on RefCOCOg.
Flickr30k
VQAv2
Model
CIDEr
VQA acc.
VLM (Jin et al., 2022)
31.0
METALM (Hao et al., 2022)
43,4
41.1
Flamingo-3B (Alayrac et al., 2022)
60.6
49.2
Flamingo-9B (Alayrac al., 2022)
61.5
51.8
BLIP-2 (Vicuma-131) 13B) (Li et al., 2023b)
71.6
65.0
Kosmos-1 (Huang al., 2023)
67.1
51.0
Kosmos-2 (1.6B)
80.5
51.1
Table 4: Zero-shot image captioning results the Flickr30k test set and zero-shot visual question answering results on the VQAv2 test-dev set.
1.1 CIDEr scores. For the few-shot setting, KosMOs-2 achieves larger improvements, highlighting its in-context learning ability.
4.3 PERCEPTION-LANGIIA TASKS
In addition multimodal grounding and referring tasks, we also evaluate KosMOs-2 on the vision- language tasks. In particular, we perform zero-shot evaluations on two popular tasks, including image captioning and visual question answering. Appendix E provides more comprehensive comparison on SEED-Bench (Li al., 2023a) between KoSMOs-2 and recent MLLMs.
For image captioning, we evaluate the model on widely used Flickr30k Karpathy split test set. The beam search algoirthm is used for caption generation, with beam size of 5. The results are reported using CIDEr (Vedantam et al., 2015) metrics evaluated by COCOEvalCap! The prompt "An image of is used to generate the image description. For visual cnuestion-answeringg we evaluate zero-shot performance on the test-dev set of VQAv2. Greedy search is used for decoding. We report VQA scores obtained from VQAv2 evaluation server?. "Question: (question] Answer: [answer)" is used as the prompt for the dataset. The image resolution is 224x224 for both two tasks.
Table 4 displays the zero-shot performance. For image captioning Flickr30k, KosMos-2 with much fewer parameters achieves a remarkable score of 80.5, which significantly outperforms Flamingo-3B (60.6), Flamingo-9B (61.5). and BLIP-2 (Vicuna-133 13B) (71.6) with large margins. For visual question answering on VQAv2, KOSMOS-2 achieves a VQA accuracy of 51.1, which is on par with KosMos-1 (51.0) but lower than BLIP-2 (Vieuna-13B) at 65.0. With more compre- hensive functions, e.g., grounding and referring capabilities, KosMos-2 2 demonstrates competitive performance peccetiionllaguuaee tasks.
4.4 LANGUAGE TASKS
We evaluate KosMos-2 2 on eight language tasks, including cloze and completion tasks (StoryCloze, HellaSwag), Winograd-style tasks (Winograd, Winogrande), commonsense reasoning (PIQA), and
'nttps://github. com http:/gthww...ooo..aa/////ccooom hhttps://eval.cr aicaaalleccaaeeeeeaeeeaaeeee /30/vervve
8

---

Published as a conference paper at ICLR 2024
Story Cloze
Hella Swag
Model
Winograd
Winogrande
PIQA
BoolQ
CB
COPA
LLM
72.9
50.4
71.6
56.7
73.2
56.4
39.3
68.0
KOSMOS-L
72.1
50.0
69.8
54,8
72.9
56.4
44.6
63.0
KosMoS-2
72.0
49.4
69.1
55.6
72.9
62.0
30.4
67.0
Table 5: Zero-shot performance comparisons of language tasks between KosMos-2, KosMos-1 1 and LLM. LLM uses the same text data and training setup to reimplement a language model as KosMOS- 1. For fair comparison, report results of KosMOS-2 and KOSMOS-L without instruction tuning. Results of KOSMOs-1 and the LLM baseline are from Huang et al., 2023.
three SuperGLUE benchmark (Wang al., 2019) datasets (BoolQ, CB, and COPA). We report the zero-shot results in Table 5. Compared with KOsMoS-I 1, KosMos-2 achieves similar performance on StoryCloze, HellaSwag, Winograd, Winogrande, and PIQA, experiences decrease in performance on CB, but shows improvement on BoolQ and COPA. In summary, KoSMOS-2 demonstrates new multimodal grounding and referring capabilities when achieving comparable performance on language tasks, which demonstrates its potential to be versatile model.
RELATED WORK
The thriving development of large language models (LLMs, Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023) has paved the way for multimodal large language models (MLLMs, OpenAI, 2023; Alayrac et al., 2022; Wang et al., 2022b; Li et al., 2023b; Huang et al., 2023; Driess al., 2023; Pan al., 2023; Lv al., 2023), which seek to integrate language understanding and reasoning with multimodal perception and comprehension. Flamingo (Alayrac et al., 2022) fuses pretrained vision encoder and an LLM by introducing gated cross-attention structures, demonstrating impressive multimodal in in-context learning capability. KOsMoS-1 (Huang et al., 2023) is another work showing impressive performance under zero/few-shh and multimodal -hain-of-thhughh prompting settings. It is trained from scratch using web-scale multimodal corpora. Recently, instruction-tuning based MLLMs (Liu et al., 2023a; Zhu et al., 2023; Dai et al., 2023; Ye et al., 2023; Gong et al., 2023) endow pretrained LLMs (Touvron al., 2023; Chiang al., 2023) multimodal capability by constructing high-quality multimodal instruction datasets, Meanwhile, some works are proposed bridge vision systems with LLMs. VisionLLM (Wang et al., 2023) provides a flexible interaction interface for visual tasks, such as object detection, and segmentation. DetGPT (Pi et al., 2023) combines an MLLM and an extra detector (Liu al., 2023b) for grounding.
cnttuuction-ollooiinn
Compared to detectors or grounding models (Chen et al., 2021: Yang et al., 2021; Li et al., 2022b; Liu al., 2023b), KosMos-2 benefits from the advantages of LLMss, such as the ability comprehend more complex linguistic descriptions and perform superior reasoning. In contrast to existing MILM methods, KosMOs-2 incorporates grounding as foundational capability for MLLMs various downstream applications, resulting in more informative and comprehensive predictions. Please see Appendix FF for more comparisons.
6 CONCLUSION
We proposed KosMos-2, a multimodal large language modal, that can ground to the visual world. Specifically, we pretrained KosMos-2 by augmenting the multimodal corpora used in KosMos- with GRIT, a large-scale dataset of Grounded Image-Tex: pairs, which is created by extracting and associating noun phrases and referring expressions in the caption to the objects or regions in the scene. KOSMOS-2 enabled new capabilities perceiving image regions and grounding text output to the visual world, which makes grounding as a foundation capability of MLLMs in many downstream applications. Extensive experiments demonstrated that KosMos-2 exhibited competitive performance in language tasks, while achieving impressive results in vision-languse language tasks, grounding tasks, and referring tasks. KOsMOS-2 2 sheds light on the big convergence of language, multimodal perception, multimodal grounding, and multimodal referring.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Some example figures are from the WHOOPS corpus (Bitton-Guctta et al., 2023), We would like to acknowledge ydshich from HuggingFace for both the online demo and the integration of our work into the HuggingFace's transformers. We would also like to acknowledge the efforts of the NVIDIA team for integrating KosMos-2 into the NVIDIA foundation model endpoints.
ETHICS STATEMENT
The model presented in this paper intended for academic and research purposes. The utilization of the model to create unsuitable material is strictly forbidden and not endorsed by this work. The accountability for any improper or unacceptable application of the model rests exclusively with the individuals who generated such content. We also put Microsoft AI Principles' into practice when developing the models.
REFERENCES
Jean-Baptiss Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022. URL https:://// httpss///pppeeeeeeeeeee net htr////rmmmmmm/// EbMuimAbPbs.
Nitzan Bitton-Guetta Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, and Roy Schwartz. Breaking common sense: WHOOPS! vision-and-language benchmark of synthetic and compositional images. ArXiv, wwwpaab3./2222222 077274, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Emanuele Bugliarello, Aida Nematzadeh, and Lisa Anne Hendricks. Weakly seakly-supervised learning of visual relations in multimodal pretraining. arXiv:2305.1422 2023a.
Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, and Aida Nematzadeh. Measuring progress in fine-grained vision-and-language understanding. arXiv:2305.07558, 2023b.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.
Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geo rey E. Hinton. Pix2seq: language modeling framework for object detection. ArXiv, abs/2109, 10852, 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, 2019.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023, URL https:
//2msyss.rrg//log/2222 -0 3000-00000000000//001000------00 vicuna/.
'https: htttpp/s/www.mccrofft.coooo// htt../////aaerrpppssilllllaaaaae
10

---

Published as a conference paper at ICLR 2024
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek B Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier- Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. ArXiv, abs/2204.02311, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. ArXiv, abs/2305.06500, 2023.
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, and Lijuan Wang. Coarse-to-fine vision- language pre-training with fusion in the backbone. ArXiv, abs/2206.07643, 2022.
Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence. Palm-e: An embodied multimodal language model. ArXiv, abs/2303.03377, 2023.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qianmengke Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. ArXiv, abs/2305.0474 2023.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purposs interfaces. ArXiv, abs/2206.06333 2022.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python. 2020. doi: 10.5281/2emodo.12113 URL
https: http//gtttttteett thub. com/ wwtp//eppllllle
exxplssio os ceappoosio/ssppyy.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor, 2022. URL https://arxiv.. www.xiii..orooooooooooooooo /rg/ /
2212.09683
Shaohan Huang, Li Dong. Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishray Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. ArXiv, abs/2302. 14045, 2023.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2765, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 11551/66 www.:///o2.....oo...ooo 2022. 2022.ac1-long 197.
URL https: httpa/caaantttoooceeeeeoecc org/
Woojeong Jin, Subhabrata Mukherjee, Yu Cheng, Yelong Shen, Weizhu Chen, Ahmed Hassan Awadallah, Damien Jose, and Xiang Ren. Grill: Grounded vision- Languueee pre-training via aligning text and image regions. ArXiv, abs/2305. 2023.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. Mdetr modulated detection for end-to-end multi-moda understanding. IEEE ICCV, pp. 1760-1770, 2021.
11

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. wwXh:2230..02643 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Internationaa Journal of Computer Vision, 123:32-73,22 2016.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv:2107.06499 2021.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. ArXiv, abs/2307.16625, 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping ianuuage-image pre- training for unified vision-langues understanding and generation. In International Conference Machine Learning, pp. 12880- PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023b.
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wen Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. ArXiv, abs/2305.061 066355, 2023c.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. ArXiv, abs/1908.00 003557. 2019.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022b.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. WrXiv:2304.08485 2023a.
Visual instruction tuning.
Siyi Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chun yue Li, Jianwei Yang, Hang Su, Jun-Juan Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. ArXiv, abs/2303.05499 .55999 2023b.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv:2301. 13688, 2023.
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Ya-Chi Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, and Furu Wei. Kosmos-2.5: A multimodal literate model. ArXiv, 1b1429 2023.
Shuming Ma, Hongyu Wang, Shaohan Huang, Wenhui Wang. Zewen Chi, Li Dong, Alon Benhaim, Barun Patra, Vishrav Chaudhary, Xia Song, and Furu Wei. TorchScale: Transformers scale. CoRR, abs/221 1. 13184, 2022.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv, abs/2306.05524, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Dana-Maris Camburu, Alan Loddon Yuille, and Kevin P. Murphy, Generation and comprehension of unambiguous object descriptions. IEEE CVPR, pp. 11-20, 2015.
12

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-4 technical report. 2023.
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei. Kosmos-g:
Generating images in context with multimodal large language models. ArXiv, abs/2310.02992 2023.
Renjie Pi Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. Detgpt: Detect what you need via reasoning. ArXiv, his/2305.141611 2023.
Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, J. Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- -or-sssse correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74-93, 2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv:2210.088000 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' clien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In IEEE CVPR, pp. 4566-4575, 2015.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. SuperGLUE: A stickier benehmark for enneral-purpose language understanding systems. arXiv:1905.0055: 2019.
Hongyu Wang, Shuming Ma, Shaohan Huang. Li Dong, Wenhui Wang, Zhiliang Peng. Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, and Furu Wei. Foundation transformers. CoRR, abs/2210.06422) 2022a.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. CoRR, abs/2205. 14100, 2022b.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-se to-quuecce learning framework. In International Conference on Machine Learning. 2022c.
Wen Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. ArXiv, abs/2305.111755 2023.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision- vision-language modeling. In European Conference on Computer Vision, 2021.
Yuan Yao, Qi-An Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Pevl: Position enhatncee pre tre-training and prompt tuning for vision- langualan models. In Conference on Empirical Methods in Natural Language Processing, 2022.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li. Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Zhang, and Feiyan Huang. mplug-owl: Modularization empowers large language models with multimodality. ArXiv, 14.23304.1418.. 2023.
Licheng Yu. Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context 2016.
in referring expressions. ArXiv,
13

---

Published as a conference paper at ICLR 2024
Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L. Berg. A joint speaker- model for referring expressions. IEEE CVPR, pp. 3521-3529. IEEE Computer Society, 2017.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv:2111.0822 2021.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2024
A EXAMPLES OF KOSMOS-2
We evaluate KOSMOS-2 on variety of tasks, including multimodal grounding, multimodal referring. vasion-language, and language tasks, as discussed in Section To provide more intuitive under- standing of the model's capabilities, we have included several visualizations in this section. Figure 5 (I) illustrates an example of multimodal grounding, while Figure (4-6) and Figure 9 (2) showcase multimodal referring via bounding boxes.
However, as previously mentioned, the grounding capability of KOSMOS-2 enables a range of new applications to emerge. For instance. Figure 6 highlights the potential of the multimodal referring capability for enhancing human-Al interaction in visual dialogue. Figure 7, our approach exhibits its in-context learning ability for fine-grained object detection, utilizing both text and image descriptions. Examples of grounded visual question answering can be seen in Figure (2-3) and Figure (I), while Figure (7) and Figure 9 demonstrate grounded detailed image captioning.
Image credits: We would like to express our gratitude for the images sourced from the WHOOPS corpus (Bitton-Guetta al., 2023), SA-1B (Kirillov al., 2023), and MS COCO (Lin al., 2014).
B TRAINING HYPERPARAMETERS
The hyperparameters of KosMOs-2 are listed in Table 6, while the instruction tuning hyperparameters are listed in Table 7.
Hyperparameters
Image embedding number
64
Hyperparameters
Location tokens
1,024
Training steps
60,000
Training steps
10,000
Warmup steps
375
Warmup steps
375
Optimizer
AdamW 2c-4
Learning rate
le-5
Learning
Batch size language instruction data
117
Learning rate decay
Lineur
Batch size of vision-language instruction data
351
Adam 8
(0.9, 0.98)
Batch size of grounded image-text pairs
Weight decay
0.01
& grounded instruction data
1404
Batch of corporu
93
Batch size of text corpora
30
Batch size of original image-capp captecn pains
1,117
Batch size of interleaved data
15
Batch size of grounded image-text pairs
1.117
size of interleaved data
47
Table 6: Hyperparameters of KosMos-2
Table 7: Instruction tuning hyperparameters of KosMOs-2
C GRIT
C.I COMPARISON WITH OTHER DATASETS
We compare the created GRIT with existing publicly accessible visual grounding datasets in Table 8. One can see that our GRIT dataset significantly outperforms other existing visual grounding datasets in terms of the number of images, objects. and text spans. With 90,614,680 images, 137,349, objects, and 114,97,223 text spans, GRIT is considerably larger dataset than any of its counterparts, such as Flickr Entities (Plummer et al., 2015), RefCOCOg (Mao et al., 2015), 5), RefCOCO (Yu et al., 2016), RefCOCO+ (Yu al., 2016), and Visual Genome (Krishna al., 2016).
In addition, the data in GR IT' is more diverse: regarding the text description, GRIT encompasses both referring expressions and noun chunks. Each referring expression or noun chunk could be associated with one more bounding boxes. More importantly, the image-caption pairs utilized in the GRIT dataset are derived from large-scale web datasets (e.g.. LAION-2B (Schuhmann et al., 2022) and COYO- (Byeon et al.. 2022)). which implies a rich variety of content covering various scenes and topics. This diversity contributes to providing a more comprehensive and realistic setting for visual grounding tasks, ultimately enhancing the generalization capabilities of models.
15

---

Published as a conference paper at ICLR 2024
Input Prompt
WERECOON TO
CAPNORY STRECC
Question: How many cows are here? Answer:
Question: doet the sign say? Answer:
the left eye of emoji
The sign says "W Welcome Camaby Street."
two cows present in the image.
Completion
WESLCOHEE TO
STRRET
(1)
(2)
(3)
Input Prompt
Samply
Simple
Sin
Q Question: W ih the biggest difference betwen bottle-1 and bottle-2? Answer:
Question: Why this animal unusual? Answer:
Q Question: ihat it? Answer:
This animal unique because turtle. Turtles are known for their slow and steady pace, whereas rabbits are known to be agile and quick. is not common for turtle to be faster than rabbit, especially in terms of speed.
Completion
buoy attached to the boat.
The biggest difference between bottle-1 and bottle-2 is the label.
(4)
(6)
snowman sits toxt a campfire in the snow. He wearing hat, scarf, and mittens. There are several pots nearby, likely containing hot meal for the snowman. The scene serene and peaceful, with the snowman looking at the fire with warm smile.
Input Prompt
Completion
Describe this image in detail:
(7)
Figure 5: Selected examples generated by KosMos-2. The examples include (1) visual grounding, (2)-(3) grounded question answering, (4)-(6) multimodal referring via bounding boxes, and (7) grounded image captioning. (Best viewed in color)
16

---

Published as a conference paper at ICLR 2024
What the of this building?
Sydney Opera House.
When completed?
1973.
What is the name of this?
Sydney Harbour Bridge.
What he doing?
The man is riding horse.
How about this man?
He holding flag.
person this one facing to?
The person the right side of the fence.
Figure 6: Examples of visual dialogue generated by KosMOS-2. (Best viewed in color)
17

---

Published as a conference paper at ICLR 2024
<p>a downy woodpecker</p>
<box><log-><lonn>>>>oox
X
Description three toed woodpecker: it has black and white stripes throughout the body and yellow crown. Description of downy woodpecker: it has white spots its black wings and some red on its crown.
With text description
<p>a downy woodpecker</p>
<box<<locu=><100gg>>/00x
With image description
This is downy woodpecker
downy woodpeckers/p>
<box><losgg><loogg>>/>>xx
Figure 7: Examples of object detection with multimodal descriptions from KosMOS-2.
18

---

Published as a conference paper at ICLR 2024
de Seis CHARlie
Input Prompt
Question: here the coach?
Question: does say? Answer:
Question: Whatmaker this image weird? Answer:
Answer: <p>The coach</p>
The image weird because the snowman sitting next to campfire the snow.
coach standing the middle the field.
"J suis Charlie!" written in French chalkboard.
Completion
de swis
CHARliE
(1) Grounded question answering
Input Prompt
Y are driving this pick man, you need
Question: What will happen to it? Answer:
Question: Where is Answer:
In a microwave oven.
Completion
go through intersection and tum right.
will melt away as it warms up.
(2) Multimodal referring via bounding boxes
Figure 8: Examples generated by KosMos-2. (Best viewed in color)
19

---

Published as a conference paper at ICLR 2024
The eagle is large bird with black and white body and yellow beak. It has large, dark wings and pointed tail. stands on a stump and spreads wings, displaying magnificent plumage.
Input Prompt
Completion
Describe this bird in detail:
Input Prompt
Completion
Six fighter jets fly in formation in the sky, creating beautiful cloud trail.
Describe this image in detail:
A man standing front a fire, blowing the flames out of his mouth. He is in the middle of tropical setting with palm trees surrounding him.
Input Prompt
Completion
Describe this image detail:
Figure 9: Examples of grounded image captioning generated by KosMOs-2. (Best viewed in color)
Dataset
Images
Objects
Text Spans
Avg Expression I Length
Flickr Entities (Plummer al., 2015)
31,783
275,775
513,644
RefCOCOg (Mao al., 2015)
26,711
54,822
85,474
8.43
RefCOCO et al., 2016)
19,994
50,000
142,209
3.61
RefCOCO+ (Yu et al., 2016)
19,992
49,856
141,564
3.53
Visual Genome (Krishna et al., 2016)
108,077
4,102,818
GRIT (Ours)
90,614,680
137,349,210
114,978,233
4.7
Table 8: Comparison GRIT with existing visual grounding datasets.
C.2 TEMPLATES FOR GROUNDED INSTRUCTION DATA
Table 9 presents the instruction templates of expression generation based on its associated bounding boxes during instruction tuning.
C.3 EXAMPLES OF GRIT
We present some examples of the GRIT corpus in Figures 10,11,12,13. The grounded image-text pairs span over various domains and contain different numbers of objects.
20

---

Published as a conference paper at ICLR 2024
1. "What is <p> htt/pp//ooee//oox=<loo/s//////>> is (expression)."
"What <p> this //p=xbon==l0oaxxx11c>>>>//>>1 This is (expression)."
"Describe <p> this htt.bpppppepppptteeeoooooo---/aa
This object (expression)
4. "<p> It butposlopettaca///oooo//////aaas
5. "<p> This </pxxboe-<Locn/x/occx>>>///b>5 (erprs/see [expression]."
"<p> The oboptt//pptbboorooooxxxoo////// is (expreesiii.sss
Table 9: Instruction templates used for expression generation.
RefCOCO
RefCOCO+
Flickr30k
RefCOCOg
Ablation Settings
val
testA
testB
val
testA
testB
val
test
val
test
Baseline
52.3
57.4
47.3
45.5
50.7
42.2
77.8
78.7
60.6
61.7
Instruction tuning
49.2
56.3
43.8
45.1
50.1
40.6
76.7
77.5
59.6
60.0
Expression data
41.4
45.1
38.1
35.7
37.1
34.3
78.5
79.0
29.2
28.3
Table 10: Ablation of linguistic description type of objects. Zero-shot Recall @ metric is tested on Flick30k while zero-shot accuracy protocol is reported on RefCOCO/+/g.
D ABLATION STUDY
In Section 2, a pipeline is proposed for constructing web-scale grounded image-text pairs in two steps: generating nan-cchunk-bounding--bbo
pairs and producing This process results two types of text descriptions: noun-chunks and referring expressions. To assess the impact of these two description types on the final visual grounding performance, ablation studies are conducted. is found that using referring expression data alone is insufficient for training the model. Therefore, a mixture of the two types of data (text spans) is used for training, which also serves as the default setting.
pairs.
From Table 10, it can be observed that the performance experiences slight degradation when the instruction tuning phase is disabled. This can be attributed to our utilization of referring expression data enhance the multimodal referring capability during the instruction tuning phase. Upon further removal of the referring expression data (i.e., disabling Step 2 in Figure 2). there is a significant decline in the results on referring expression comprehensive benchmarks. This highlights the effectiveness of the construction pipeline we proposed in Section 2.
E EVALUATION ON SEED-BENCH
Existing benchmarks for MILMs are limited by inadequate qualitative examples and unsuitable assessments for open-form output. To address, SEED-Bench (Li et al., 2023a) was proposed as su- perior benchmark, which consists of 19K multiple choice questions with accurate human annotations and covers 12 evaluation dimensions across both image and video modalities. This comprehensive and objective benchmark enables precise and in-depth evaluation of MLLMs. On SEED-Bench, we compare KOsMOS-2 with popular MLLMs including MiniGPT4 (Zhu et al., 2023), LLaVA (Liu et al., 2023a), BLIP-2 (Li et al., 2023b), InstructBLIP (Dai et al., 2023), MultiModal-GPT (Gong et al., 2023), mPLUG-OW (Ye et al., 2023), VideoChat (Li et al., 2023c) and Video-ChatGPT (Maaz et al., 2023), Table 11.
As shwon in Table 11. KOSMOS-2 demonstrates remarkable performance with significantly fewer parameters than the counterparts. Although it is marginally outperformed by InstructBLIP (Dai et al., 2023), KosMOS-2 surpasses the other models in the comparison. Specifically, KosMos-2 achieves the best results in scene understanding, instance location, instance interaction, visual reasoning, and action recognition tasks.
One noteworthy observation is that KosMos-2 excels in several instance-level tasks (e.g., instance location, instance interaction), indicating that its grounding capability is crucial for fine-grained
21

---

Published as a conference paper at ICLR 2024
Table 11: Performance comparison on SEED-Bench (Li al., 2023a). Ti to T12 represent various
tasks in the fields of Image and video understanding and reasoning: Ti Scene Understanding,
T2 - Instance Identity, T3 Instance Attribute, Ti Instance Location, Ts Instance Counting. T6 Spatial Relation, T7 Instance Interaction, Ta Visual Reasoning, To Text Recognition, Tio- Action Recognition, Tii Action Prediction, and T12 Procedure Understanding. Furthermore, Ti represents the average performance across the first nine image-based tasks, Tv signifies the average performance on the last three video-related tasks, and TAUl indicates the mean performance over all twelve tasks.
Performance on 12 tasks
Model
Language Model
7
7:
F;
T
T5
T;
Tio
Tau
Lorge Longuage Models
LLaMA
LLaMA-7B
26,3
27.4
26.2
28.3
25.1
28.8
19.2
37.0
9.0
26.6
$3.0
23.1
26.2
27.3
26.8
Vieuna
Vicuna-7B
23.4
30.7
29.7
30.8
28.6
29.8
18.5
13.4
28.2
27.3
23.8
28.5
Muhinodul Large Language Models (Image)
MultiModal-Gi GPT
LLaMA-7B
43.6
37.9
31.5
30.8
27.3
30.1
29.9
51.4
18.8
34.5
36.9
25.8
24.0
29.2
33.2
LLaNA
LLaMA-7B
42.7
34.9
33.5
28.4
41.9
30.8
27.8
16.8
27.7
37.0
29.7
21.4
19.1
23.8
33.5
LLaMA-7B
49.7
45.3
32.5
36.7
27.3
32.7
44.3
54.7
28.8
37.9
26.7
17.9
26.5
23.0
34.0
MiniCPPT4
Vicuna-78
47.4
42.8
Flan-TSXXI.
59.1
43.2
36.7
55.7
45.6
25.9
49.7
47.5
46.4
InstuctBLiPP
Vicuna-7B
60.2
53.9
65.6
43.6
57.2
40.3
52.6
47.7
43.5
58.8
34.5
49.6
23.1
38.1
$3.4
(osMos-2tours)
only 1.3B
63.4
57.1
58.5
44.0
41.4
37.9
$5.7
60.7
25.9
54.4
41.3
40.4
27.0
37.5
50.0
Multioodil Large Language Models (Vidro)
Video-ChatGiPT
LLaMA-7B
37.2
35.5
29.5
23.7
42.3
25.9
33.9
27.6
23.5
31.2
VideoChar
Vicuna-1B
47.1
43.8
34.9
40.0
32.8
34.6
42.3
50.5
17.7
39.0
34.9
36.4
27.3
33.7
37.6
understanding and reasoning. This highlights the superiority of KoSMos-2 in handling complex tasks that require deeper understanding of the underlying data.
F DISCUSSION WITH MORE METHODS
In the field of Vision LsniuunLa Models (VLMs), several innovative approaches have been proposed, emphasizing the incorporation of spatial information or object detection in pretraining. OFA (Wang al., 2022c) targets unifying various cross and mono modal tasks including image generation, visual grounding, image captioning, image classification, language modeling, etc., and has achieved impressive performances, by utilizing the concept of bounding boxes as tokens from Pix2Seq (Chen et al., 2021). PEVL (Yao et al., 2022) also processes spatial positions as discrete tokens (Chen et al., 2021) and integrated them with language tokens in a unified masked language modeling framework. X-VLM (Zeng et al., 2021) presents an approach that concentrates on multi- multi-grained pretraining. It leverages bounding boxes to glean region- visual features, aligning them with meeigraieed text descriptions through contrastive learning. GLIPv2 (Zhang et al., 2022) model streamlines the process by merging localization pretraining with vision-language pretraining. It employs three tasks: phrase grounding, region- word contrastive learning, and masked language modeling. GRILL (Jin et al., 2023) leverages object-text alignments for learning object grounding and localization, facilitating task transferability, demonstrating adaptability across various tasks such visual question answering, captioning, and grounding tasks, with zero or few training instances.
Unlike these approaches, our proposed model, KosMos-2, aims unlock the referring and ground- ing capabilities of multimodal large language models (MLLMs) by training location and language tokens in auto-regressive paradigm. KosMos-2 not only performs well on conventional vision- language tasks such as image captioning, visual question answering, and visual grounding, but also integrates these capabilities into downstream tasks to enable new applications in an open-ended style, thereby extending the capabilities of MLLMs. As Bugliarello et al. emphasize. teaching VLMs object concepts essential for effectively learning fine-grained skills (Yao et al., 2022: Zeng et al., 2021; Li et al., 2022a). This perspective provides a plausible explanation for our impressive evaluation results on conventional benchmarks and comprehensive benchmark for MILMS.
Previous research (Li et al., 2022a; Lee et al., 2021; Bugliarello et al., 2023a) has underscored the significance of data curation bolstering performance. this work, we have taken different approach by creating the GRIT dataset, specifically designed to unlock new capabilities for MLLMs. The methodology employed in the construction of this dataset is generalizable and could offer valuable insights for the larger research community in creating large,scale, task-specific datasets.
22

---

Published as a conference paper at ICLR 2024
Smoll Dow filled with creomy light greer
on olu minium troy
roasted vegetable solod
Figure 10: Example from GRIT. Caption: "A serving of kale and roasted vegetable salad on an aluminium tray served with small white bowl filed with creamy light green avocado Caesar dressing".
A Keto Chicken Nugget
0 bowl of keto honey mustord
Figure 11: Example from GRIT. Caption: "A Keto Chicken Nugget being dipped into a bowl of keto honey mustard.".
23

---

Published as a conference paper at ICLR 2024
The Sydney skyline
Solor cells on 0 red roof
Figure 12: Example from GRIT. Caption: "Solar cells on red roof are the foreground. The Sydney skyline is in the background.""
The background
Woman standing outdoors in city landscape and wearing hijab a hijob
Her arm
young girl who is hugging her side
Figure 13: Example from GRIT. Caption: "Woman standing outdoors in city landscape and wearing a hijah. Her arm is around a young girl who is hugging her side. The background is blurred.".
24


Reviews:
[{"review_id": "xksheMqEjF", "reviewer": "Reviewer_vJ2h", "summary": "The paper introduces \"KOSMOS-2\", a Multimodal Large Language Model (MLLM) that has the ability to perceive object descriptions, such as bounding boxes, and ground textual content to the visual world. This is achieved by representing specific text spans, like referring expressions and noun phrases, as links in Markdown format, connecting the text to location tokens that denote the object's position. A significant dataset called GRIT, consisting of grounded image-text pairs, is constructed to train the model. This dataset is built using subsets from previous image-text pair datasets, namely LAION-2B and COYO-700M. Experimental results demonstrate KOSMOS-2's leading performance in grounding and referring tasks, as well as competitive results in traditional language and vision-language tasks. This grounding capability allows KOSMOS-2 to be used in various applications, such as grounded image captioning and grounded visual question answering. The study underscores the convergence of language, multimodal perception, and world modeling, a vital progression toward artificial general intelligence.", "questions": "Is there data cleaning process to make sure that training dataset do not contain any image from evaluation dataset?\n\nFor table 5, what's the training cost of LLM, Kosmos-1 and Kosmos-2. If LLM is trained longer and match the gpu hours of kosmos-2, will that change the results in table 5.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The core innovation is the method to link text spans in image captions to spatial coordinates of the objects or regions in the image. These coordinates are converted into location tokens and attached to the text span, forming a \"hyperlink\" that connects the image's objects or regions to its caption. When trained, KOSMOS-2 can link these text spans in generated text to specific image regions, resulting in more accurate and comprehensive vision-language predictions. The model can also identify and refer to objects using pronouns, enhancing its reasoning capabilities. \n\nBesides that, the paper is well written and clearly presented.", "weaknesses": "The GRIT dataset is trained on Grit together with text to image instructional dataset, and the work claims that the model maintain the conventional multimodal capability. It is important to evaluate the model on more multi understanding tasks to validate the modeling approach. The model is only evaluated on Flickr30k and VQAv2, a more comprehensive evaluation is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KvbPoF8j28", "reviewer": "Reviewer_n3ax", "summary": "The paper presents a way to allow multimodal language models to refer to and ground objects in images. Grounded text spans are represented as [text span](bounding boxes), and similarly one could use such formats to input and output bounding boxes. A separate grounding model is used then to annotate boxes for web image-text data and Kosmos-2 is pre-trained on such image-grounded text data. Evaluation shows that the model archives zero/few-shot performance comparable to previous supervised grounding / grounded understanding models.", "questions": "Will some of the grounded data (converted from public data) be made public?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is simple and effective. Using such markdown-like formats allows the model to easily take boxes as input or output grounding boxes. The authors also spend time discussing the details in converting image-text data into such grounded data, including how to extract the correct noun chunk (Section 2), and how to convert boxes into a markdown-like format (Section 3.1), which is welcomed.\n\n- The evaluation shows the model can handle diverse tasks: 1) accepting a phrase and outputting boxes; 2) accepting boxes and outputting descriptions; 3) classical VL tasks such as VQA and language understanding tasks.", "weaknesses": "1. Limited methodology innovation\n\n    There is no significant methodology improvement, as outputting bounding boxes as discrete tokens has been extensively studied in prior work such as [1]. \n\n    [1]. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework. Wang et al., 2022.\n\n2. Ablation study\n\n     While the evaluation is extensive, there is no ablation study or discussion of key design choices or error analysis. Thus it is hard to draw further insights from the experiments. Below I just list a few questions that I have but I would encourage the authors to include ablation study / error analysis.\n\n     a. How much impact does the data cleaning process (Section 2) have on the final performance? \n\n     b. Where does the in-context learning ability come from? Is it solely from interleaved image-text data or are in-context grounded data specifically included?\n\n     c. Can we fine-tune Kosmos on downstream grounding tasks? If so, will the system match previous fine-tuned models?\n\n     d. Few-shot in-context prompting results are shown only for RefCOCOg. Could we also do few-shot prompting for grounding tasks such as Flickr30K?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YfeBgoYtmh", "reviewer": "Reviewer_Cmrj", "summary": "The paper proposes Kosmos-2, a Multimodal Large Language Model (MLLM) that can not only do vision-language tasks based on image inputs, but also enables both inputting and outputing bounding box regions. The authors first introduce how to construct a training dataset of grounded image-text pairs, then illustrate how they enable the model to do grounding and referring by using bounding box representation as a hyperlink. The experimental results demonstrate the new grounding and referring capabilities of the model, and also comparable image understanding ability with other MLLMs.", "questions": "I am curious about the performance of integrating an object-detection model with Kosmos-2. Say, if we provide the location of the objects for VQA tasks, what performance can the model achieve?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is promising and important. By enabling MLLM to do grounding and referring, it brings more flexibility to the model and enables it to do a lot more fancy things. We can imagen how convenient it will be if users can specify a region when querying a MLLM. The paper makes a first step towards building such a model together with a training dataset.\n2. Experimental results show that the model is comparable with previous MLLM on various traditional vision-language tasks, while shows new capabilities for grounding and referring.", "weaknesses": "I don't see a big weakness here. One thing is that the proposed model achieved much worse performance than the previous grounding & referrring models. Ideally, with language model's stronger language capabilities, the model should achieve better performance on these tasks. Therefore, there are still rooms here for improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wPf6aijJQ5", "reviewer": "Reviewer_873R", "summary": "This paper introduces Kosmos-2, a multimodal LLM-based architecture pretrained to perform object localization along standard (image-grounded) text generation. Object localization is also framed as text generation by converting bounding box coordinates to discrete tokens that are added to the language vocabulary. The pretraining data is obtained through an automatic pipeline that extracts phrase–bbox from existing large-scale image caption data. Evaluation on localization-based tasks (phrase grounding, referring expression comprehension/generation) shows that the model acquires strong zero-shot and few-shot learning abilities, while maintaining, while maintaining similar performance on other multimodal and text-only tasks as Kosmos-1 (from which the model is initialized).", "questions": "1. Can you identify any specific abilities of Kosmos-2 that lead to improvement in BoolQ and COPA? And what about systematic failures in CB?\n2. In Sec 4.1.2, does the input sequence also include the token “<grounding>” after “</p>”? If not, why? If I understood well, having the token “<grounding>” would better resemble the pretraining GrIT data format.\n3. Will you open-source the code to reproduce the pipeline of Figure 2 to produce dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed pipeline to transform image captions into phrase–bbox pairs is interesting, and it holds promise for further development of data manipulation techniques to improve model performance.\n2. The proposed model modifications to use the newly generated data consolidate and effectively extend previous approaches that aimed at grounding LMs by adding object location in text.\n3. The resulting model obtains strong zero- and few-shot performance on grounding tasks (Sec 4.1 and 4.2), whilst generally maintaining the performance of the model it was initialized from (Kosmos-1)\n4. The paper is generally well-written and easy to follow", "weaknesses": "1. While generally clear, I do not think I would be able to reproduce the pipeline of Figure 2. The submitted code does not seem to include the steps required to obtain data similar to what is in GrIT. This heavily limits the applicability of the proposed method and future comparisons that aim at improving data manipulation strategies.\n2. My main concerns are related to some of the evaluation methodology and claims made in the paper. \n- (2a) While results are indeed strong for grounding and referring expressions, I disagree that “Kosmos-2 achieved impressive results on language and vision-language tasks”. The model achieves competitive performance with its baseline. \n- (2b) Using Flickr30K as captioning benchmark is suboptimal due to the small number of images (1K) – I encourage the authors to report performance on COCO, which most work uses as benchmark for captioning. \n- (2c) While the authors claim that “grounding capability born with KOSMOS-2 enables it to be applied to more downstream tasks, such as grounded image captioning, and grounded visual question answering” no such tasks are tested. An evaluation on a dataset like “VizWiz-VQA-Grounding” [1] would be appreciated towards grounding these claims. Likewise, claims about better action modeling (Conclusion) are unrelated to the proposed model.\n- (2d) In Section 4.4, the authors state that “KOSMOS-2 demonstrates new capabilities when achieving comparable performance on language tasks.” This is after showing similar performance on most tasks, worse performance on CV and better performance on BoolQ and COPA. What kind of text-only capabilities are achieved by Kosmos-2? Are there patterns that allow us to pinpoint why the model performs better in these two tasks? And what about performance on CB, the drop is much larger (14.2pp) than any (or even the sum) of the gains on BoolQ and COPA.\n3. Another concern that I have is about the unnecessarily claims towards world modeling and AGI. I agree that (better) grounding LLMs to vision goes towards these goals, however, I do not think this should be stressed in the title and abstract (which I invite the authors to consider modifying). While adding additional visual grounding abilities, this approach does not ground “models to the world” in a much more significantly extended way than concurrent work. This can mislead the readers, and given the increased interest in our field from the general public, I believe we researchers should not make claims stronger than needed. The paper proposes good contributions to improve visual grounding, and I think they should be framed as such.\n4. The Related Work section needs substantial expansion. Discussion on models like PEVL [2] and X-VLM [3] that pretrain VLMs with object detectors are missing, as well as some of their capabilities [4]. There is also no mention of other approaches towards data curation that can improve model performance, including work like BLIP [5], VSG [6] and other relevant work in NLP [eg, 7].\n\n---\n[1] Chen et al. Grounding Answers for Visual Questions Asked by Visually Impaired People. CVPR’22.\n\n[2] Yao et al. PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models. EMNLP’22.\n\n[3] Zeng et al. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. ICML’22.\n\n[4] Bugliarello et al. Measuring Progress in Fine-grained Vision-and-Language Understanding. ACL’23.\n\n[5] Li et al. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML’22.\n\n[6] Bugliarello et al. Weakly-Supervised Learning of Visual Relations in Multimodal Pretraining. arXiv 2305.2305.\n\n[7] Lee et al. Deduplicating Training Data Makes Language Models Better. ACL’22.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)