CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
Authors: Junyan Li
Delin Chen
Yining Hong
Zhenfang Chen
Peihao Chen
Yikang Shen
Chuang Gan
Venue: ICLR 2024 poster (2024)
Keywords: vision-language model
compositionality
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Our proposed CoVLM improves compositional reasoning ability of VLM through dynamically communicating between vision and language, achieving the SOTA on both compositional reasoning task and traditional VL tasks.

Abstract: A remarkable ability of human beings resides in compositional reasoning, i.e., the capacity to make "infinite use of finite means". However, current large vision-language foundation models (VLMs)  fall short of such compositional abilities due to their ``bag-of-words" behaviors and inability to construct words that correctly represent visual entities and the relations among the entities. To this end, we propose CoVLM, which can guide the LLM to explicitly compose visual entities and relationships among the text and dynamically communicate with the vision encoder and detection network to achieve vision-language communicative decoding. Specifically, we first devise a set of novel communication tokens for the LLM, for dynamic communication between the visual detection system and the language system. A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far. The proposed regions-of-interests (ROIs) are then fed back into the LLM for better language generation contingent on the relevant regions. The LLM is thus able to compose the visual entities and relationships through the communication tokens. The vision-to-language and language-to-vision communication are iteratively performed until the entire sentence is generated. Our framework seamlessly bridges the gap between visual perception and LLMs and outperforms previous VLMs by a large margin on compositional reasoning benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on ARO top-1 accuracy). We also achieve state-of-the-art performances on traditional vision-language tasks such as referring expression comprehension and visual question answering.

Published as a conference paper at ICLR 2024
COVLM: COMPOSING VISUAL ENTITIES AND RELA- TIONSHIPS IN LARGE LANGUAGE MODELS VIA COM- MUNICATIVE DECODING
Junyan Li UMass Amherst
Delin Chen
Yining Hong
Wuhan University
University of California, Los Angeles
Zhenfang Chen MIT-IBM Watson AI Lab
Peihao Chen
Yikang Shen
South China University of Technology
MIT-IBM Watson Al Lab
Chuang Gan
UMass Amherst and MIT-IBM Watson Al Lab
ABSTRACT
A remarkable ability of human beings resides in compositional reasoning, ie., the capacity make "infinite use of finite means". However, current large vision- language foundation models (VLMs) fall short of such compositional abilities due to their "bag-of-words" behaviors and inability to construct words that correctly represent visual entities and the relations among the entities. To this end, we propose CoVLM, which can guide the LLM explicitly compose visual entities and relationships among the text and dynamically communicate with the vision encoder and detection network to achieve vision-language communicative decod- ing. Specifically,
first devise set of novel communication tokens for the LLM, for dynamic communication between the visual detection system and the language system. A communication token is generated by the L.LM following visual entity or a relation, to inform the detection network to propose regions that are relevant the sentence generated so far. The proposed regions-of-interests (ROIs) are then fed back into the LLM for better language generation contin- gent on the relevant regions. The LLM is thus able compose the visual enti- ties and relationships through the communication tokens. The and language-to-ve visiin communication are iteratively performed until the entire sentence is generated. Our framework seamlessly bridges the gap between vi- sual perception and LLMSs and outperforms previous VLMs by a large margin on compositional reasoning benchmarks (e.g.. 20% in HICO-DET mAP, 14% in Cola top-1 accuracy, and ~ 3% on ARO top-1 accuracy). We also achieve competitive performances traditional vision-language tasks such as referring expression comprehension and visual question answering
INTRODUCTION
A remarkable ability of human beings resides in compositional reasoning: the capacity construct an endless number of novel combinations from finite set of known components, i.e., "infinite use of finite means" (Chomsky, 1965; 1957; Montague, 1970). As depicted in Figure 1, for someone who has never witnessed a scene where person sits on a sulky, it's not hard to render this conclusion by combining the known components "man", "is sitting on" and "sulky". Compositionality is omnipresent in the language such that sentence is made up of words like nouns ("man"" and verbs ("sit"). also exists ubiquitously in vision so that we could easily detect visual entities such as the person and the sulky, composed with relationships like "sit on". It's believed by cognitive scientists that the meaning of sentence lies in the interaction between an utterance and external situations
Project page: https:/ //vi www. cs. uma 8. w...edd//ooo edd/Covi.M/

---

Published as a conference paper at ICLR 2024
Training
Existing VLM Models
CoVLM
the horse
sulky
Vanilla Decading
Communicative Decoding A man
C man is sitting
sitting on
sitting on horse
Chair grass
Sofa
car
groud
Figure 1: Comparison with existing VLMs. Previous models take in whole image as input, im- pairing the compositionality of VLMs. Our CoVLM inserts communication tokens into the LLM after visual entitiesrrelaioonhiiss to enable language-to- -o-viiiino and vision-to-language communica- tion, improving compositionality to large extent.
that can be perceived the meaning of a noun phrase is linked to a visual entity, and the meaning of a verb phrase is linked to relational property (Janssen & Partee, 1997). From the meanings of the subject, verb phrase, and object, the sentence built in a systematic and compositional way.
Current Vision- LaiionLanguage Models (VLMs), however, tend to fall short of such compositional abili- ties (Ma et al., 2023; Cascante- -oniilee---lllaaaaee et al., 2023; Doveh al., 2022; Zhao et al., 2023). As noted by recent works, deficiency of compositionality in these VLMs is likely due to the hypothesis that they behave like "bag-of- of-wwrds" (Yuksekgonul et al., 2022) - that they merely memorize by rote frequent co-occurrences of words, but fail to construct words that could correctly represent objects and the relations between objects. Previous works (Zhao et al., 2023; Cascante- Bonilll et al., 2023) have shown that VLMs struggle lot when relationships are involved. We can also come this con- clusion from Figure 1, in which the models utilize the shortcut learned from pre-training that "a man sits horse" appears frequently and there's a and horse in the image, utterly overlooking the real object, sulky, that the person is sitting on.
Delving into the architectures of these VLMs and how they infuse images into LLMs, find that these VLMs deviate from the way human beings perform compositional reasoning from sev- eral perspectives. First, they feed one single image as whole into LLMs and generate language descriptions based on the holistic image embedding. This is inconsistent with object-centric repre- sentations in vision, through which the whole image can be constituted by visual entities and more importantly, relationships between the entities. Second, these methods disregard the interaction be- tween the sentence parts and the ingredients in the images. The generation of new word by the LLM is not linked to specific visual entity or relationship but is contingent on previous words and holistic image features instead. Although series of works have been proposed to strengthen VLMs' compositional abilities (Doveh et al., 2023), they mainly probe the problem by proposing additional datasets. However, as stated by recent analysis on compositionality (Doveh et al., 2022), collecting specialized large-scale data teach vision-language models the missing compositionality is imprac- tical, as finding specialized text-image pairs for each kind and possible value of the visual entities and their relations is rather expensive. In this paper, we approach the essence of this problem from the perspective of model architecture, unveiling compositional structure of LLM that can conduct step-by-step communication with visual components and relationships.
We propose CoVLM, which guides the LLM to explicitly compose visual entities and relation- ships among the text, and dynamically communicate with the detection network achieve vision- language communicative decoding. Specifically, we devise novel set of communication tokens for dynamic interaction and communication between the detection network and the L.M.. Commu- nication tokens are generated by the LLM, after the language tokens that denote visual entities or relationships. Upon the generation of communication tokens, detection network utilized to de- code the regions relevant to the generated language sequence so far, and propose several bounding box proposals. The features of relevant regions are then fed back to LLM by communication tokens, conditioned which the LLM decodes the subsequent tokens. The bottom-up vision-to-lan toainngg-- and top-down manguage-to-vistte communicative decoding are iteratively performed until all words and tokens are generated. The paradigm is shown the right part of Figure
2

---

Published as a conference paper at ICLR 2024
We first evaluate our CoVLM on compositional reasoning tasks, including predicting the object entity given the subject and the relationship (ARO (Yuksekgonul et al., 2022)). matching the correct captions describing the relation between two images with similar entities (Cola, (Ray et al., 2023)), and human-object interaction detection (HICO-DET, (Chao et al., 2015)). We outperform baseline VLMs by a large margin (e.g., 20% in HICO-DET mAP, 14% in Cola top-1 accuracy, and 3% on ARO top-1 accuracy). We also achieve competitive results on vision-language tasks such as referring expression comprehension and visual question answering.
RELATED WORKS
2.1 VISION-LANGUAGE MODEL (VLM)
A proliferation of VLMs with remarkable commonsense reasoning abilities have been proposed recently. Among them, Flamingo (Alayrac et al., 2022) employs cross-attention and perceiver sam- pler to attend to visual contexts and enables visual context learning. BLIP2 (Li et al., 2023) uses QFormer to attend to salient visual context for better language generation based on the visual con- text. LLaVA (Liu et al., 2023a) performs image-text alignment first and then conducts instruction finetuning, MiniGPT-4 (Zhu et al., 2023) aligns a frozen visual encoder with L.LM using just one projection layer. mPLUG-Owl (Ye et al., 2023) also involves a two-stage method for aligning image and text. There are also recent papers that push VLMs to the 3D domain (Hong et al., 2023).
Recently, there has been series of works that utilize LLMs for visual segmentation tasks. Specif- ically, VisionLLM (Wang et al., 2023) uses an LLM-based decoder which makes predictions about bounding boxes and polygons given language instructions. DetGPT (Pi al., 2023) can interpret human instruction, reason about the visual scene with common sense knowledge, and finally output the objects of interest. GPT4Rol (Zhang et al., 2023) capable of processing the user instructions that contain interleaved sequences of language and spatial information. LISA (Lai et al., 2023) proposes the sabbeddinn-as-maase paradigm to unlock the segmentation capability. However, the vision-language communication of these VLMs is one-way and one-time, merely using language in- structions to generate segmentations or input segmented regions into the LLMs. KOSMOS-2 (Peng et al., 2023) infuses location tokens after visual entities into the language generation process. How- ever, the communication is purely from the language system to the image for segmentation, while the grounded visual regions are not fed back the language system. Furthermore, none of these VLMs tackle the relations or compositionality in the language inputs. In this paper, we propose CoVLM with a set of communication tokens for composing visual entities and relations and communicating between visual and language systems at each step.
2.2 COMPOSITIONALITY IN VISION AND LANGUAGE
Compositionality is a hallmark of human intelligence and plays an indispensable role in vision and language. Previous works exploring the compositionality in vision and language cover variety of tasks such as visual question answering (Agrawal et al., 2017), generation (Liu et al., 2023b), retrieval (Saito et al., 2023), planning (Ajay et al., 2023) and so on. A of datasets have been proposed for examining the compositionality of vision-language models (Hudson & Manning, 2019; Johnson et al., 2016; Agrawal al., 2017: Krishna et al., 2017: Ma et al., 2023). Specifically, the Attribution, Relation, and Order (ARO) benchmark (Yuksekgonul et al., 2022) is a benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order. Recently, VL-Checklist (Zhao et al., 2023) is framework to evaluate VLM's M's abilities to recognize objects, attributes, and relations. Cola (Ray et al., 2023) analyzes VLMs' compositional ability in detail and proposes text-to-image retrieval benchmark to compose objects with their relations. Evaluation of VLMs on these benchmarks and metrics shows current VLMs struggle with compositionality. Furthermore, set of works find it particularly frustrating for VLMs when relationships are involved (Conwell & Ullman, 2022; Zhao et al., 2023). In this paper, we especially focus on relational compositionality, with the help of the aforementioned datasets and metries.
3

---

Published as a conference paper at ICLR 2024
Vision-Language Communication
Object Detector
ROI Pooling
ROI Pooling
<visual> <previsual> Features
CLIP
CoVLM
<obj> A bread <lobj> <visual> <box> is on the left of <previsual> <prebox> <obj> salad </obj>
Figure 2: Overview of our CoVLM framework. Our vision module consists of CLIP encoder to encode the image, and an object detector which takes in the image together with language inputs generate relevant regions. For language modeling, we insert set of communication tokens into the LLM, which can appear after a visual entity with a <visual> token or after a relationship with <previ sual> token. The last hidden layer of the LLM is then sent the object detector to propose regions relevant to the language inputs so far. This is termed as top down language-to ro-vision communication. Next, in vision-to-language communication, the features of the proposed regions are fed back to LLM via <box> <prebox> token for further language generation.
3 COVLM
Most state-of-the-art Vision Language Models (e.g., LLaVA (Liu al., 2023a), Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023)) take an image and text prompt as inputs and output a text sequence. Several recent VLMs (e.g., VisionLLM (Wang al., 2023), LISA (Lai al., 2023), KOSMOS-2 (Peng et al., 2023)) enable a new ability to output segmentation masks based on the text input. Specifically, KOSMOS-2 generates location token denoting discretized bounding box for each visual entity to ground the visual entity to the image. However, the communication is purely from the LLM to the image for segmentation, while the grounded visual regions are not fed back to the LLM. In addition, the location tokens are generated after the visual entities, thus failing to assist in the process of generating word tokens based on grounded visual entities and relations. In short, the cut-off between the vision module and the LLM deprives previous VLMs of the crucial visual compositional ability. On the other hand, detection networks like Faster RCNN (Ren et al., 2016) generate region proposals and classify the proposals, but can not interact with the language models.
In stark contrast to previous VLMs, our CoVLM stands out with its pioneering integration of detec- tion networks into LLM to enable the seamless interaction between the vision module and the LLM and compositionality
visual entities and relations. As shown in Figure 2, first devise set of special communication tokens for flexibly switching between the visual detection module and the LLM. For LLM, we use pre-trained Pythia model (Biderman et al., 2023) that handle language tokens as inputs and outputs, as well as visual embeddings and special tokens which are mapped the same embedding space as the language tokens, to constitute the LLM representations. The vision module consists of an image encoder that produces features to feed into the LLM and a detec- tion network that proposes region proposals that are relevant to previous language inputs. Top-down language-to-vising communication is achieved by concatenating the last hidden state of the LIM- encoded features to the image embeddings and inputting them into the detection network, which proposes relevant regions conditioned on the LLM representations. Bottom-up vision-to-lams longage communication extracts the features of the relevant regions and concatenates the features back into LLM for further language generation. In this way, the LLM is equipped with visual composionality. We give the details below.
3.1
VISION MODULE
Our vision module consists of parts: an image encoder and detection network.
4

---

Published as a conference paper at ICLR 2024
Image Encoder. In this paper, we use the CLIP ViT-L model (Radford et al., 2021) for encoding the image. We use a linear mapping layer to map the image embeddings to the same embedding space as the Pythia language embedding space. We then append the image embeddings to the beginning of the language sequence.
Detection Network. Our detection network is similar the YOLOX (Ge eta., 2021), The detection network takes as inputs two things: 1) the image embeddings of the whole image (N x N x D, where N is the patch size and D the embedding dim); 2) the last hidden state of the LLM so far (1 x D). The LLM embedding is expanded and concatenated to the same dim as the image embedding, yielding final multi-modal embedding of size N N 2D, and sent to the detection network. The detection network outputs W x N x bounding boxes and N x N confidence scores. After non-maximum suppression, we keep set of bounding boxes as regions of interest (ROIs). To extract the embeddings of one ROI, we extract the features of all patches that are covered by the ROI, and average pool to yield a box embedding of size We choose the cropped image features of bounding boxes with top scores.
3.2 LANGUAGE MODELS
We utilize the pre-trained Pythia model (Biderman et al., 2023) as the backhone of our L.M.M In addition to language tokens, also devise set of special communication tokens to facilitate com- positional vision-language modeling and communication, as is shown in Figure 2. We list the set of tokens below:
<obj>, </obj>: these two tokens enclose of language tokens referring visual entity <visual>: this token is for switching to the vision module after visual entity token V1 is captured by LLM, so the vision module could attend to the visual entity
<box>: this token receives the feedback from the vision module, concatenating the image fea- tures detected 81 back into the LLM
<previsual>: this token for switching the vision module after a relation r to a previous visual entity U1 is detected (and before the visual entity that is in relation to dı is generated).
<prebox>: this token switches back from the vision module after potential regions of t22 are detected, and concatenating the features better generate the language description of U2-
The generation of communication tokens for visual entities and relations enables us to decompose the language sequences into smaller components, where each component connects to the vision module, thus improving compositionality.
3.3 VISION-LANGUAGE COMMUNICATION
The dynamic interaction and communication between the vision module and the language model can be iteratively performed through the special communication tokens introduced above.
Top-Down Language-to- Vi-iiin Communication. Top-down communication is achieved by first generating the Evisual. token <previ surrvvisall. token. After the token is generated, we sum- marize the language information generated so far by taking the last hidden state of the LLM. This information gives the vision module goal or task to attend to, just like the human visual system (Buschman & Miller, 2007). Contingent on the information so far, the vision module then uses the detection network propose several ROIs, and extracts the features of these ROIs.
Bottom-Up Vision-tooLanguage Communication. Bottom-up communication is achieved by gen- erating the <box>> token or <prebox> token. The ROIs generated by the vision module are then fed back into the L.M to assist further language generation. For instance, if the <prebox> contains regions relevant to bread on the left of". the I.LM is then capable of absorbing this information and generating "salad".
3.4 MODEL PRE-TRAINING
Pre-training data. We create a large-scale grounded image-text dataset that consists over 97M image-text pairs from the pre-training data of BLIP-2 (Li et al., 2023). The images are from variety
5

---

Published as a conference paper at ICLR 2024
ARO Accuracy
Cola
HICO-DET
Model
Accuracy Top-1
mAP Non-Rare
Top-1
Top-5
Rare
Full
Vision-language Alignment Models
CLIP (Radford et al., 2021)
6,93
21.12
21.42
FLAVA (Singh et al., 2022)
4.59
12.76
24.76
Vision-language Generative Models
OpenFlamingo3B (Awadalla et al., 2023)
2.55
7.11
18.10
BLIP (Li et al., 2022)
29.78
54.18
41.43
BLIP-2 ViT-I. OPT2.78 (Li et al., 2023)
29.73
54.91
35.71
KOSMOS-2 (Peng et al., 2023)
19.88
43.69
30.48
33.51
17.83
21.26 39.00
CoVLM 1,4B
32.46
55.70
44.29
50.82
35.47
Table 1: Compositional reasoning ability comparison with vision-language alignment models and generative models on three datasets. Visualization results are shown in the Appendix.
of datasets including COCO (Lin al., 2014), CC3M (Sharma al., 2018), CC12M (Changpinyo et al., 2021), Visual Genome (Krishna et al., 2017), SBU (Ordonez et al., 2011) and subset of LAION400M (Schuhmann et al., 2021). Similar KOSMOS-2 (Peng et al., 2023), we apply grounding pipeline to the image-text pair to associate the text spans in the caption to their corre- sponding visual entities the image. The pipeline consists of three steps: First, we use Ground- ingDINO (Liu et al., 2023c) to detect objects and their corresponding textual description. Inspired by KOSMOS-2 (Peng et al., 2023), we then apply spaCy (Honnibal et al., 2020) to expand the grounded words to grounded expressions to enrich their linguistic meaning. Finally, we insert com- munication tokens around the textual description to finalize the grounded data. More details about how create the pre-training dataset be found in the Appendix.
Pre-training settings. We trained two models: CoVLM 1.4B and 2.8B, which use Pythia- 1.4B and 2ythia-2.8B 8B as the LLM respectively. Both of them use CLIP ViT-L/14 (Radford et al., 2021) the image encoder. We load the huggingface checkpoint for these models and fully fine-tune the whole model during pre-training. More details can be found in the Appendix.
EXPERIMENTS
4.1 EVALUATION ON COMPOSITIONAL REASONING TASKS
We aim to probe the model's ability to reason about entities with detailed attributes in an image and also the relationships between two entities. Descriptions of the three datasets evaluating on below, the metric we use for evaluation, and the baselines we used are in the Appendix. All experiments are conducted in zero-shot manner.
4.1.1 ARO
Setup. For our model and other vision-language generative models, we feed the model with the image and text prompt "entity_A A relation". considering the model output as predicted entity For our model, we further insert a <visual> token after entity_A and a <previsual> sual> token after relation encourage the language model to better communicate with the visual branch. For vision- language alignment models, we use all 890 candidates entiry_B _B to build 890 possible captions in the form of "entity relation entiry B" and pick the top- t captions that have the highest similarity score with the image as the 1/to1/ predictions.
Results. In Table 1, our model achieves superior performance, outperforming all other vision- language generative models and alignment models. It indicates that our model has a better ability to understand relations among visual entities in an image, and can better infer one visual entity using information of the presence of other visual entities and their relations. We also notice that the alignment models perform worse in this task. We hypothesize this is because the aligmment models (Yuksekgonul
trained using contrastive learning, which makes them behave like bag-of-wone et al., 2022). This makes them more easily to be misdirected by other objects in the image and produce the wrong prediction, instead of using the relationship to infer the correct

---

Published as a conference paper at ICLR 2024
mAP Non-Rar
Model
Zero-shot
Rare
Full
InteractNet (Gkioxari et al., 2018)
7.16
10.77
9.94
CDN (Zhang et al., 2021)
X
27.39
32.64
31.44
GEN-VLKT (Liao et al., 2022)
X
29.25
35.10
33.75
RLIPy1-ParSe (Yuan et al., 2022)
X
26.85
34.63
32.84
RLIPVv-PPrSSeDA (Yuan et al., 2023)
x
43.23
45.64
45.09
RLIPvI-ParSe (Yuan al., 2022)
15.08
15.50
15.40
RLIPy2-PurSeDA (Yuan al., 2023)
27.97
21.90
23.29
CoVLM 1.4B
50.82
35.47
39.00
Table 3: Comparison with task-specific methods on HICO-DET.
4.1.2
COLA
Setup. For our model and other vision-language generative models, we calculate the perplexity score between caption and all candidate images and choose the image with lower perplexity as the prediction. Specifically, we feed the model with one image and a caption rity_a relation enfity_b". We caleulate the average perplexity of the overall text output. Notably, for our model, we will insert <visual> and <previsual> tokens after entity a and relation, respectively to encourage vision-language communication. For vision-language alignment models, we directly report the results from Cola (Ray et al., 2023).
the form "en-
Results.
In Table 1, our CoVLM significantly outper- forms both alignment and generative methods by a large margin. We attribute the performance the <previsual> token which helps re- trieve the visual information of the entity b for better describing its detailed attributes text form, thus leading to lower complexity for the ground-truth caption. Also, the <visual> to- ken helps to better localize entity allowing the model to better localize the area of entity Jh according to relation. In Table 2, also com- pare our zero-shot results with task ssscc-iccificc methods proposed in Ray et al. (2023) which fine-tunes CLIP (Radford al., 2021) and FLAVA (Singh al., 2022) the training data of Cola. Our method still achieves the best performance, demonstrating the superiority of generalization and robustness of our model.
Model
Acc.
CLIP MM-Pred (Ray al., 2023)
41.42
CLIP + MM-Adapter (Ray et al., 2023)
40.95
FLAVA + MM-Pred (Ray et al., 2023)
39.04
FLAVA + MM-Adapter (Ray et al., 2023)
40.47
CoVLM 1.4B
44.29
Table 2:
Comparisons with task-specific super- vised learning methods on Cola.
4.1.3 HICO-DET
Setup. For our method and other generative models, we predict HOI (subject, verb, object) in two steps: a) recognizing the interaction categories represented by verb, and b) localizing the subject and object. To determine the existence of interaction, we manually build positive and negative phases with verb, i.e. "the person is verb" and "the person is not verb" and calculate their perplexities using the generative model. If the perplexity the positive phase is lower than the negative one, we consider this verb exists in the image. For all detected verb, we feed the model with the image and text prompt "the person verh" predict object and the location of the person and object. Notably, for our CoVLM, we use the inserted <visual> and <previsual verb to predict the locations. Since the output of alignment methods does not contain the object location, ignore these methods the HICO-DET dataset.
tokens after the person and
Results. Table presents the zero-shot results on the HICO-DET test set. Our model significantly outperforms KOSMOS-2 (Peng al., 2023). We attribute the performance improvement to the <previsual> token that forces the model to communicate with the imput image to localize the area of object and predict the text of object. In comparison, KOSMOS-2 only feeds the image

---

Published as a conference paper at ICLR 2024
RefCOCOg test
RefCOCO+
RefCOCO
Model
val
testA
test
val
testA
testB
GRILL (Jin et al., 2023)
47.50
ReCLIP (Subramanian et al., 2022)
59.33
59.01
47.87
50.10
45.10
45.78
46.10
47.07
KOSMOS-2 2 (Peng et al., 2023)
60.57
61.65
45.48
50.73
42.24
52.32
57.42
47.26
CoVLM 1.4B
60.87
61.91
47.62
50.93
44.16
48.19
53.17
43.18
CoVLM 2.8B
61.23
62.33
48.87
52.51
44.71
49.32
53.67
44.49
Table 4: Comparison of referring expression comprchension on three datasets.
information at the beginning of the input, and thus the model may suffer from language prior to predicting a wrong object. In Table 3, we also compare our zero-shot results with the task-specific supervised learning methods. Our model achieves comparable results the Non-Rare and Full sets. Notably, our zero-shot result exceeds all supervised learning methods in the Rare set, demonstrating the generalization ability of our model.
4.2 EVALUATION ON VISION-LANGUAGE TASKS
addition to compositional reasoning tasks, we also evaluate object localization and complex vision-language understanding abilities of our model. All experiments are conducted in zero-shot manner.
4.2.1 REFERRING EXPRESSION COMPREHENSION
Setup. This task requires a model to locate the bounding box of object specified by language description. We follow KOSMOS-2 (Peng et al., 2023) to use three well-established benchmarks namely RefCOCOg (Mao et al., 2016), RefCOCO+ (Yu et al., 2016) and RefCOCO (Yu et al., 2016). All these datasets use images from COCO (Lin et al., 2014). Both RefCOCO and RefCOCO+ are annotated by a two-player game, where the spatial relations are excluded in ReiCOCO+. The RefCOCOg contains longer referring expressions and spatial relations.
For our COVLM, we feed the model with <visual> will generate multiple bounding boxes with their confidence scores. Instead of choosing bounding box with the highest score as a prediction, we use <previsual> to further measure the alignment between the box and expression. Specifically, we select the bounding box with the highest confidence score and subsequently choose additional bounding boxes from the remaining set whose confidence scores exceed predefined threshold (0.5 times the highest score in our case) as candidates. For each candidate, we feed the model with <previse www.sssaxxprbbx-ee>.. job j>expression<//bb box region into <prebox>, Then we calculate the perplexity of the expression for this bounding box candidate and choose the one with the lowest perplexity as final prediction.
and the
and put the image feature of the bounding
Results. In Table 4, our CoVLM 2.8B variant performs the best on both val and test sets of Ref- COCOg and RefCOCO+, demonstrating its superior localization ability. On RefCOCO, we achieve comparable performance with KOSMOS-2. Note that KOSMOS-2 has been instruction fine-tuned on the data with the same form as the referring expression task, while our CoVLM directly trans- ferred to this new task without any form of instruction fine-tuning.
4.2.2 VISUAL QUESTION ANSWERING
Setup. This task requires model to answer questions about an image. Following BLIP-2 (Li al., 2023), evaluate VQAv2 (Goyal al., 2017) dataset and report the zero-shot accuracy on the test-dev set. We use the prompt "Question: {question] Short Answer:".
Results. In Figure 3, CoVLM 2.8B model achieves better performance compared with MetaLM (Hao et al., 2022), VLKD (Dai et al., 2022), OpenFlamingo38 (Awadalla et al., 2023), and KOSMOS-2 (Peng al., 2023), and has small margin compared with Flamingo3B (Alayrac et al., 2022) and BLIP-2 ViT-L OPT Li et al. (2023). We hypothesize the accuracy margin may stem from the generative model generating diverse answers that align conceptually with ground
8

---

Published as a conference paper at ICLR 2024
53
50
48,0
Accuracy (%6
544.
45 65..552
41
Setting
<previsual>
<visual>>
<prehox>
<box>
ARO Top-1
Cola
X
29:60
38.
2
30.08
34,76
/
3
32.26
36.19
VILL 027275
VIKD
KOSMO0S-3
4
32.46
44.29
OpenlTaninge10
COVLK 1:48
Slaningsti
CWVE 23
Table 5: Ablation study of different sets of available com- munication tokens.
Figure 3: VQAv2 test-dev results.
truth, yet may not exactly match the annotation, affecting the evaluation. To get better insight into how well our model performs on VQAv2, we conduct a round of human evaluation for our CoVLM 2.8B model and BLIP-2 ViT-L 277 model a randomly selected subset with 1000 samples. The human evaluation accuracy for our model is 57.11 1 while the accuracy for BLIP-2 is 56.62, suggesting that the performance gap between model and BLIP-2 is negligible.
4.3 ABLATION STUDY
Compared to other VLMSs our COVLM introduces a novel set of communication tokens to improve the compositional reasoning ability. To evaluate the effectiveness of each type of communication token and the effectiveness of the design of bidirectional communication of the visual module and language module, we create the following four settings: 1) No communication token at all. 2) No <previsual>> and <prebox>, so there are no communication tokens to compose a relationship. 3) No <prebox> and <box>, so the communication only happens from the language module to the vision module. 4) All communication tokens are presented. For each setting, we pre-train our CoVLM 1.4B model under the same setting and evaluate on ARO and Cola benchmarks. Table shows the evaluation results for these four settings. We summarize the insights as follows:
1. With no communication tokens, our model's performance on compositional reasoning tasks is very similar to BLIP-2. This is reasonable because we share the same pre-training data with BLIP- 2, thus also inheriting a similar performance on downstream tasks.
2. Generate tvisual>> >/<bo>>> after the object does not help compositional reasoning. It is rea- sonable because adding extra information after the object description won't 't help the generation of that object description itself as this is an auto-regressive generation process.
Not putting <prebox>/<box back into the generated sequence will hurt the compositional reasoning ability for complex object description. We can find this insight in the last two rows of the ablation study table. ARO Top-1 accuracy does not hurt much if we do not put <prebox>/<box back, while the performance for Cola will drop significantly. This is because in ARO, the predicted object usually a simple phrase without any attribute, such as "horse" and "car". During train- ing, the model can learn to bind this kind of simple concept with the sprevisual><<visus tokens, so merely generating <previsuse wttpiissaa///visaaaaaa compositional reasoning ability ARO benchmark. However, the object description in Cola is much more complex, such as "yellow vehicle" and "standing man", which requires a close inspec- tion of the visual feature of the object. In this case, <prebox><<box> tokens which contain more fine-grained information about the visual entity can play an important role in assisting LLM to focus on these complex objects and generate more faithful and related tokens thereafter.
is adequate for enhancing the model's
5 CONCLUSION
In this paper we propose CoVLM, which can guide the LLM to explicitly compose visual entities and relationships among the text, and dynamically communicate with the detection networks to achieve vision-language communicative decoding. We outperform previous VLMs by a large margin on compositional reasoning benchmarks (e.g., ~ 20% in HICO-DET mAP, ~ 14% in Cola top- 1 accuracy, and 3% in ARO top-1 accuracy). We also achieve competitive performances on referring expression comprehension and visual question answering. However, we do not yet address object-attribute and spatial event compositionally much, which are crucial future directions.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We are grateful to the anonymous reviewers for their valuable feedback. This work was partially funded by grants from Google, Amazon, and Adobe. We also extend our thanks to AiMOS for supplying the computational resources necessary for this project.
REFERENCES
Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-vqa: A compositional split of the visual question answering (vqa) v1.0 dataset, 2017.
Anurag Ajay, Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenen- baum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a vi- sual language model for few-shot learning. ArXiv, abs/2204. 14199, 2022. URL https: //api. www.ntemmintteeeeee www.tiicccchca....... wwtpaa/pgg/ooooooooooooo 248476411.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 01390, 2023.
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Buschman and Earl K. Miller. Top-down versus bottom-up control of attention in the prefrontal and posterior parietal cortices. Science, 315:1860 1862, 2007. URL https: //api. semant
144021759.
Paola Cascante-Bonine Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gül Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, and Leonid Karlinsky. Going beyond nouns with vision & language models using synthetic data, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: A benchmark for recog- nizing human-object interactions in images. In Proceedings the IEEE International Conference on Computer Vision (ICCV), December 2015.
Noam Chomsky. Bibliography, pp. 115-120. De Gruyter Mouton, Berlin, Boston, 1957. ISBN 9783112366009 doi:
1233999999 URL https: //ooi. org/
1515/978311231600
Noam Chomsky. Bibliography, pp. 253-264. The MIT Press, 50 edition, 1965. ISBN
9780262527408. URL hwtpthtpph/wwet iastor.org.o / ct 1kkk11kk111 10.
Colin Conwell and Tomer David Ullman. Testing relational understanding in text-guided image generation. ArXiv, 0bs02208.00005,, 2022. URL https:///////ccc.ttt :ttpss///ppl /api.. httpappp.aaaaaaee semanticschol oorgg//rraoroooooooooooooooo
CorpusTD::51224300
10

---

Published as a conference paper at ICLR 2024
Wenliang Dai, Lu Hou, Lifeng Shang. Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation on clip via vision- saion-language knowledge distillation, arXiv preprint arXiv:2203.063383 2022.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Rameswar Panda, Roei Herzig, Eli Schwartz, Donghyun Kim, Raja Giryes, Rogério Schmidt Feris, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision language models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2657-2668, 2022. URL https:
//api. semant cacholar. httpg//oruusuui : 253734406.
Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim. Paola
Amit Alfassy, Rameswar Panda, Raja Giryes, Rogério Schmidt Feris, Shimon Ullman, and Leonid Karlinsky. Dense and aligned captions (dac) promote compositional reasoning in vl models. ArXiv, abs/2305.19595, 2023. URL https: /ttp://appii.. semant CorpusID:25898738
.org/
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021, 2021.
Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human- object interactions. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 8359-8367, 2018.
Yash Goyal, Tejas Khot, Douglas Summers-Stapy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Yanu Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general- purpose interfaces. arXiv preprint arXiv:2206.06336, 2022.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models, 2023.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, Adriane Boyd, et al. spacy: Industrial- strength natural language processing in python. 2020.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for and compositional question answering, 2019.
visual reasoning
Theo M.V. Janssen and Barbara H. Partee. Chapter 7 compositionality. In Johan van
Benthem and Alice ter Meulen (eds.), Handbook of Logic and Language, pp. 417-473. doi: https://doi.or//00.10t URL https: htap:s://ww..../ sci www.seiieenceietttccommm wttpe//ttttttccccceeeeeee...c
North-Holland, Amsterdam, 1997. ISBN 978-0-444-8171443
B978-0444811744-35000 atticlle/pii/9978444431
Woojeong Jin, Subhabrata Mukherjee, Yu Cheng, Yelong Shen, Weizhu Chen, Ahmed Hassan Awadallah, Damien Jose, and Xiang Ren. Grill: Grounded vision-language pre-training via align- ing text and image regions. arXiv preprint arXiv:2305.1 14676, 2023.
Justin Johnson, Bharath Hariharan, Laurens der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual
reasoning, 2016.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
11

---

Published as a conference paper at ICLR 2024
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imann pre-
training for unified vision-language understanding and generation. International Conference on Machine Learning, pp. 12888-12900 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre -reitraining with frozen image encoders and large language models, 2023.
Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang. Xiaobo Li, and Si Liu. Gen-vikt: Simplify asso- ciation and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20123-20132, 2022.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Nan Liu, Yilun Du, Shuang Li, Joshua B. Tenenbaum, and Antonio Torralba. Unsupervised compo- sitional concepts discovery with text-to-image generative models, 2023b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023c.
Ilya Loshchilov and Frank Hutter. Decoupled arXiv:1711.05100 2017.
weight decay regularization. arXiv preprint
Zixian Ma, Jerry Hong. Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna.
Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10910-10921, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 11-20, 2016.
Richard Montague. Universal grammar. Theoria, 36(3):373-399 1970. doi: 1970.tb00434.x.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023.
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, and Tong Zhang. Detgpt: Detect what you need via reasoning, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A. Plummer, Ranjay Krishna, and Kate Saenko. Cola: A benchmark for compositional text-to-image retrieval, 2023.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, 2016.
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen- Yu Lee, Kate Saenko, and Tomas Pfister. Pic2word: Mapping pictures words for zero-shot composed image retrieval, 2023.
12

---

Published as a conference paper at ICLR 2024
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernmeed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 2556-2565, 2018.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15638-15650, 2022.
Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. arXiv preprint arXivv2204.0599) 2022.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model also an open-ended decoder for vision-centric tasks, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Eision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October 11-14, 2016, Proceedings, Purt II 14, pp. 69-85. Springer, 2016.
Hangjee Yuan, Jianwen Jiang, Samuel Albanie, Tao Feng, Ziyuan Huang, Dong Ni, and Mingqian Tang. Rlip: Relational language-image pre-training for human-object interaction detection. Ad- vances in Neural Information Processing Systems, 35:37416-37431, 2022.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Samuel Albanie, Yining Pan. Tao Feng, Jianwen Jiang, Dong Ni, Yingya Zhang, and Deli Zhao. Rlipv2: Fast scaling of relational language-image pre- training, arXiv preprint arXiv:2308 09351, 2023.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- languae models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations, 2022.
Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaoho Li. Mining the benefits of two-stage and one-stage hoi detection. Advances in Neural Information Processing Systems, 740192999-7220, 2021.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interess 2023.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 PSEUDO ALGORITHM FOR COMMUNICATIVE DECODING
Algorithm Communicative Decoding
Input: 224x224 RGB image Faw and text prompt Praw Output: VLM generated tokens
I: I t vision woonnenncdde(raa
2: k- text eeweeddingPPrert.
3: 5 <- empty list
4: S + Append(S, !)
5: S Append(S, P)
6: repeat
7:
X + Auto-regressively predicted next token of S
8:
S + Append(S,
9:
if X is <previsual> then
10:
H < last hidden state of <previsual> ROIs detector(I. H) for each ROI ROIs do
12:
IROI - Crop(I, <prebox> + MeanPoolin((noo)
14:
S + Ap <prebox>)
16:
end for
else if X is <visual> then
18:
S + Delete the latest <previsual> and all its corresponding <prebox> in S H < last hidden state of <visual>
19:
20:
ROIs +- detector(I, H) for each ROI ROIs do
21:
22:
IROI Crop(I, ROI)
23:
<box> + MeanPooling(/noi)
24:
S Append(S, <box>)
25:
end for
end if
27:
until X is EOS token
Algorithm describes how our communicative decoding works. Our model takes RGB image with a resolution of 224x224 and text prompt as input. The RGB image will first encoded to image embedding by the vision encoder and the text prompt will be encoded by the LLM's text embedding. Then, we put the image embedding in front of the encoded prompt to form a multimodal prompt, and feed this multimodal prompt into LLM for auto-regressive generation.
The generation process are mostly the same as the LLM's original auto-regressive next token predic- tion process, except that we include iterative and automatic insertion of communication tokens to facilitate the compositional reasoning ability. <previ sual> is automatically generated by LLM via next token prediction, and arbitrary number of ROIs be detected through the object detector module. We set the confidence score threshold to be 0.05 and retain all ROIs that has a confidence score higher than 0.05. For each ROI, we crop that region from image embedding, and perform mean pooling to get token, namely <prebox>, that contains the visual information of the ROL. All <prebox>s are automatically inserted into the token sequences so that the future generation process can use the information of these tokens. The generation of <visual> and <box> are order to prevent the localization information leak
similar to <previsuall and <prebox>, induced by the latest <previ sual> and <prebox>, we delete the latest <previ sual> and <prebox> before we use the last hidden state of the new <visual> to detect objects.
A.2 PRE-TRAINING DATASET
Similar to KOSMOS-2 (Peng et al., 2023), we adopt a pipeline making use of of-the-box open vocabulary detector create grounded pre-training dataset. It conssts of three steps:
14

---

Published as a conference paper at ICLR 2024
Step-1:
Generating bounding-box-worr pairs. We use GroundingDINO (Liu et al., 2023c) to detect objects in the image and link the bounding box of the object to words in the text. We keep bounding boxes whose highest similarities are higher than 0.35 and extract the words whose similar- ities are higher than 0.25 as the words that correspond to a bounding box. Non-maximum suppres- sion algorithm is applied eliminate bounding boxes that have high overlap with other bounding boxes linked to the same word.
Step-2: Expanding grounded words to grounded expressions. In practice, we observe that GroundingDINO often fail to link the whole referring expressions to an object in the image. For ex- ample, for the expression "man with a hat on his head", GroundingDINO will only link "man" the person in the image, but not the whole expression. This will limit the model's 's ability to understand complicated expressions. Inspired by KOSMOS-2 (Peng et al., 2023), we apply spaCy (Honnibal et al., 2020) to obtain each word's 's dependency relation in the sentence, and expand grounded word to a grounded expression by recursively traversing the dependency tree of that word and concatenate eligible children words based on the linguistic rules.
Step-3: Assigning bounding boxes to the special communication tokens. Given the expressions and their associated bounding boxes in a grounded image-text pair, we can now insert the special communication tokens into the text and assign the bounding boxes to them. We follow KOSMOS- 2 (Peng et al., 2023) to enclose the expression in tokens around them. For a given expression with a single bounding box, the resulted input se- quence for that expression either in the form of "<obj jexporbssxsssssnn...... //bj>vv//bvoiioooooo sual boual><boxx or seprevisual><ee <prebxx joppeexresssion///////eoo obj>" depending on the position of the ex- pression in the sentence. If an expression is associated with multiple bounding boxes, we add multi- ple <prebox> or <box>. If it is the first expression in the sentence, we use the form with a trailing <visual> token. Otherwise, we randomly select one from these two available forms.
pair, and then add communication
A.3 PRE-TRAINING DETAILS
Apart from the grounded image-text pair dataset we created, also use The Pile (Gao et al., 2020) as part of our pre-training dataset. The total pre-training loss consists of the language modeling loss and the detection loss, with loss weight of 0.025 for the detection loss. We pre-train for 20k steps and use a batch size of 2,304 for grounded image-text data and a batch size of 2,304 for The Pile data. AdamW W (Loshchilov & Hutter, 2017) optimizer is employed with learning rate of 1.0e and B = (0.9, 0. 0.999). We do not apply weight decay to the weights of LLM and CLIP, but apply weight decay of 0.05 to the detection network.
A.4 COMPOSITIONAL EVALUATION DATASET
Datasets and Metrics. We conduct experiments ARO (Yuksekgonul et al., 2022), Cola (Ray et al., 2023), and HICO-DET (Chao et al., 2015) datasets.
ARO contains 23,937 testing images. One entity pair in an image is annotated as a tuple (en- tity_A, A, relation, entity B). Given entiry A and relation, the model is required to predict entity B out of all candidate objects. We use top-1 and top-5 accuracy as evaluation metrics.
Cola contains 420 testing images, where each image is paired with a caption describing the relation between two entities similar to ARO. The entity in Cola is described with more attribute details (e.g., texture, color, and size) which require the model to perform more fine-grained object recognition. We conduct experiments on the multi-object part of this dataset, where two images with similar entities and their corresponding captions are paired as testing sample. The model is required to correctly match the corresponding caption for both two images. We report the accuracy following Cola (Ray et al., 2023). HICO-DET contains 9,658 testing images, with 600 HOI categories constructed by 80 object categories and 117 verb classes. Each instance of human-object interaction is represented as a triplet, denoted as (subject, verb, object), accompanied by their respective bounding boxes. The model is required to not only recognize the HOI categories but also localize the subject and object. Following Chao et al. (2015), we report the mean AP (mAP) on 3 splits, namely a) Rare: 138 HOI categories with less than 10 training instances, b) Non-Rare. the remaining 462 HOI categories, and e) Full: all 600 HOI categories.
15

---

Published as a conference paper at ICLR 2024
ARO
Cola
HICO-DET
Compositional Entity Prediction
Compositional Text-to-image Retrieval
Human-Object Interaction
decorative blue colorful painted large oblong vose black round table
the horse eating grass
<person, straddle, horse>
Figure Explanation of three evaluated compositional reasoning tasks.
Baselines. We mainly compare our CoVLM with two types of methods, namely the vision-language alignment model (i.e., CLIP (Radford et al., 2021), FLAVA (Singh et al., 2022)) and the vision- language generative model (i.e. OpenFlamingo (Awadalla et al., 2023), BLIP-2 (Li et al., 2023) and KOSMOS-2 2 (Peng et al., 2023)). The vision-language alignment models learn vision encoder and a language encoder, which pull close the features of paired image and while pushing away the unpaired one. The vision-language generative model takes input the image and text and auto- regressively generates sequence of text.
A.5
VISULIZATION OF COMPOSITIONAL REASONING RESULTS
We show the visualization results for our model, BLIP-2 2 and KOSMOS-2 on the three compositional reasoning tasks.
Figure shows the qualitative results for ARO. Compared to BLIP-2 and KOSMOS-2, we can rank ground truth object higher thanks to the <previsual> token to localize the object.
right of the fence
the street is to the left of [hee we
Ronblop in Condidone Outputr
Ronking Outpoti
BLIP2
BLIP2 3
Kosmos-2 Ours
Kosmos-2 17
left of [th pot
the gym is to the right of [te trees
Ranking in Condidaur Outputs
Runking Candidate dutputs
BLIP2 53
Kasmos-2 Ours
Kosmos-2 43
Ours
steam
the bothtub is near [the wallpaper)
Pankirp in Condiduse Ovoputs
Roskingg Condidote Outpuci
BLP2
62
Kosmos-2
Kosmos-2 23
Ours
Figure 5: Qualitative results on ARO.
Figure shows the results for Cola. The presence the <prebox> token can encode the visual feature of the ROI, helping the model better understand the attribute of the objects due to its zoom-in effect. Also, relationship can be explicitly used by <previ sual> token better infer the visual entity.
Figure shows the comparison of our model and KOSMOS-2 on the HICO-DET task. In general our model gives more accurate bounding box for object localization. Also, when there are two identical objects in the image, i.e., the bottom-right example in Figure 7, our model can take advantage of the <previsuall token which guide model pay attention to the region of the correct object (the bench that the person is lying on), instead the wrong object (the bench that behind the person).
16

---

Published as a conference paper at ICLR 2024
white bow' small
red apple green opple top large round plate
sige block onrow on round sign
small bowi on white round plate
BUP2
DLIP2
Kosmop-2
Koumos-2 2
Nosmos-2
Ours
GT
baby groy young stonding elephont neor stantlng gray lorge ndult baby niephant baby kneeling elephant the standing wolking large elephant
sitting womon the left of standing
blue pointed
mon
oblong vuse round black vase on blue toble
young
altting man
SUP2
SLIFZ
BLIF2
Kasmos-2
Koomas-2
Kasmos-2
Ours GT
Ows
G7
Figure 6: Qualitative results on Cola.
Kosmos-
Ours biows
GT
Koumas-2 person
GT
person
cake
eats at
dining toble
Nosmos-2
Ours washes
G7
Kosmos-2
Ours
G7
person
lies
bench
Figure Qualitative results on HICO-DET.
Figure 8 illustrates the performance consistency of our model across diverse and ambiguous sce- narios. Specifically, in instances where the expression inputs exhibit ambiguity such as having multiple boxes corresponding to the same text our model generates all potential bounding boxes with scores surpassing the threshold.
Figure 9 depicts the functioning of the model in the image captioning task. Notably, when the model produces nouns with modifiers, such as "an man". red fire hydrant" and "a glass of champagne", it considers them as integral units and automatically generates "<obj>>n old man</obj>>> "<obj>A red fire hydrant</ bydrant</obja"" "<obj>a glass of champagne</ode / /obj>" in- stead of parsing the objects individually, for instance, "an old <obj>man</obj>" and "A red <ob jo fire hydrant</obj>""
Figure 10 delineates instances of model failures. Various texts were input under the same picture to obtain the output. As depicted in (a), our model proficiently recognizes normal-sized objects. However, challenges arise when dealing with minuscule depicted the picture, making it difficult for the model to yield the desired results. Moreover, as illustrated in (b), the model successfully identifies aligned objects of the same kind the picture. While our model adeptly produces corresponding bounding boxes when multiple objects of the same kind neatly arranged, the scenario becomes more challenging crowded scenes when the object arrangement is irregular, often resulting in the model's inability to output all the bounding boxes.
blurred objects, such the "spoon"
17

---

Published as a conference paper at ICLR 2024
Referring Expression Comprehension
Prompt Template <lmage>-obji-ee (iange--oobpeeett///ooo---aaaa
(al
(b)
(a)
Origin
[b) people
[c] text = the woman holding purple umbrella
(4)
test = the white honse distance
Figure 8: Qualitative results on Referring Expression Comprehension with various boxes output.
Image Coptioning
Task Prompt Template
<image>
Caption:
Caption:
old man with mustrche and suit holding phone in his hands
man in tuoedo holds glass of champagne his hend while looking the
cobjaan </ebp--vnuual-chhes mustache jobi-wvla.ab-ooo
sobj-
</ong--veuutbbbeee cobjo
sobjaa
<dabp-ovsuall thach <nhjo glass champagne
</obp-cvisullo cbose holding cobj phone
</abp-o/suall-bore in coejo his hand
ciebp--visuabbbboo in cobj> his hands
c/oby-ae Poby--vvuuslb-bee while looking the camera.
Caption:
Caption: red fire hydrant neat to red pole.
man riding a snowboard down snow covered slape.
Raw output:
<obj- man c/cbj--visuab-ooe riding cobj> <debp-ovsualb-leess
cobj- to cotje a red polec http///ooootsuubbbbbo...
slope.
Figure Qualitative results on Image Caption.
Failure Coses
Prompt Template simggoxaolpttoekkaaapeeaaaa
"people"
twel
(4) Fallures small objects
endironmntt
Figure 10: Qualitative results on failure cases.
18


Reviews:
[{"review_id": "scciMc6z1e", "reviewer": "Reviewer_onDF", "summary": "This paper introduces Compositional VLM, a new Large Vision-language Model that can compose visually grounded concepts and relationships within a given text input. It achieves this by employing a set of special tokens to manipulate the interactions between the LLM and a visual object detection network. This bi-directional communication of vision-to-language and language-to-vison occurs through multiple iterations as an output sentence is generated. The proposed model outperformed prior works across various compositional reasoning tasks and vision-language tasks.", "questions": "-\tWhy did you opt to create your own dataset of similar scale for pretraining, rather than utilizing the data from KOSMOS-2?\n-\tCould you provide further analyses of the contributions of these communication tokens?\n-\tOn the task REFERRING EXPRESSION COMPREHENSION: the proposed model outperforms the KOSMOS-2 model, but with a small gap. While the grounding of vision-language in KOSMOS-2 was reasonably good compared to the proposed model, it exhibited significantly poorer performance in the previous three tasks. Can you provide insights into these differences?\n\n-\tCan you clarify the large margins between KOSMOS-2 and the proposed method in Table 1? What would be the main contributor to this significant performance leap?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The object-centric approach presented in this paper is interesting as I believe visual objects and words are of the same level of abstraction of representation. I also appreciate the idea of multi-step interactions between an LLM and a visual object detection network, much like older multi-step reasoning techniques in the joint vision-language understanding literature.\n\n- The experiments demonstrate significant improvements over the existing works across a number of downstream tasks and benchmarks.", "weaknesses": "- The presentation of the method is brief, making it hard to understand. It is not clear how the communication tokens are generated. I recommend providing a pseudo algorithm to describe the method and making the input/output for each step more readable.\n\n- The proposed pre-training procedure seems to incorporate a lot of engineering tricks and data processing techniques from KOSMOS-2, without adequate attribution to previous works. The paper also lacks justifications for the selection of sub-components in the proposed method, making it hard to interpret the results. \n\n- Pretrained data and downstream tasks (i.e., RefCOCO/RefCOCO+, VQA v2) share the same pool of visual data taken from COCO and Visual Genome. This raises concerns about the validity of the reported performance on these downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pfnqWjkjoV", "reviewer": "Reviewer_Aw71", "summary": "The paper presents Compositional VLM, a vision-language architecture allowing the language model (LLM) to communicate with the vision encoder(s) to generate its output iteratively. Several new tokens, such as <box>, <prebox>, <visual>, and <previsual>, are introduced to facilitate this communication between the two components working on different modalities. The goal is to improve the ability of VLMs to capture the compositional structure found in vision-language tasks. The model is pre-trained on a large corpus of grounded image-text pairs from sources such as COCO, CC3M, CC12M, Visual Genome (Krishna et al., 2017), SBU, and LAION400M. The model is thoroughly evaluated on many tasks, such as compositional visual understanding, referring expression comprehension, VQA, and human-object interaction. The quantitative performance shows the benefit of improved communication between vision-language modules and outperforms considered VLM baselines on public benchmarks.", "questions": "My major concerns are the generalization capabilities and the lack of qualitative results and ablation studies, detailed in the weaknesses section.\n\n--- Post-rebuttal update ---\nI am raising my score after the authors' response.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The paper addresses an important topic for VLMs. Imbuing VLMs with compositional visual understanding is an important direction of research, and the proposed approach is an interesting mechanism to achieve it.\n+ The bidirectional communication between the vision and language modules is interesting and is an important component that needs to be introduced and explored in VLM research. Existing works do not have iterative communication, and relying only on global features for alignment/pre-training is insufficient to exhibit compositional understanding.\n+ The quantitative performance is strong and shows consistent gains over the considered baseline VLMs on several tasks and benchmarks.\n+ The paper is well-written and the idea is simple and presented intuitively, making it easy to follow and understand.", "weaknesses": "- My primary concern is about the pre-training data and how to interpret the results. The pre-training dataset consists of text-image pairs from COCO, CC3M, CC12M, Visual Genome (Krishna et al., 2017), SBU, and LAION400M. The evaluation datasets share considerable overlap with the pre-training data. For example, COLA is composed of data from GQA (derived from Visual Genome), CLEVR, and LVIS (also includes COCO). Similarly, ARO is based on Visual Genome, and HICO-DET is based on Flickr (with objects limited to those from COCO). I understand that the training data is essentially the same as BLIP-2 and that it is common for VLMs to be trained on these datasets. The question does remain: given this overlap, how well does the model generalize? The generalization can be quantified based on two factors - tasks and domains. From the task perspective, the generalization of the proposed approach seems to be limited, going by the performance of VQA. There are no convincing arguments for the generalization beyond training data. How about the performance on PACO[1]? It does not have a **significant** overlap with the pre-training datasets and should provide some measure of generalization.\n- There are very few qualitative results presented. There are 4 examples in the supplementary, but they are very limited beyond that. By reading the paper, it is hard to understand when/where/why the model fails. For example, is there a reason why the model has a higher performance on the \"rare\" class from HICO-DET? It would be good to understand the success and failure modes and have a discussion on the approach's qualitative performance as opposed to the purely quantitative take.\n- There are no ablations presented. What does the choice of encoders/pre-training data/tokens have on the model?\n- On that note, how sensitive is the model to the template used for prompting?\n\nReferences:\n[1] Ramanathan, Vignesh, et al. \"Paco: Parts and attributes of common objects.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O7kuEjKGnv", "reviewer": "Reviewer_fA94", "summary": "Current vision-language models (VLMs) lack this ability due to their \"bag-of-words\" approach and inability to accurately represent visual entities and their relationships. The proposed Compositional VLM addresses this by guiding the model to explicitly compose visual entities and relationships and enabling dynamic communication between the visual detection system and the language system. This is achieved through the introduction of communication tokens, which guide the detection network to propose relevant visual regions based on the generated sentence. These regions are then integrated into the language generation process. This iterative communication between vision and language continues until a complete sentence is generated. This approach effectively bridges the gap between visual perception and language models, significantly outperforming other VLMs in compositional reasoning benchmarks. It also performs well in traditional vision-language tasks like referring expression comprehension and visual question answering.", "questions": "1. When integrating the object detection model within the auto-regressive language model, what would be the time complexity for each individual inference? Would it be very slow?\n\n2. Given that the proposed Compositional VLM necessitates text parsing even during inference, how resilient is the model to inaccuracies or ambiguities in token placement? For instance, the phrase \"a black dog\" could be parsed as \"a <obj>black dog<\\obj>\" or \"a black <obj>dog<\\obj>\". How does the model handle such variations?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. **Dataset Collection**: The paper collect new object level bounding box to image-text pairs ensures that the data is not just vast but also well-curated, enhancing the model's training and performance.\n\n2. **Reasonable Model Design**: The introduction of the Compositional VLM, with the token design framwork and the interation between visual detector LLM, displays a systematic approach towards bridging the gap between visual entities and their textual descriptions.\n\n3. **Comprehensive Experiments**: With the grounded image-text pairs, the paper offers a detailed experimental setup, validating the model across various compositional reasoning benchmarks.\n\n3. **Strong Performance**: The Compositional VLM showcases impressive results, especially when compared to previous vision-language models.", "weaknesses": "1. **Fundamental Oversimplification of Compositionality:** The Compositional VLM framework, though innovative, may not truly capture the essence of disentangling objects and their relationships within images. Instead of delving deep into the inherent complexities of this challenge, the method leans towards gathering object-text-image paired datasets and reinforcing their connections. This approach, while seemingly effective, might only be a surface-level solution rather than addressing the root of the compositional problem.\n\n2. **Scalability Concerns**: The model's heavy reliance on precise associations between text captions and visual entities raises questions about its scalability. Can it consistently perform well in diverse or ambiguous scenarios, or will it be constrained by the specificity of its training data?\n\n3. **Rigid Token Implementation**: The manual nature of token positioning (with <obj> around the object and <box> after them), suggests a lack of flexibility in the model. This rigidity could hamper the model's adaptability, especially when faced with varied or unforeseen testing scenarios. For example, if we have a \"a man between two dog\", how to deal with the <box> for multiple instance of dog.\n\n4. **Operational Inefficiency**: The necessity for pre-parsing sentences and manually inserting tokens, even during testing, indicates potential operational bottlenecks. This could impede real-time applications and demands additional preprocessing steps, detracting from the model's overall efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)