CoVLM: Composing Visual Entities and Relationships in Large Language
  Models Via Communicative Decoding
Authors: name: Junyan Li, name: Delin Chen, name: Yining Hong, name: Zhenfang Chen, name: Peihao Chen, name: Yikang Shen, name: Chuang Gan

A remarkable ability of human beings resides in compositional reasoning,
i.e., the capacity to make "infinite use of finite means". However, current
large vision-language foundation models (VLMs) fall short of such compositional
abilities due to their "bag-of-words" behaviors and inability to construct
words that correctly represent visual entities and the relations among the
entities. To this end, we propose CoVLM, which can guide the LLM to explicitly
compose visual entities and relationships among the text and dynamically
communicate with the vision encoder and detection network to achieve
vision-language communicative decoding. Specifically, we first devise a set of
novel communication tokens for the LLM, for dynamic communication between the
visual detection system and the language system. A communication token is
generated by the LLM following a visual entity or a relation, to inform the
detection network to propose regions that are relevant to the sentence
generated so far. The proposed regions-of-interests (ROIs) are then fed back
into the LLM for better language generation contingent on the relevant regions.
The LLM is thus able to compose the visual entities and relationships through
the communication tokens. The vision-to-language and language-to-vision
communication are iteratively performed until the entire sentence is generated.
Our framework seamlessly bridges the gap between visual perception and LLMs and
outperforms previous VLMs by a large margin on compositional reasoning
benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on
ARO top-1 accuracy). We also achieve state-of-the-art performances on
traditional vision-language tasks such as referring expression comprehension
and visual question answering.