OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
Authors: Tao Zhang
Xiangtai Li
Hao Fei
Haobo Yuan
Shengqiong Wu
Shunping Ji
Chen Change Loy
Shuicheng YAN
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-modal modeling
universal model
Primary area: machine_vision
TL;DR: A uniform model for image-level, object-level, and pixel-level reasoning and understanding.

Abstract: Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding
OMG-LLaVA
Tao Zhang', Xiangtai Li21 1, Hao Fei?, Haobo Yuan², Shenggiong Wu?,
Shunping Ji', Chen Change Loy², Shuicheng Yan
Wuhan University Skywork AI S-Lab, NTU "Bytedance
Project page: https: / /Ixtgh. gi io/ coprojjcct/////oooooooooooocccce pemoojecc/mm.... llava
Email: xiangtai94 @ gmail.com and zhang tao@whu.edu.co
Abstract
Current universal segmentation methods demonstrate strong capabilities in pixel- level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically,
universal segmentation method the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's 's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaV achieves image-level, object-level, and pixel-level reasoning and understanding in single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.
1 Introduction
With the development of transformer models [94; 6: 93; 40; 71; 92; 64; 126; 49; 87; 10; 19; 58), recent works in both natural language processing (NLP) and computer vision raise one common trend: adopting one unified model solve multiple tasks. For example, large language models (LLMs) [93; 40; 92] adopt scale-up models to solve multiple NLP tasks and achieve better results than previous expert models. In vision, have also seen a similar trend [19; 58; 100; 99; 47; 112), adopting one model to solve multiple tasks or sub-tasks, including detection, segmentation, video analysis, low-level vision, pose estimations, and more tasks. Different methods adopt different transformer designs, including
learning [99; 100], unified decoder [19; 58], and unified tokenizer [86; 16: 58]. In summary, benefiting from the scalability and flexibility of the transformer, adopting one model for all tasks has made great progress [19; 71; 72: 70; 126; 88; 87].
Work done when Tao Zhang is an intern at Skywork Al. 1; Project leader. Corresponding author: Xiangtai Li and Shumping Ji.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Convesation Genernties
Melepic Trorge Desciition
Imegeleee Conseratiion
chendit on the parc pidea pasille
Intructie:: allce the namber on mat' skir?
Intrachinn: Please deenna the vlapioo
Answer The nambher
skire x 2--
Anttraaiion wheut quere the was
Abseen: The wee calygiy Art-
Auwer: The mee playing ianenal Intrrchion: ally o there red shor
doable dicter har- Thi bur i ported ov The bur crosd, wilk the di awcrieed
Ansar: coalo
suer
Ressoning
pests soald to dos co cesind necsoo one pomblity ohee the state
Rotsening Sepreettatios
Viazi Pharpl-ttee Comneration
Inseractio:: Plecse daterbu "Repasin
the platty heis, wkich eoile an the slape's posic what Shay piaylty
Aucolat
hepe the person seand on the coere
foan He it kelaley doy in bo amm,, mowly affecton and
Iesnacioon Plets desardo shick the <Regian standing
Pridrandon New many diitert an cie
Retoring Segmestation
Plasse reament the trali tkat Aithare
Atowe:: The man stundoy at rag
instraction: Wikst she cobr of eix mible scover?
Inseaation: Piacse resment Arower ie x [505]
frower
Hemrativ Segrestation
Segnantorion Caplaring
Inseg-level Ciption
anekt paany camd,
indea Asdliy ite meno She whs petbiny the zades cheasty ofrictin cowerds csonat-
clardding aust
Understanding
steeding the grouy area of
2- ehak skeep
con, The actro shanalng frost
press Risd
is shoadiy on dest end tbe soman oppent to
Regionslovd Caption
ergayiy
The image faturur white poeap truck posked streat: The traze it lars,
plated en
wikiis
TROPPE
"Earope") The dook
laptog cibiad
enace pocerssed on the steeel, and stpuors parked fooet of the keste's dranway The consoon futtre streets otiir
pastioned toe reaht
Pixel-Level
Object-Level
Image-Level
Figure 1: The comprehensive capabilities of OMG-LLaVA. OMG-LLaVA can handle variety of pixel-level, object-level, and image-level understanding and reasoning tasks.
Meanwhile, by combining vision models and language models [71; 72: 70; 64; 65; 107]. research on multi-modal models also adopts wunsformer-basees design. One representative work, LLaVA [71; 72; 70], treats visual tokens as the inputs of LLMs and makes LLMs understand visual contents. Several works adopt similar designs [3; 13; 64; 18: 25], and all of them are termed Multi-modal Large Language Models (MLLMs). After that, most research focuses on improving MLLM benchmarks in various ways, including increasing data sizes [14; 18; 70] and enhancing the visual encoders [131; 24: 18] and visual resolutions [110; 18; 65; 25]. However, LLaVA- -ike models cannot output precise location information since they only carry out leaell analysis. Thus, recent works [126; 133; 11: 82; 13; 119; 128; 87; 67) try to fill this gaps by adding extra detection models for object level analysis, mask decoder for pixel-level analysis, visual prompts, and also propose task-specific instruction tuning with various datasets. By providing extra detection data and decoder, the updated MLLMs can perform localization output. However, these models [135; 96; 49] are specifically tuned on specific tasks, losing the ability of LLaVA for image level analysis, such caption and visual question answering. Meanwhile, several works [126; 49; 87; 78] adopt LLMS as agents to collaborate with various visual models or generation models. Despite the works being simple and effective, the inference and parameter costs are huge due to the multiple visual encoders and decoders. Moreover, there are no specific designs for task unification.
Motivated by the previous analysis, we ask one essential question: Can we bridge image-level, object-level, and pixel-level tasks into one MILMM model with only one LLM, one visual encoder, and one visual decoder? Back to the universal perception models, we can leverage these models to help us build stronger MLLM to unify three-level inputs, including image, object, and pixel levels. In particular, we adopt OMG-Seg [58] as our universal perception model due to its simplicity and effectiveness in various segmentation tasks.
In this work, we present OMG-LLaVA, an elegant MLLM that bridges image-level, object- and pixel-level reasoning and understanding tasks in one model. We preserve the basic lixel-level segmentation ability of OMG-Seg by freezing the visual encoder and decoder, as shown in the bottom left of Fig. 1. Since the LLM processes text input, OMG-LLaVA can also perform referring segmentation, reasoning segmentation, and grounded conversation and generation, shown the top left of Fig. 1. Moreover, shown in Fig. 1, with the help of LLMs, OMG-LLaVa can also perform

---

Text Roperne
Test
Pecsption Regerse
Tou Ropannc
Basepion Berzense
Reezens
Paserfice
Aepuenee
Ropueee
Enoakkr
LLM
Tinee
Enzoder
Fastars
Tran ioozationll
Penoptosa Modal
Iseac Eazader
1IM
Tree
Paroptiax Medale
Esoder
Irous
LLM
Regior
Text
Proeption
Uiivenal
1IM
Teat
Instuciion
Imuge
Prozzte
Vieu!
Irage
Inage
Phorpees
Viual
Iaage
Phorpes
Voza
(b)
(0)
Figure 2: Summary of Current MLLM Architectures: (a) MLLMs with only image-level capability, including [71; 72; 70; 65], etc., (b) MLLMs with object-level capability, including [126; 87]. (c) MLLMs with pixel-level capability, including [49; 88), etc., (d) MLLMs with both object-level and pixel-level capabilities but with a very complex system, such as [87], (e) OMG-LLaVA's architecture, which possesses an elegant and simple design while having image-level, object-level, and pixel-level capabilities.
image-level understanding as LLaVA, including caption and conversation, where most MLLMs for grounding lose such ability. In addition, OMG-LLaVA also supports the visual prompts as inputs, which results in object level understanding. such as visual prompt-based conversation and region-level captions. We achieve all these abilities using one L.M,, one encoder, and one decoder.
In particular, better encode the visual segmentation outputs, we propose a perception prior embedding module to absorb the object queries into object-centric visual tokens, which are the inputs of LLMs.. We present a unified instruction formation strategy, which lets the model accept visual images, texts, and visual prompts as inputs and generate the response of text, segmentation tokens, segmentation masks, and labels. Following the LLaVA [71], we adopt pretraining and instruct tuning pipelines. Extensive experiments show the effectiveness of our components and training strategy. In addition to visual segmentation, OMG-LLaVA can also achieve good enough performance on datasets, including COCO panoptic segmentation, VIPSeg video panoptic segmentation, refCOCO, refCOCO+, refCOCOg referring expression segmentation, GranDf grounded conversation generation, and refCOCOg region caption datasets. We hope our research can inspire the research on MLLM design in a more elegant way for the community.
2 Related Work
Multimodal Large Language Models. Early multimodal models [53] explore better fusion strate- gies, various feature extractors, and different meta-architectures. Most works focus on single tasks, such as caption and VQA. With the development of the large language models [6; 93; 40], recent works [52; 3; 92; 71; 17] mainly explore building an instruction-tuning pipeline for multiple multi- modal benchmarks [39; 74; 62; 32). LLaVA [71; 70; 69; 106; 136; 30; 28] is one earlier work that treats visual features as tokens. After that, several works [ [126] explore visual cues to enhance the visual inputs of LLaVA. On the other hand, several works [129; 127; 88; 131; 24; 25; 66; 130; 83; 38; 49] add extra components adapt LLaVA for visual grounding, detection, segmentation, and video analysis. In particular, several works explore language-driven grounding and segmentation. However, these works are all trained with specific purpose. We aim to build the simplest model to unify segmentation, instruction tuning, and prompt-driven segmentation in one model. To the best of knowledge, we are the first model to achieve this goal.
Unified Segmentation Models. The vision transformers [10: 26; 79; 94] have led to research interest in universal segmentation. Recent works [95; 19; 123; 56; 21; 117: 115; 73; 91; 124; 122; 139; 111: 54; 141] have developed mask classification architectures with an end-to-end set prediction approach, outperforming previous specialized models [12; 46; 36; 57; 34; 60: 140] in both image, video and generalization segmentation tasks [45; 61; 59]. In particular, several works explore open-world segmentation, including entity segmentation [85; 84]. open-vocabulary segmentation [125; 105]. Meanwhile, several works [58; 41; 111; 112; 33; 2] adopt one model with shared parameters to perform various segmentation tasks. One recent work, OMG-Seg [58]. first unifies image, video, open- veen-vocabulary. and interactive segmentation in one simple model. However, all of these works focus visual segmentation and cannot generate interactive text and visual prompts, like MLLMs. Our work builds such a bridge to align MLLMs, visual segmentation, and prompt-driven segmentation models from joint co-training and model sharing, which serves new baseline for this field.
Language-driven Location and Segmentation. Early works [120; 68; 44; 23; 104; 135] in this direction mainly define the various language-driven tasks, including referring segmentation and
3

---

Table I: Comparison of capabilities of different models. We include several representative methods here. Our DMG-LLaVA offers the most comprehensive capabilities, encompassing image-level object-level. and pixel-level understanding and reasoning. Compared to [87; 35J. OMG-LLaVA features an elegant and simple system architecture with only a single visual encoder.
Method
Visual Encoder
Caption
Image-level
Comversation
Visual Prompts
Objoct-level
Caption
Conversation
Universal Seg RES
Placl-lvel
GCG
LLAVA [71]
MiniOPT4 [1:2]
mPLUG-Owl [118]
LLaMl-Adapler [132]
Gemini
InterVLL [18]
V.sionL..M (97)
Shikra (13)
Point Box
Kaomos-2 [83]
Box
GPT4Ral [1.35]
Ferret [119]
Point & Box & Mask Masic Point Box Mask
Osprey [126]
SPHINX-V [67]
LJSA (49]
GLAMM (87)
Box
Groundhug [134]
Point & Box & Mask Box
AnyRet [35]
PixelLM [88]
GSVA [109]
Groma [78]
Box Point & Box & Mask Mask
VIP-LLaVA [S]
PSAL
LaSagnA [102]
UMG-See [SS]
Point
LMVA
Foint & Box & Mask
referring localization. Most works [31; 5; 116; 77; 103; 105] design effective fusion modules to achieve better performance. Mcanwhile, several works [55; 103; 108; 49; 87; 126; 81] explore more complex segguage-driver tasks from various aspects, including robustness, reasoning, and region-level caption. IISA [1 14] involves reasoning-base -based segmentation. Then, GLaMM [87] annotates a new dataset and proposes region-level caption and segmentation tasks. Meanwhile, several works [29; 72] use LLMs as agents to assign different visual experts. In contrast to these works, our method is a more elegant baseline, which contains only one visual encoder, one LLM, and one decoder.
Visual Prompts. With the prompting ability LLMs, several works [I [100; 99; 4; 138; 90; 51; 81] also explore visual prompting methods in vision. According the design and purposes, these works can be divided into different aspects, including learnable tokens [138], mask visuualmoodelee
for different
tasks [100; 27; 98], and various visual prompting encoders for visual outputs [99; 101; 125; 47J. Our OMG-LLaVa Va also supports visual prompts for better interaction with the user's inputs, showing the potential for product purposes.
Methodology
3.1 Task Unification
Motivation and Our Goals. The LLMs unify most NLP tasks token generation tasks and exhibit strong reasoning and instruction-following capabilities. As shown in Fig. (a), LLaVA- like models [71; 70; 69; 110; 65; 131; 24; 25; 18; 64) further introduce visual tokens into LLMs, enabling LLMs to understand visual information and perform visual-based reasoning. However, they cannot accomplish fine-grained visual tasks like object-level and pixel-level understanding and reasoning. As shown in Fig. 2 (b). [126; 133; 11; 82; 13; 119] introduce region-level visual embeddings, allowing LLMs achieve object-level understanding and reasoning tasks. However, these models rely on complex region embedding extraction designs. In addition, most cannot perform pixel-level understanding tasks. Thus, as shown in Fig. 2 (c).[49; 88; 35] introduce segmentation tokens, enabling LLMs to output segmentation masks and thus handle lixel-level understanding and reasoning tasks. Nonetheless, they require a large segmentation module, such as SAM [47], making the system highly redundant. As shown in Fig. 2 (d), GLAMM [87] combines the above pipelines to handle object-level and pixel-level tasks. However, this significantly increases the system's complexity and redundancy. Additionally, GLAMM relies explicit instructions from the user, losing the perception ability to handle basic pixel-level understanding tasks such as instance segmentation, semantic segmentation, panoptic segmentation, and interactive segmentation.

---

Answer: 1t's weoden uop- pier </p> (5EG), surrounded by the spo lake </p= (556) and the spa
forese </p> (5E0]- A. blue <p> thy </p> [550] is above the pier and the <p> mountolns </p> (569)
Decode
Perception Prior Einhoddiigg
LLM
OMG Decoder
Text Projector
Projector
Tokenizo
Imngee Enceder
Question: <lmagoi' Please describe the <Ragioni and respord with interlesved mosks
Visual Procapts
Object centric
OMG-Seg
(Point, Box and Mask)
Visual Tokens
Visual Tokenss
Figure 3: The Overview of OMG-LLaVA. OMG-LLaVA consists OMG-Seg and LLM. OMG-Seg tokenizes the image into pixel-centric visual tokens, the detected objects, and inputs visual prompts into object-centric visual tokens. Additionally, the [SEG] token output by LLM is decoded by OMG-Seg into segmentation masks. OMG-Seg remains frozen at all stages.
In this paper, we focus on addressing all the challenges above in more simple yet elegant way. Our OMG-LLaVA unifies image-level (such as image caption and image-based conversation), object-level (such region caption and visual prompt-based conversation), and pixel-level (such as universal segmentation, referring segmentation, reasoning segmentation, and grounded conversation generation) visual understanding and reasoning tasks into token-to-token generation. The framework follows simple and elegant system design, including only one visual perception module and one large language model.
Unified View of Different Tasks. We model various tasks as the token-to-token generation to bridge the gap between image-level, object-level, and pixel-level understanding and reasoning. To support these tasks, we define three types of tokens: text tokens Tú, pixel-centric visual tokens Tpe. and object-centric visual tokens Tor. Text tokens encode textual information. Pixel-centric visual tokens represent dense image features, providing the LLM with comprehensive image information. Object-centric visual tokens encode the features of specified objects, offering the LLM object-centric information, and can be easily decoded into segmentation masks.
Then, all the tasks be unified as:
Tout Tout LLM(TTR Tin T:n))
(1)
For example, in the classic image-level understanding task, i.c., image caption, a text response Tout generated based text instruction T.Th and image features The. In the object-level understanding task, region captioning, the text response Tout is generated based on text instruction TiM image features T and specified object-centric visual tokens Tix. The pixel-level reasoning task, referring segmentation, involves generating object-centric visual tokens Tout based text instruction T and image features Tim- Additionally, OMG-LLaVA can support various mixed-level tasks, such as providing grounded descriptions around specified objects.
Pixel-centric visual tokens can be obtained by tokenizing images using a CLIP backbone as the tokenizer. However, object-centric visual tokens require encoding object information to be easily decoded into segmentation masks. Therefore, methods like mask pooling in Osprey [126] and ROI pooling in GLaMM [87] fail to meet these requirements. We found that universal perception decoder can meet all the requirements. Thus, we chose the OMG-Seg decoder [58] as the object-centric tokenizer due to its comprehensive capabilities.
3.2 OMG-LLaVA Framework
The framework of OMG-LLaVA is shown in Fig. 2 (e). OMG-LLaVA comprises large language model (LLM) and a frozen universal perception module. The universal perception module encodes images and visual prompts from users into pixel-centric and object-centric visual tokens. It obtains object-centric visual tokens output by the LLM into explicit segmentation mask responses. The LLM accepts text instruction tokens and pixel-centric and object-centric visual tokens from the universal perception module as inputs and then outputs text responses along with object-centric visual tokens. The detailed architecture of OMG-LLaVA is illustrated in Fig. 3. The universal perception module
5

---

Attention Mask for Point, Box, Mask Prompts
OMG Decoder
Point prompt
Center
Mask prompt
Visual Prompts
Musked Cross- Afrettonn
Slokal Attationn
Encode
Box prompt
Enccdde
(Point, Bax and Musk) Encode
Sclf Attention
Mask
Mask
Predict
Predicr
Visual Prompts Queries
B tD
Crop
Learable Queries
K&V
Object Queries
Attention Mask
Altention Mask
Attention Mask
Image Feature
Figure 4: The Architecture of the OMG Decoder. A simple attention mask generation strategy enables the OMG decoder to encode point, box, and mask prompts.
comprises an image encoder, an OMG decoder [58], and a non-trainable perception prior embedding component.
Image Encoder. To maximize the perception capabilities of the universal perception module, use the ConvNeXi-L [75]-based CLIP [86] model as the image encoder and employ a high image resolution (1024x 1024). However, the large image resolution results in excessive visual tokens input into the LLM, leading to significantly higher computational costs than using lower-resolution images (such as 2224 or 336x 336). We address this issue by utilizing the lowest resolution image features (32x downsampling). Additionally, we use the pixel shuffle operator to further reduce the image features' resolution. Ultimately, the downsampling factor for the image features used to generate visual tokens is 64, meaning that a 1024x 1024 image produces 256 visual tokens.
OMG Decoder. We utilize the OMG decoder [58] to generate object-centric visual tokens, furnishing the LLM with information regarding the primary objects in the image and those mentioned by the user's imput visual prompts. As shown on the left side of Fig. the OMG decoder comprises masked cross-attention [19] and self-attention layers. The OMG decoder's input includes a set of learnable object queries [20; 19; 10] for automatically capturing all objects of interest and visual prompt queries derived from encoded input visual prompts [47]. The visual prompt queries and learnable object queries are collectively termed object queries. The OMG decoder probes feature for object queries from the image features by employing masked cross-attention and models relationships between objects through self-attention. The object queries can be decoded into segmentation masks and object categories via a simple FFN layer. With the OMG decoder, OMG-LLa VA can efficiently tokenize object information into object-centric visual tokens, thereby equipping the LLM with information about objects in the image and those referenced by the user.
The OMG decoder can accept point prompts as input. While box and mask prompts can be easily converted into point prompts, this crude conversion significantly loses prompt information, compli- cating the explicit encoding of the user's intent. To address this, we can impose constraints on the attention masks of the masked cross-attention layers based on the visual prompt to precisely encode the object information referenced by the prompt. As depicted on the right side of Fig. 4, we utilize the box coordinates to define attention masks for all pixel features outside the box for box prompts. Similarly, we directly employ the provided object mask to generate attention masks for mask prompts. With this straightforward attention mask modification strategy, OMG-LLaVA can accurately capture the user's visual prompts, encompassing point, box, and mask prompts.
Perception Prior Embed- ding. We find that directly combining a frozen per- ception module with LLM doesn't perform well, as also observed in LISA [49]. To retain the full capabili- ties of the universal percep- tion module, OMG-LLaVA doesn't 't fine-tune the percep- tion module to adapt to the output of the large language model. Instead, we propose
Filter buckground queries
(HW,N,) (N₂.C)
Prdditt
!
Object Queries
Visual Tokens
Query Embedding
Mask Score
Image Feature
Figure 5: The process of the perception prior embedding strategy. The per- ception prior embedding strategy integrates object queries into image features based segmentation prior.

---

a perception prior embed-
ding strategy to tackle this challenge. Fig. 5 illustrates the perception prior embedding strategy.
First, we fuse the image features FF E R HW xC outputted by the image encoder with the object queries Q RA,xC outputted by the OMG decoder D. Specifically, mask M E gN obtained from the object queries and the corresponding confidence score 1x derive mask score MS IR. HW xN, for each pixel for the object queries:
utilize the segmentation
MS = Softmar( M @ S.diiim---. = =11
(2)
Then, we compute a weighted average of the object queries Q based on the mask score MS and obtain the corresponding weighted object queries for each pixel. Pixel-centric visual tokens Tpp obtained by adding the weighted object queries to the image features F:
Tp MS Q+F
Additionally, we treat the foreground object queries as object-centric visual tokens Tow. The object- centric visual tokens To are concatenated with the pixel-centric visual tokens Tp to form the visual tokens To (T Too), which are input the LLM to provide rich perception prior information.
Visual Projector and Text Projector. Following [71], we use an MLP the visual projector, which is responsible for mapping visual tokens to the LLM's text embedding space. Since our visual tokens are pixel-centric and object-centric tokens, the visual projector comprises two MIPs, each handling one type of visual token separately. Inspired by [49; 87], we also use simple MLP to map the LLM output's s hidden states the [SEG] token to the visual space.
Instruction Formulation. OMG-LLaVA can accept visual input, text input, and visual prompt input and output responses and segmentation token, segmentation masks and labels. Thus, can handle tasks such as image captioning, image-based conversation, region captioning, visual prompt-basee conversation, referring segmentation, reasoning segmentation, grounded conversation, etc. We use unified instruction formulation to support these functionalities. As shown in Fig. 3, there are three special tokens: <Image>, <Region>, and [SEG]. Before being fed into the LLM, the <Image> token is replaced by visual tokens Tr, and the <Region> token can be replaced by any object-centric visual token encoded by the visual prompt. The [SEG] token in the LLM's output sent to the frozen OMG decoder to be decoded into a segmentation mask.
3.3 Training and Testing Setup
Training. Following LLaVA [71], our OMG-LLaVA performs two-stage training: pretraining and instruction tuning. During the pretraining stage, the perception model and L.M frozen, and only the visual and text projectors can be tuned. In addition to the text regression loss, we apply regularization penalties the visual projector P, and text projector P. to preserve object-centric information as much as possible.
Cpretrain = Lrezt + Cregs Creg Creg=
= (Tor - (T..
During instruction tuning, in addition to finetuning the visual projector and text projector, LoRA [37] to finetune the LLM. Following [87; 58), besides the text regression loss, we apply cross-entropy loss and dice loss [80] supervise the segmentation mask decoded by the [SEG] token, as shown in following (We set a = 5 3 = 2 by default):
Lintruction = Liens + Lmask, Lmask = aLoF + BLDICE
(5)
Testing. The image-level, object-level, and pixel-level understanding and reasoning tasks can all be encompassed within the Eq. 3.1 paradigm. During the inference stage, we encode the necessary task requirements, such as text prompts, visual prompts, and image features, into tokens to input into the L.M. The output tokens of LLM are then decoded into text responses and segmentation mask responses according to the task definition. We refer the readers to check the more details the appendix.
7

---

Table 2: The comprehensive comparison of OMG-LLaVA and other MLLMs regarding pixel-level and object- level understanding and reasoning capability and performance. "-" indicates the method does handle this task. indicates that the method used the GranD dataset [87] for pretraining, which is significantly larger than the datasets used by other methods.
Visual Encoder Num
COCO
VIPseg
refCOCO cloU
refCOCO+ cloU
GCG METEOR
refCOCOg(C)
Method
PQ
VPQ
AP50
METBOR
OSprey [126]
16.6
LISA [49]
74.1
62.4
13.0
25.2
NcXT-Chat [128]
74.7
65.1
12.0
LaSagnA [102]
2
76.8
66.4
GSVA [109]
76.4
64.5
AnyRef
74.1
GLaMM: [87]
79.5
72.6
15.2
28.9
15.7
Pixell M (88]
73.0
66.3
OMG-LLaVA
53.8
14.9
Table 3:
Performance on referring expression segmentation datasets. indicates finetuning the referring expression datasets.
The evaluation metric cloU. "ft"
Freeze Decoder
Visual Encoder
refCOCO TestA
refCOCO+ TestA
refCOCOg
Method
TestB
Val
TestB
Test
LISA [49]
X
74.1
76.5
71.1
62.4
67.4
56.5
66.4
68.5
LISA(ft) [49]
74.9
79.1
72.3
70.8
58.1
67.9
70.6
PixelLM [88]
73.0
76.5
68.2
71.7
58.3
69.3
70.5
GSVA(f)) [109]
78.9
73.5
65.9
73.3
LM
75.6
77.7
71.2
63.6
69.7
58.9
70.2
OMG-LLaVA(R)
78.0
80.3
74.1
69.1
73.1
63.0
72.9
72.9
OMG-LLaVA(R)
77.2
79.8
74.1
68.7
73.0
61.6
71.7
71.9
Table 4: Performance on grounded conversation generation datasets. "ft" indicates finctuning on the GranDF [87] dataset. indicates that the method used the GranD dataset [87] for pretraining.
Visual Encoder
Val
Test CIDE
Methods
ft
METEOR
CIDEr
AP50
miOU
METHOR
AP50
mIOU
Kosmos-2 2 [82]
/
1
16.1
27.6
17.1
55.6
15.8
27.2
17.2
56.8
LISA [49]
/
2
13.0
33.9
25.2
62.0
32.2
24.8
61.7
GLaMMi [87]
4
2
15.2
43.1
28.9
65.8
14.6
37.9
27.2
64.6
OMG- -LLVA
x
1
13.8
36.2
26.9
64.6
13.5
33.1
26.
62.8
OMG-LLaVA
14.9
41.2
29.9
65.5
14.5
38.5
28.6
64.7
Table 5: Ablation study on RES and GCG datasets.
refCOCO cloU
refOOCO+ cloU
refCOCOg cloU
GCG
Methods
gloU
gloU
glaU
METROR
mīoU
Bascline (MO)
58.7
61.0
52.6
55.0
55,8
13.2
$1.0
Perception prior embedding (MI)
72.5
74.3
63.2
65.4
67.8
70.6
13.6
62.1
Object query input (M2)
74.4
75.9
64.4
66.2
68.5
71.5
13.8
63.6
4 Experiment
Dataset Setup. During the pretraining stage, we use the LLaVA pretraining dataset [71] to perform visual-text alignment, following LLaVA. The instruction tuming process OMG-LLaVA involves diverse range of tasks and datasets. For image-level understanding and reasoning tasks, we use the LLaVA dataset [71: 72; 70], which includes 665K descriptions, reasoning, and conversation data. For object-level understanding and reasoning, we use the object-level description and conversation data from the Osprey dataset [126] and the object-level point-prompt data from the MDVP dataset [67], which contain approximately 74K and 200K data, respectively. For pixel-level understanding and reasoning, we use the referring segmentation datasets, including refCOCO, refCOCO+ [42), refCOCOg [121], and refClef, totaling 74K data. Additionally, semantic segmentation datasets, including ADE20k [137] and COCO-stuff [7), totaling 26K data, and the grounded conversation generation dataset GranDf [87], containing 200K data, used.

---

Implementation Details. We use the pre- pre-trained ConvNext-L [75] OMG-Seg [58] as the universal perception module and InterLMM-7B [9] as the LLM for OMG-LLaVA. We adopt xtuner code- base [22] to build our model and data pipeline. The image is resized to x024x 1024. During the pretraining stage, only the visual projector and text projector are trained, with an initial learning rate set to 1e-3. During the instruction tuning stage, the initial learning rate is set to 2e-4, with only the perception model kept frozen, and the LLM is fine-tuned using LoRA [37]. The maximum sequence length in the LLM is set to 2,048. All training is conducted on four NVIDIA A800 GPUs with 80GB of memory. The pretraining stage and instruction tuning stage look 7 hours and 48 hours, respectively.
4.1 Main Results
Comparison with MLLMs. OMG-LLaVA is comprehensively compared with current MLLMs with perception capabilities, and the results are shown in Tab. 2. OMG-LLaVA demonstrates the most comprehensive capabilities. It achieves performance comparable to the SOTA referring segmentation, grounded conversation generation, and region captioning. Additionally, OMG-LLaVA retains basic segmentation ability, enabling it to handle universal image and video segmentation tasks. Compared to other MLLMs, OMG-LLaVA features simple and elegant system design, incorporating only a single visual encoder.
Referring Expression Segmentation. We evaluate OMG-LLaVA on refCOCO, refCOCO+, and refCOCOg, with the results shown in Tab. 3. OMG-LLaVA outperforms LISA [49] by 1.5 cloU, 3.2 cloU, and 4.3 cloU on the validation sets of refCOCO, refCOCO+, and refCOCOg. respectively, while keeping the OMG decoder frozen and using only a single visual encoder. When we unfreeze the OMG decoder and finetune OMG-LLaVA on the referring expression segmentation task, OMG-LLaVA achieves 78.0, 69.1, and 72.9 cloU on refCOCO, refCOCO+, and refCOCOg, respectively, surpassing LISA by 3.1, 4.0, and 5.0 cloU, Compared to PixeILM [88], OMG-LLaVA shows performance improvements of 5.0 cloU and 3.6 cloU on refCOCO and refCOCOg, respectively.
Grounded Conversation Generation. Grounded conversation generation is a comprehensive and complex task that involves both image-level and pixel-level understanding and reasoning. MLLMs need to have the ability to provide fine-grained image descriptions and pixel-level understanding, linking the objects in the image captions to the corresponding segmentation masks. As shown in Tab. 4, when trained with comparable data, OMG-LLaVA surpasses LISA [49] by 1.9 METEOR and 7.3 CIDEr in image description ability. In terms of pixel understanding, OMG-LLaVA VA also outperforms LISA by 4.7 AP50 and 3.5 mIoU, even though LISA uses SAM and finetunes its segmentation decoder. Despite GLaMM [87] using much more training data than OMG-LLaVA, OMG-LLaVA demonstrates comparable pixel-understanddng capabilities, outperforming GLaMM with 0.6 CIDEr, 1.4 AP50 and 0.1 mIoU on the test set.
4.2 Ablation and Analysis
Ablation Study. We conduct ablation studies on referring expression segmentation and grounded conversation generation datasets, with all train- ing and testing settings consistent with the main experiments. We use a simple combination of
Plote deniita due image ittarieed aptestatis
OMG-Seg [58] and LLaVA [71] as our baseline, similar to LISA [49], where the [SEG] tokens output by the L.L were input into OMG-Seg
janeley terfa
the
to obtain segmentation masks, with OMG-Seg kept frozen.
Poare temoont the Joth Bav
As shown in Tab. 5, the baseline performed poorly on the RES datasets. Similarly, it ex- hibited low segmentation quality on the GCG dataset. This is because the LLM did not ac- quire any segmentation priors and needed to generate segmentation queries based on image features and adapt them to the input of the frozen perception module, which is challenging task.
Figure 6: Visualization of the effectiveness of the pro- posed strategies. The left part shows the baseline (MO in Tab. 5), the middle part shows the model with perception prior embedding (MI Tab. 5), and the right part shows the model with both perception prior embedding and object query input (M2 in Tab. 5).

---

When using our proposed perception prior em- bedding strategy, OMG-LLaVA exhibits perfor- mance gains of 13.8 cloU, 10.6 cloU, and 11.7 cloU on refCOCO, refCOCO+, and refCOCOg, respectively. Additionally, the perception prior embedding strategy also brings performance improvement of 11.1 mloU on the GCG dataset and slight improvement in image description capability (0.4 METEOR). When foreground object queries were provided to the LLM, OMG-LLaVA further improved its performance by 1.9 cloU on refCOCO and 1.5 mIoU on GCG.
We conducted a visualization analysis of the proposed strategies. As shown in the left part of Fig. 6, the simple baseline has poor capability in associating text and segmentation, which the crucial reason for its poor performance on RES. When using our proposed perception prior embedding strategy, the object query and pixel features are explicitly integrated according to the perception prior, resulting in significantly enhanced text-segmentation association capability, By adopting the object query input strategy, the quality of some challenging segmentation cases, such as the lower right corner of the fence in Fig 6, slightly improves.
We provide visualization results of OMG-LLaVA on multiple image-level, object-level, and pixel-level tasks in Fig. 1. Additional qualitative visualization results or compara- ble visual results for referring expression segmentation and grounded conversation generation are presented in the appendix.
Qualitative Results.
5 Conclusion
We present new MILM, OMG-LLaVA, which bridges image-level, object-level, and pixel-level understanding and reasoning in one model. Our method only contains one image encoder, one LLM, and one decoder. With proposed perception prior embedding and unified task instruction tuning, OMG-LLaVA can perform over 8 different multi-modal learning tasks, as well as preserving the visual perception ability of the OMG-Seg baseline. Our method can achieve comparable results compared with previous combined works with much fewer trainable parameters and computation costs. We hope our work can inspire the community to rethink the design of the MLLM meta-architecture to minimize the model components and maximize the MLLM's 's functionalities.
Acknowledgment. This work was supported by the National Natural Science Foundation of China (grant No. 42171430). This work is also supported by the CCa-Kuaishou Large Model Explorer Fund. It is also partially supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contributions from the industry partner(s).
References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical report: A highly
capable language model locally on your phone. arXiv preprint arXiv:2404. 140.:429444/9 2024. 23,
[2] Ali Athar, Alexander Hermans, Jonathon Luiten, Deva Ramanan, and Bastian Leibe. Tarvis: unified
architecture for target-based video segmentation. In CVPR, 2023. 3
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:23308. /2966, 2023. 3.3
[4] Amir Bar. Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. In NeurlPS, 2022. 4
[5] Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. In CVPR, 2022. 4
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020. 1.
[7] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes context. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 1209-1218, 2018.
8
[8] Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. In CVPR, 2024. 4 [9] Zheng Cai, Maosong Cao. Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Intern1m2 technical report. arXiv preprint arXiv:240..17297 2024. 9, 23, 24
10

---

[10] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier. Alexander Kirillov, and Sergey
Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 3.
[11] Chi Chen, Ruoyu Qin, Fuwen Luo, Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. visual instruction tuning for multimodal large language models. arXiv preprint arXiv:2308. 13437, 2023.
2. 4
[12] Kai Chen, Jiangmiao Pang. Jiaqi Wang. Yu Xiong. Xiaoxiao Li, Shuyang Sun. Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid task cascade for instance segmentation, In CVPR, 2019. 3
[13] Keqin Chen. Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023, 2,
[14] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi- models with better captions. arXiv preprint arXiv:2311.1799,
2023. 2
[15] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin. et al. Are we on the right way for evaluating large vision-language models? arXiv
preprint arXiv:2403. 20330, 2024, 24
[16] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling
framework for object detection. ICLR, 2022. 1
[17] Wei-Ge Chen, Irina Spiridonova, Jianwei Yang. Jianfeng Gao, and Chunyuan Li. iava-interatttie:: An
all-in-one demo for image chat, segmentation, generation and editing. arXiv preprint arXiv:2311.0053
2023. 3
[18] Zhe Chen, Weiyun Wang, Hao Tian, Shengiong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites, arXiv preprint arXiv:2404. 16821, 2024. 2,4
[19] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-
attention mask transformer for universal image segmentation. In CVPR, 2022. 1, 3,
[20] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification not all you need
for semantic segmentation. In NewrlPS, 2021.
[21] Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang.
Segment and track anything. arXiv preprint arXiv:2305.065 06558, 2023.
[22] XTuner Contributors. Xtuner: A toolkit for efficiently tine-tuning Ilm. https://github. com/
InternLM/xtuneee 2023. 9
[23] Henghui Ding, Chang Liu, Shuting Hc, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale
benchmark for video segmentation with motion expressions. In ICCV, 2023. 3
[24] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang. Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li. Hang Yan, Yang Gao, Xinyue Zhang. Wei Li. Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang. Yu Qiao, Dahua Lin, and Mastering frre-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401. 16420, 2024. 2, 3, 4
Jiaqi Wang. dheemlm-xcomposert
[25] Xiaoyi Dong, Pan Zhang. Yuhang Zang. Yuhang Cao, Bin Wang. Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang. Yining Li. Hang Yan. Yang Gao, Zhe Chen. Xinyue Zhang, Wei Li. Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingeheng Zhang, Jifeng Dai, Yu Qiao, Dahua A pionecring large lasoun-lagguage model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.065: 42, 2024, 2, 3, 4
Lin, and Jiaqi Wang. neemilm-xcommoser2-kkeee
[26] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image is
worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. 3
[27] Zhonghin Fang, Xiangtai Li, Xia Li, Joachim M Buhmann, Chen Change Loy, and Mengyuan Liu. Explore in-context learning for 3d point cloud understanding. NewrIPS, 2024. 4
[28] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Mcishan Zhang, Mong-Li Li Lee, and Wynne Hsu. Vidco-of-thought Step-by-ster video reasoning from perception to cognition. In /CML, 2024. 3
[29] Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. arxiv preprint. 2024. 4
[30] Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua. and Shuicheng Yan. Enhancing
video-language representations with structural spatio-temporal alignment. PAMI, 2024, 3
[31] Guang Feng. Zhiwei Hu, Lihe Zhang, and Huchuan Lu. Encoder fusion network with co-attention
embedding for referring image segmentation. In CVPR, 2021. 4
[32] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng. Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark
for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023. 3, 2,, 24
[33] Xiuye Gu. Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang. Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo. Huizhong Chen, Liang-Chies Chen, et al. Dataseg: Taming a universal multi-dataset
multi-task segmentation model. arXiv preprint X 306.0/736, 2023. 3
[34] Meng-Hao Guo, Cheng-Ze Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. NeurlPS, 2022.
[35] Junwen He, Yifan Wang. Lijun Wang. Huchuan Lu, Jun- Yan He, Jin-Peng Lan, Bin Luo, and Xu- ansong Xie. Multi-modal instruction tuned llms with fine-grained visual perception. arXiv preprint
11

---

S.Kiv:2403.02960 2024, 4,8
[36] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 3
[37] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arwiv:2106.00000 09685,
2021. 7.9
[38]
Kuan-Chih Huang, Xiangtai Li, Lu Qi, Shuicheng Yan, and Ming-Hsuan Yang. Reason3d: Searching and reasoning 3d segmentation via large language model. arXiv preprint arXiv:2405. / /7427, 2024, 3
[39] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and
compositional question answering. In CVPR, 2019. 3
[40] Louis Martin Hugo Touvron, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Bahaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and et al. IJama 2: Open foundation and
fine-tuned chat models. orXiv:2307. 232888888.888 2023. 1,
[41] Jitesh Jain, Jiachen Li. MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. OneFormer: One Transformer to Rule Universal Image Segmentation. CVPR, 2023, 3
[42] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects
in photographs of natural scenes. In EMNLP, pages 787-798, 2014. 8
[43] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo. Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In ECCV, pages 235-251. Springer, 2016. 23, 24
[44] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring
expressions. In ACCV, 2018. 3
[45] Dahun Kim, Jun Xie, Huiyu Wang. Siyuan Qiao, Qihang Yu, Hong Seok Kim, Hartwig Adam, In So
Kweon, and Liang-Char Chich Chen. nubeformer-dceellb:: Video mask transformer. CVPR, 2022. [46] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation.
CVPR, 2019.
[47] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. ICCV, 2023. 1,4, [48] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified
image classification, object detection, and visual relationship detection at scale. IJCV, 2020. 26 [49] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint 2r3: 00692, 2023. 1, 2, 3, 4, 6, 7, 8, 9, 23,
24, 25, 26
[50] Bohao Li, Rui Wang, Guangzhi Wang. Yuying Ge, Yixiao Gc, and Ying Shan. Seed-bench Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307. 2023. 23, 24
[51] Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Chunyuan Li, Jianwei Yang. et al. Visual in-context prompting. arXiv preprint arXiv:2311.1360/ 2023.
4
[52] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-
training with frozen image encoders and large language models, 2023.
[53]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified -aiguae understanding and generation. In ICML, 2022. 3
[54] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-basce visual segmentation: A survey, arXiv pre-print,
2023.
[55] Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj. and Yan Lu. Robust referring vidco object segmentation with cyelic structural consensus. In /CCV, 2023,
[56] Xiangtai Li, Shilin Xu, Yibo Yang, Guangliang Cheng, Yunhai Tong, and Dacheng Tao. Panoptic- partformer: Learning a unified model for panoptic part segmentation. In ECCV, 2022. 3
[57] Xiangtai Li, Ansheng You. Zeping Zhu. Houlong Zhao, Maoke Yang, Kuiyuan Yang, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In ECCV, 2020.
[58] Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding. Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg-seg: Is one model good enough for all segmentation? CVPR, 2024. 1, 4,
6. 7, 9, 26
[59] Xiangtai Li, Haobo Yuan, Wenwei Zhang, Guangliang Cheng, Jiangmiao Pang, and Chen Change Loy.
Tube-link: A flexible cross tube baseline for universal video segmentation. In ICCV, 2023. 3
[60] Xiangtai Li, Li Zhang, Guangliang Cheng, Kuiyuan Yang, Yunhai Tong, Xiatian Zhu, and Tao Xiang.
Global aggregation then local distribution for scene parsing. IEEE TIP, 2021. 3
[61] Xiangtai Li, Wenwei Zhang, Jiangmiao Pang, Kai Chen, Guangliang Cheng, Yunhai Tong, and Chen Change Loy. Video k-net: A simple, strong, and unified baseline for video segmentation. In CVPR, 2022. 3
[62] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- li-Rong Wen. Evaluating object
hallucination in large languala models. EMNLP, 2023, 3
[63] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. 2023. 23, 24
[64] Yanwei Li, Chengyao Wang, and Jiaya Jia. Alama-vid: An image is worth 2 tokens in large language models, arxiv preprint ar arXiv:2311.
2023. 1, 2,
[65] Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng
Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modalii vision language models. arXiv
12

---

preprini arXiv:2403. 18814. 2024, 2.3.4
[66]
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint 2xX11:2311.1022 2023. 3
[67]
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang. Shanghang Zhang, and Hongsheng Li. Braw-and-underste
Leveraging visual prompts enable milmss to comprehend what you want, arXiv preprint arXiv:2403.200 2024, 2. 8
[68] Chang Liu, Henghui Ding, and Xudong Jiang. GRES: Generalized referring expression segmentation. In
CVPR, 2023. 3
[69]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction
tuning, 2023. 3, 4, 23, 24
[70] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lee. Llava-next:
Improved reasoning, ocr, and world knowledge, January 2024. 1, 2, 3, 4,
[71] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrlPS, 2023.
1, 2, 3, 4, 7, 8, 9
[72] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang. Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint
trXiv:2311.0544.
2023. 1, 2, 8
[73] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang. Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, ct al. Grounding dino: Marrying dino with grounded pre-training for open-set object
detection. arXiv preprint arXiv:2303. 0350499999 2023. 3
[74] Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, ct al. Mmbench: Is your multi modal model an all-around player? arXiv
preprint arXiv:2307,000 230700687... 2023, 3, 23, 24
[75] Zhuang Liu, Hanzi Mao, Chao-Yur Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A
convnet for the 2020s. CVPR, 2022. 6.9
[76] Pan Lu. Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, MhngChhunn Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. 24
[77] Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi- task collaborative network for joint referring expression comprehension and segmentation. In CVPR, 2020. 4
[78] Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization
for grounding multimodal large language models. arXiv preprini arXiv:2404. 13013, 2024. 2. 4
[79] Sachin Mchta and Mohammad Rastegari. Mobilevit: Light- general-purposse and
vision transformer. In ICLR, 2022. 3
[80] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks
for volumetric medical image segmentation. In 3DV, 2016.
[81]
Ting Pan, Lulu Tang, Xinlong Wang, and Shiguang Shan. Tokenize anything via prompting. arXiv preprint arXiv:2312.00 029 2023. 4
[82] Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos- Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023. 2,
4, 8
[83] Lu Qi. Yi-Wen Chen, Lehan Yang, Tiancheng Shen, Xiangtai Li, Weidong Guo, Yu Xu, and Ming- Hsuan Yang. Generalizable entity grounding via assistance of large language model. arXiv preprint N.iv:244020025555 2024. 3
[84] Lu Qi. Jason Kuen, Tiancheng Shen, Jiuxiang Gu, Weidong Guo, Jiaya Jia, Zhe Lin, and Msuan Yang. High-quality entity segmentation. In ICCV, 2023. 3
[85] Lu Qi, Jason Kuen, Yi Wang. Jiuxiang Gu, Hengshuang Zhao, Philip Torr, Zhe Lin, and Jiaya Jia. Open world entity segmentation. TPAMI, 2022. 3
[86] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from S [87] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M. Anwer, Eric Xing. Ming-Hsuan Yang, and Fahad S. Khan, Glamm: Pixel grounding
natural language supervision. In ICML,, 2021. 1, 6
large multimodal model. CVPR, 2024. 1, 2, 3, 4, 5, 7, 8, 9, 23, 24, 25, 26
[88] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao 7hao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. arXiv preprint arXiv:3312.022220 2023. 3, 4, 8, 9. 23
[89] Shuai Shao, Zeming Li. Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In CVPR, 2019. 26
[90] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang. Qiying Yu, Zhengxiong Luo, Yueze Wang, Yong- ming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv:2312. 13286, 2023, 4
[91] Shuyang Sun, Weijun Wang, Andrew Howard, Qihang Yu, Philip Torr, and I Liang-Chieh Chen. Remax: Relaxing for better training on efficient panoptic segmentation. NewrlPS, 2023. 3
13

---

[92] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: a family of highly capable multimodal models, arXiv preprint arXiv:2312. 18855 2023, 1.3
[93] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv:2300. 13971, 2023. 1.3
[94]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. 1,3
[95] Huiyu Wang. Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end
panoptic segmentation with mask transformers. In CVPR, 2021.
[96] Junchi Wang and Lei Ke. Lim-seg seg: Bridging image segmentation and large language model reasoning. 2024.
arXiv preprint vXiv:2404.08766
[97] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, arXiv preprint 0rXiv:2305.11175 2023.
[98] Xinshun Wang. Zhongbin Fang. Xia Li, Xiangtai Li, and Mengyuan Liu. Skeleton-in-context Unified
skeleton sequence modeling with in-context learning. CVPR, 2024, 4
[99] Xinlong Wang. Wen Wang. Yue Cao. Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning, In CVPR, 2023. 1, 1.4
[100] Xinlong Wang. Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt:
Segmenting everything in context. In ICCV. 2023. 1,4
[101] Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengeheng He, Weizhu Chen, Zhangyang Wang. and Mingyuan Zhou. In-context learning unlocked for diffusion models. arXiv preprint 235552330..01115 2023. 4
[102] Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma. Lasagna: Language-bass segmentation
assistant for complex queries. arXiv preprint arXiv:2404.00 08506, 2024. 4, 8. 23
[103] Jianzong Wu, Xiangtai Li, Xia Li, Henghui Ding, Yunhai Tong, and Dacheng Tao. Towards robust
referring image segmentation. IEEE-TIP, 2024.
[104] Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang. Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong. Ziwei Liu, et al. Towards language-driven video inpainting via multimodal large
language models. CVPR, 2024. 3
[105] Jianzong Wu, Xiangtai Li, Shilin Xu, Haoho Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangming Zhang, Yunhai Tong. Xudong Jiang, Bernard Ghanem, and Dacheng Tao. Towards open vocabulary learning: A
survey. arXiv pre-print, 2023. 3.4 4
[106] Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization in multimodal Ilm. arXiv preprint arXiv:2406.05127. 2024.
3
[107] Shengqiong Wu, Hao Fei, Leigang Qu. Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal I1m.
ICML, 2024. 2
[108] Tsung-Han Wu, Giscard Biamby, David Chan, Lisa Dunlap, Ritwik Gupta, Xudong Wang, Joseph Gonzalcz, and Trevor Darrell. See, say, and segment: Teaching Imms to overcome false premises. arXiv preprint arXiv:23/2.08366 2023, 4
[109] Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan. Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. arXiv preprint arXiv:2312 1010003,,, 2023, 4, 4.8
[110] Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd:
Imm perceiving any aspect ratio and high-resolution
images. arXiv preprint vrXiv:2403.11000 2024. 2, 4
[111] Shilin Xu, Haobo Yuan, Qingyu Shi, Lu Qi. Jingbo Wang, Yibo Yang. Yining Li, Kai Chen, Yunhai Tong, Bernard Ghanem, Xiangtai Li, and Ming- -suun Yang. Rap-sam: Towards real-time all-purpose segment
anything. arXiv preprint, 2024.
[112] Bin Yan, Yi Jiang. Jiannan Wu, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Universal instance perception as object discovery and retrieval. In CVPR, 2023, 1.3
[113] An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin. Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu. Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue. Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao. Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.1065
2024. 23, 24
[114] Senqiao Yang. Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.1722 2023.
4
[115] Zongxin Yang, Jiaxu Miao, Yunchao Wei, Wenguan Wang, Xiaohan Wang, and Yi Yang. Scalable video object segmentation with identification mechanism. TPAMI, 2024. 3
14

---

[116] Zhao Yang, Jiaqi Wang. Yansong Tang, Kai Chen. Hengshuang Zhao, and Philip HS Torr. Lavt: Language-
aware vision transformer for referring image segmentation. In CVPR, 2022. 4
[117] Zongxin Yang. Yunchao Wei, and Yi Yang. Collaborative video object segmentation by multi-scale foreground-backgroudd integration. TPAMI, 2021.
[118] Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi. Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang. Fei Huang. and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality. 2023. 4
[119] Нaoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang. Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. ICLR, 2024. 2,
4
[120]
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In CVPR, 2018. 3
[121]
Licheng Yu. Patrick Poirson. Shan Yang, Alexander C Berg. and Tamara Berg, Modeling context in referring expressions.
ECCV, pages 69-85. Springer. 2016. 8
[122] Qihang Yu, Huiyu Wang, Dahun Kim, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Cmt-deeplab: Clustering mask transformers for panoptic segmentation. In CVPR, 2022, 3
[123] Qihang Yu, Huiyu Wang. Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. k-means mask transformer. In ECCV, 2022. 3
[124] Qihang Yu, Huiyu Wang. Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. k-means Mask Transformer. In ECCV, 2022. 3
[125] Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open-vocabulary
sam: Segment and recognize twenty thousand classes interactively, arXiv preprint, 2024. 3,
[126] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei 7hang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv:2312.10032, 2023. 1, 2. 5.8
[127] Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. arXiv preprint arXiv:2305. 10300.18221 18279, 2023. 3
[128] Ao Zhang. Liming Zhao, Chen-Wei Xie, Yun Zheng. Wei Ji, and Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXiv:2311.0444
2023. 2, 8
[129] Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, and Jianwei Yang. Llava-grounding: Grounded visual chat with large
multimodal models, 2023. 3
[130] Hang Zhang, Xin Li, and Lidong Bing, Video-Ilama: An instruction-tuned audio-visus language model for vidco understanding arXiv preprint 20X.x:2306.0285. 2023. 3
[131] Pan Zhang, Xiaoyi Dong, Bin Wang. Yuhang Cao, Chao Xu, Linke Ouyang. Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang. Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang. Yu Qiao. Dahua Lin, and Jiaqi Wang, Internlm-xcomponn xcomposer: vision-language large model for advanced text-image comprehension and composition. arXiv preprint
arXiv:2309.1511 12. 2023. 2, 3, 4
[132] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfci Hu, Shilin Yan, Pan Lu, Hongsheng Li. and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init
attention. arXiv preprint arXiv:2303.
2023.
[133] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao. Wengi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model region-offinterest. arXiv preprint 2037v:2307.0360) 2023. 2.4
[134] Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In CVPR, pages 14227-14238, 2024. 4
[135] Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large
multi-modal model. arXiv preprint arXiv:2403. 0.003./499, /4598, 2024. 3,4
[136] Xiangyu Zhao, Xiangtai Li. Haodong Duan, Haian Huang. Yining Li. Kai Chen, and Hua Yang. Towards
semantic equivalence of tokenization in multimodal 11m. arXiv preprint, 2024. 3
[137] Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic
understanding of scenes through the ADE20K dataset. CVPR, 2017. 8
[138] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vasion-language models. In IJCV. 2022. 4
[139] Qianyu Zhou, Zhengyang Feng, Qiqi Gu, Jiangmiao Pang, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Context-aware mixup for domain adaptive semantic segmentation. JEFF TCSVT,
2023.
[140] Qianyu Zhou, Xiangtai Li, Lu Hc, Yibo Yang. Guangliang Cheng, Yunhai Tong, Lizhuang Ma, and Dacheng Tao. Transvod: End-to-end video object detection with pratial-tempora transformers. TPAMI,
2022. 3
[141] Yikang Zhou, Tao Zhang. Shunping Ji, Shuicheng Yan, and Xiangtai Li. Dvis das: Improving video segmentation via dynamic anchor queries. arXiv preprint 2441:220040000080 2024. 3
[142] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing lasion-language understanding with advanced large language models, arXiv preprint arXiv:2304. 10592,
15

---

2023. 4
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We outlined the contributions in the abstract, specifically proposing an elegant MILLM architecture that achieves image-level, object-level, and pixel-level understanding and reasoning capabilities using only single visual encoder.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed the limitations of the paper in the appendix, including the inability to segment part-level objects due to the constraints of OMG-Seg.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details of the proposed method are included in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
All the code and model weights will be released upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Saaiic/guidesSCooee setiibmmssiooooiccyy... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nppp.pccoom cttps/cccppbbccc/ceccccccccccccc heabbiic/guidesss Cooeeubmiis ssion iesionnPoicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details included the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: NA.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All the experiments are conducted on 8 A800 80G GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp//eerssp../// cetpblii//thissuuueeellleel dines??
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
No potential societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

Table 6: Performance on image-level benchmarks.
Method
MME [32]
MMBench [74]
SEED-Bench [50]
POPE [63]
A12D [43]
Training only with LLaVA dataset
LLaVA 1.5 [69]
1422/267
68.5
65.9
86.7
$6.6
OMG-LLaVA
1448/282
67.5
68.9
89.7
61.7
Co-tratning with LLaVA datuset and segmentation datasets
LISA [49]
1/1
0.4
0.0
0.0
PixcILM [88]
309/135
17.4
0.0
0.0
LaSagnA [102]
0/0
0.0
0.0
0.0
GLaMM (87)
14/9
36.8
0.94
28.2
OMG-LLaVA
1177/235
47.9
56.5
80.0
42.9
Table 7: Performance with different LLMs.
LLM
ref0000 CloU
GloU
refOCCo CloU
GloU
MME perception
reasoning
MMBench
SEED.Bench
POPE
A12D
MMstar
SQA
Phi3-3.8B [1]
76.5
78.0
67.8
70.0
1291.6
265.0
59.6
60.6
86.7
56.9
37.1
61.7
M2-7LM2-77 [9]
76.3
77.8
67.7
69.9
1177.1
235.4
47.9
56.5
80.0
42.9
33.1
57.8
Qwen2-7B [113]
76.7
78.2
71.2
1215.7
251.1
62.8
84.3
52.6
A
Appendix
Overview. In this appendix, we will first give more implementation and training details of our method. Then, we present more detailed ablation studies on several component designs. Next, we present more detailed visualization results. In the end, we discuss the limitations and future work.
A.1 More Implementation Details
Pre-training. Following LLaVA, OMG-LLaVA first performs pre-training learn the projector that projects visual tokens into the text space. During the pre-training stage, we freeze the visual encoder, OMG head, and LLM to train the visual projector for projecting visual tokens into the text space and to train the text projector for restoring the projected object-centric visual tokens to the segmentation embedding. The training data used in the pre-training stage is the same as that used in LLaVA. In this stage, the OMG-LLaVA is trained for epoch. The batch size is 256, with 32 per GPU, and the learning rate 0.001.
Supervised fine-tuning, During the instruction tuning stage, we freeze the visual encoder and OMG head, finetune the L.M using LoRA, and fully finetune the text and visual projectors. We train OMG-LLaVA for epoch on all instruction tuning datasets, including the LLaVA instruction tuning dataset, referring expression segmentation datasets, semantic segmentation datasets, grounded conversation generation datasets, mask-based visual prompt datasets, and point-based visual prompt datasets. The batch size is 128, with 16 per GPU, and the learning rate is 2e-4.
Inference details for each task. OMG-LLaVA generates answers token by token during the inference stage based on the given question. We use fixed template for the referring expression segmentation task to create the question: "Please segment (EXPRESSION) in this image." " In rare cases where OMG-LLa VA does not predict the [SEG] token, we use empty mask as the segmentation result. We use the fixed question for the grounded conversation generation task: "Could you please give me detailed description of the image? Please respond with interleaved segmentation masks for the corresponding parts of the answer." For other tasks, we remove special tokens such as <p>, </p>, and [SEG] from OMG-LLaVAY responses to ensure the answers contain only text.
A.2 More Experiment Results.
Evaluation results on image-level benchmarks. We evaluate OMG-LLaVA on several image-level benchmarks, including MME [32], MMBench [74], SEED-Bench [50], POPE [63], and A12D [43] benchmarks. The evaluation results are shown in Tab. 6. When jointly Co-training with image- level and pixel-level datasets. OMG-LLaVA achieves 1412, 47.9, 46.5, 80.0 and 42.9 on MME, MMBench, SEED-Bench, POPE and AI2D benchmarks, respectively. Compared with GLaMM [87], PixelLM [88], and LISA [49], OMG-LLaVA demonstrates significant performance improvement.
23

---

Table 8: Ablation study of projector for object-centric visual tokens.
refCOCO cloU
refCOCO+
refCOCOg cloU
re(C
Methods
Cross Atm.
Individual
gloU
cloU
gloU
gloU
METEOR
Baseline (MO)
74.5
75.9
63.6
65.9
68.7
71.0
13.6
M1
72.3
73.7
60.6
63.0
66.5
69.2
13.2
M2
72.3
74.1
60.8
63.5
65.4
68.6
13.1
Table 9: Ablation study on format of segmentation-based tasks. The first now represents the RES task using the fixed answer: "Sure, it [SEG]." and the GCG task using "<p> Expression </p> [SEG]. The second row represents the segmentation tasks' answer format being unified as "<p> Expression </p> [SEG].
refCOCO
retCOCO+ cloU
reCCOCOg cloU
GCG
Format
cloU
gloU
gloU
gloU
METEOR
AP50
mlOU
is (SFG].
75.5
76.5
65.8
67.8
70.6
72.3
13.8
27.3
64.4
<p> Expression </p> (SEG).
75.6
76.8
65.6
67.6
70.7
72.6
13.8
26.9
64.6
Table 10: Ablation study on segmentation embeddings.
refCOCO cloU
refCOCO+ cloU
refCOCOg
gloU
gloU
cloU
gloU
Last layer's hidden state Mean of all layers' hidden states
74.3
75.5
64.5
66.4
70.0
71.9
74.3
75,8
64,5
66.4
68.7
71.4
Concatenate of all layers' hidden states
70.0
71.1
61.8
63.6
62.3
64.4
When training only with the LLaVA [69] dataset, OMG-LLaVA achieves 1730, 67.5, 68.9, 89.7, and 61.7 on MME, MMBench, SEED-Bench, POPE, and A12D benchmarks. OMG-LLaVA outperforms VL-1155 [69] with 41, 3.0, 3.0, 5.1 on MME, SEED-Bench, POPE, and A12D benchmarks with the same training data.
Performance with diverse LLMs. We construct the OMG-LLaVA using diverse LLMs. The performance shown in Tab. 7. In addition InternLM2 [9], we have tried using PHI-3.8b [1] and Qwen2-7B [113], which achieved better performance on pixel-level and image-level benchmarks than InternLM2.. When using the stronger Qwen2-7B, OMG-LLaV achieves 76.7 cloU and 69. cloU on RefCOCO and RefCOCO+ benchmarks, and 1466.8, 62.8, 60.7, 84.3, 52.6, 37.2, and 66.4 on MME [32], MMBench [74], SEED-Bench [50], POPE [63], A12D [43], MMstar [15] and SQA [76] benchmarks.
A.3 More Detailed Ablation Studies.
Projector for object-centric visual tokens. We conducted ablation experiments on the vision projector. The results are shown in Tab. 8. We simple MLP projector as the bascline for object-centric visual tokens. When we added a cross-attention layer to the projector, performance on segmentation and visual prompt-based tasks decreased. This is because the introduction of the cross-attention layer caused the object-centric visual tokens to incorporate too many pixel-centric visual tokens, leading interference with the object information. Furthermore, when the projector for object-centric visual tokens generated from visual prompt input and object queries is not shared, performance declines segmentation and visual prompt-based tasks. Therefore, shared MI.P projector can effectively project object-centric visual tokens into the text space.
Answer format for segmentation-bast based tasks. In LISA [49], the response for the referring expres- sion segmentation task is fixed as "Sure, it is |SEG]." " However, this fixed answer may interfere with the cnatruttion-oollvvviii instructions. In GLaMM [87], for the grounded conversation generation task, the response is typically "<p> Expression </p> [SEG]." Since the "Expression" is flexible and variable, the LLM is less likely to overfit to a fixed response.
ability of the LLM, leading it to respond with "Sure, it is [SEG]." for new
We conduct ablation experiments the answer format for segmentation tasks, and the results are shown in Tab. 9. We find that unifying the answer format for segmentation tasks (including RES and GCG) as "<p> Expression </p> [SEG]" yields better performance. This more flexible answer
24

---

Express: the grass
the sand
the table
the smallest chair
beside the black sofa
Image
LISA
GLaMM
Ours
Figure 7: Qualitative comparison on the referring expression segmentation task. LISA uses the 13B LLM, while Gl aMM and our proposed OMG-ILaVA use the 7B I.I.M.
format not only achieves better performance in the referring expression segmentation task compared the fixed answer but also avoids the damage the LLM's instruction-follooinng ability.
Segmentation embeddings. We conduct ablation experiments on the generation strategy of segmen- tation embedding, and the results shown in Tab. 10. We explore whether the hidden states of the intermediate layers corresponding to the [SEG] token are helpful for segmentation. Compared to using the hidden states of the last layer of the [SEG] token the segmentation embedding, using the mean of the hidden states from all layers as the segmentation embedding resulted negligible im- provement refCOCO but led to significant performance drop the more challenging refCOCOg. Concatenating the hidden states from all layers the [SEG] token as the segmentation embedding resulted in significant performance drop across all RES tasks. Therefore, the hidden state of the last layer already contains sufficient features to generate the segmentation mask, and introducing hidden states from other intermediate layers does not yield better segmentation results.
A.4 More Visualization Results
Qualitative comparison with SOTA methods. We conduct qualitative comparisons and analyses on various tasks, including referring expression segmentation, grounded conversation generation, and image-based conversation, against the SOTA methods LISA [49] and GLaMM [87]. Fig. 7 shows the visualization results of the RES task for LISA, GLaMM, and our proposed OMG-LLaVA, OMG- LLaVA demonstrates more stable segmentation performance than LISA and GLaMM. Additionally, OMG-LLaVA exhibits better image and text understanding capabilities than LISA (13B) and GLaMM, illustrated in the fourth column with the example of "the smallest chair".
25

---

Fig. 8 shows the visualization results of the GCG task for GLaMM [87] and OMG-LLaVA, Our proposed OMG-LLaVA provides more detailed and accurate descriptions of the scene, such "light- house" and "bear." " Additionally, DMG-ILLaA demonstrates more stable segmentation capabilities, as seen in the "mountain" in the bottom-right corner image.
Fig. 9 shows the visualization results of the visual prompt-based description task for GLaMM [87] and OMG-LLaVA, VA. Compared GLaMM, OMG-LLaVA supports more flexible visual prompts, including point, box, and mask prompts. Additionally, OMG-LLaVA can generate more detailed object captions and demonstrate more accurate image understanding.
Fig. 10 shows the visualization results of the image-based conversation task for LISA [49], GLaMM [87], and OMG-LLaVA. Compared to LISA and GLaMM, OMG-LLaVA has stronger instruction-followee
ability. For example, when answering the question, "What is the number on the jersey of the athlete squatting on the ground?" both LISA and GI GLMMM incorrectly segmented "the jersey of the athlete squatting on the ground." " Compared to GI GLMMM, OMG-LLaVA can provide more detailed and accurate answers to user questions. Compared LISA, OMG-LLaVA demonstrates stronger scene understanding and reasoning abilities. For instance, in question 3 of Fig. 10, LISA gave an utterly incorrect answer despite using a larger LLM (13B).
Visualization results of RES. We provide additional visualization results of OMG-LLaVA on the RES task in Fig. 11. OMG-LLaVA demonstrates strong understanding of spatial relationships and human actions, enabling it to accurately and reliably segment the specified objects based on these descriptions. Furthermore, even without training on any reasoning segmentation data, OMG-LLaVa exhibits the ability to perform reasoning segmentation. As shown in Fig. 12, OMG-LLLaV can infer the target based on the question and accurately segment the corresponding object.
Visualization results of GCG. As depicted in Fig. 13, our method performs well on the grounded conversation generation task. OMG-LLaVA demonstrates strong scene understanding and object segmentation capabilities. Although some objects are overlooked, this is due to the omission many objects in the image captions of the Grandf dataset. We believe that using Suuherr data for training would result in even better performance for OMG-LLaVA.
Visualization results of visual prampts-based description. Fig. 14 shows more visualization results for the visual prompt-base basee description task. OMG-LLaVA supports input of point, box, and mask- based visual prompts and provides detailed descriptions. These descriptions include information about the objects and their relationships with other objects in the scene.
A.5 Limitation and Future Work Discussion
Limitations of OMG-LLaVA, Although OMG- achieves image-level, object-level, and pixel- level capabilities with a concise and elegant architecture, much room still exists for improvement. Firstly, joint training with pixel-level understanding data often leads to decreased image-level capa- bility, phenomenon widely observed in LISA [49] and GLaMM [87]. This challenge could be ad- dressed by organizing the data to eliminate this conflict. Secondly, due to the lack of multi- muati-graaullaitty segmentation capability in OMG-Seg, OMG-LLaVA cannot perform part-level segmentation. This challenge could be addressed using a more powerful and universal perception module by adding part-level visual inputs.
Several future directions can be explored with our new meta-architecture. We list two potential directions. including video and more instruction-tumm -otniinnoontttnngggggeeee data. Although OMG-Seg [58] can acquire the video inputs, OMG-LLaVA still cannot perform pixel-level spatial-temporal reasoning. This due to the lack of such datasets. Moreover, more instruction-tuning data involve more localization outputs, and multiple round conversations can be used to build a stronger MILM model. For example, we plan to use full GLaMM datasets [87] and more detection datasets [48; 89] for joint co-training as future work if more computation resources are available.
Future Works.
26

---

Image
GLaMM
A black truck is driving on dirt road- The truck is also parked on the same dirt road- The sky visible over the truck-
The image features a building standing The sky is overarching the building.
The image showcases living room with chair and a potted plant plated on rug window is situoted beside the chair and wall The is positioned on the wooden Aoor
Ours
A black Jeep is parked on dirt road, surrounded by a vast sandy landscape: The
The image features lighthouse with sky overhead- The sky also seen grass.
living room with large window and wooden fooor with rug. A potted plant is placed on the wooden floor A chair positioned beside the potted plant: The chair also situated on the wooden Aooor The wooden Roor is connacted to the wall and the window the door is embedded in wall-
image
scene is set in a desert clear
environment sky visible-
Image
GLaMM
image features dog standing grass The dog positioned front of mountain The grass is attached to the mountain
The image showcases book and cabinet placed The cabinet situated beside wall A light is also visible, positioned beside the book.
in black shirt and black parts is walking in the mountains
rug
Ours
The image shows bear standing grass- bear positioned in front of mountain- The sky is visible over the mounbain
image showcases vintage, wooden cabinet with built-in bookshelf: The cabine positioned light fixture is hanging from the ceiling The cobinet is also situated beside table, which is also placed on the rug- Book is on the wooden floor.
woman seands in frent waterfall mountainous region, surrounded by tall snowy cliffs and greenery, creating a beoutiful and serene
and
the table: The rug
Figure 8: Qualitative comparison the grounded conversation generation task.
27

---

OMG-LLaVA
GlaMM
coffee cup is prominently positioned the image- a large cup with a brown handle, and it's plated on a saucer The cup is filled with warm beverage, likely coffee, and it's the main focus the scene
of coffee
small, round pastry is located in the middle of the plate: It's the second pastry from the left and is the second row from the bottom.
macaroon on the left of the plate
small monkey seen the image, hanging from branch The monkey appears to be baby, possibly baby monkey, and is positioned towards the left side of the image.
monkey hanging from tree branch
A large, leafy tree is situated in the foreground of the image: It appears to be mature tree, possibly pine tree, and is lacated near house- The tree is the main focus of region, out against the backdrop house:
tree in front of house
stands
A silver car parked the side of the road, occupying significant portion of the image. The car's color is white, and it appears to be a sedan- The car is positioned in front of a blue house, which is also visible in the scene
grey car parked on the side of the road
Figure 9: Qualitative comparison the visual prompt-based description task.
28

---

Could you please give
detailed destription of the image?
OMG-LLaVA
image coptures baseball game prograss, with batber swinging wearing buseball uniform, and the cotcher positioned beliidd him, ready to cotch the boll- There are several other in the including few more players the field and couple spectators shanding
pitch-
spectators: 5ome bench is also visible the bachground, likey for players to rest or spoctators to sic-
further awapy
GLaMM
baseball player halding bet on field-
baseball proyress, with better standing the holding baseball bub- butter weating uniform appears
The
praparing for swing- in the scene, including tesmmotes and opponents, all facused on the game- total of six people con be seen the with some standing close to the batber and others positioned firther A baseball glove visible the averyone engaged in the game.
several people
LISA(13B)
held by the players- The abmosphere lively,
Whet the number the jersey of the athlete squatking ground?
OMG-LLaVA
The number on the jersey of the baseball player squatting on the ground $2:
GLaMM
(566)
Sure, the segmentation result [566).
LISA(13B)
Why the abhlete in the middle squabeing?
OMG-LLaVA
The atklete the middle squetting lifely in the process of catching o ball: baseball, when player is on base, the opposing team tries to score runs hitbing the ball and running around the bases: The catcher's role is runners, proventbing botter, ready to catch the ball if the batéer misses or if the ball is his into the dirt-
the ball throum image, cotcher squatting in front
pitcher
from advancing
GLaMM
The athlete in the middle squabbing the coécher, who is preparing to catch the ball thrown by the pitcher-
basebell player in the middle is squabeing because he preparing to hie the ball during the game- As batder, his position is crucial for making contace with the preched ball and potentially scoring runs for team- squetting, the player lowers his center ability quickly
LISA(13B)
increases seability, and improves pitch- This position allows player generote more power io his swing, which can result stronger hit and patentially longer distances for the ball to be hit-
Figure 10: Qualitative comparison the image-based conversation task.
29

---

Please segment who holding the baseball bat.
Please segment the person raising the arm.
Please segment the person with red hat.
Please segment the tree in the background.
Please segment the fence.
Please segment the right girl.
Please segment out the pizza that is closer to the mon.
Please segment the middle beer bottle
Please segment the person holding fork
Figure 11: More visualization results of referring expression segmentation.
Please segment out what is separating the spectators from the athletes.
Please segment out what the man is most likely eat fill his stomach.
Please segment out what this woman will to the tennis ball.
Please segment out the food with the most fiber.
Please segment out what helping the woman block the sunlight.
Please segment out what help the little boy feel cooler.
Figure 12: Visualization results of reasoning segmentation.
30

---

The image shows zebra standing on dirt- A bird is also present on dirt- The dirt is attached to rock-
The image shows a bench situated on snow The bench located front tree- sky visible the
shateboarder grinding his board ramp
The image shows placed dining table The cake is attached to bow/-
giraffe standing behind another boy blue uniform kicks giraffe- giraffe standing in front of another giraffe
soccer ball
Figure 13: More visualization results grounded conversation generation.
Point-prompt
Box-prompt
Mask-prompt
The purple umbrella in the image large umbrella, providing protection from the falling snow. It is held by the man, who is walking alongside the woman. The umbrella is likely
The athlete wearing blue helmet is the batter in the baseball game. He is in the process of swinging the bat, attempting to hit the ball that is coming towards him. The batter is likely focused on the ball's trajectory and timing his swing to make contact with it.
The sheep in the image large, wooly animal with thick coat of wool. It is lying down the grass, appearing to be relaxed and comfortable. The sheep is
waterproof. allowing them stay dry while they walk through the snow.
being sheared by man, who is carefully removing the wool from its body.
Figure 14: More visualization results of grounded conversation generation.
31


Reviews:
[{"review_id": "kAJwziFeRO", "reviewer": "Reviewer_czBJ", "summary": "This work proposes, OMG-LLaVA, a unified framework for image-level, object-level, and pixel-level vision-language comprehension. In particular, OMG-Seg, a universal image segmentation model, is integrated with a LLaVA-like multimodal large language model (MLLM), so that various image-level (e.g., image captioning, visual question answering), object-level (e.g., promptable segmentation, region captioning), and pixel-level (e.g., referring expression segmentation, grounded conversation generation) vision-language tasks can be performed. By using one unified visual encoding and decoding module for objects and pixel masks, OMG-LLaVA poses a simple design and shows competitive performance compared with prior models.", "questions": "In addition to the weaknesses mentioned above, there are two minor questions:\n\n1. What is the “pixel shuffle operator” (Line 171)?\n\n2. To be consistent with Equation 2, should the segmentation mask be written as $\\mathcal{M}\\in\\mathbb{R}^{HW\\times N_q}$ instead of $\\mathcal{M}\\in\\mathbb{R}^{N_q\\times HW}$?", "limitations": "The authors have discussed the technical limitations, but have not discussed potential societal impacts. There is no explanation about why this work has no societal impacts, either.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Overall, OMG-LLaVA’s architecture design is more unified and concise when compared to prior models like LISA and GLaMM where more than one visual encoder is applied.\n\n2. OMG-LLaVA is able to perform a wide range of vision-language comprehension tasks with one model.\n\n3. The method description is generally clear and easy to follow.", "weaknesses": "1. [Performance] Although this work claims OMG-LLaVA as a generalist model, it cannot outperform prior state-of-the-art models on any specific task. This performance difference would weaken the applicability. In Table 3, the performance should also be compared with GLaMM (which has significantly better performance than LISA).\n\n2. [Technical contribution] Overall, OMG-LLaVA seems a direct integration of OMG-Seg and LLaVA. The most novel design in this work seems to be the perception prior embedding, which enables the connection between OMG-Seg and the object visual tokens in LLaVA. The technical contribution is thus somewhat limited. It would be great if the authors could better discuss the novel technical contributions of this work.\n\n3. [Image-level tasks] Although this work claims OMG-LLaVA can unify image-level, object-level, and pixel-level reasoning and understanding tasks, there is no quantitative evaluation in terms of image-level tasks such as VQA.\n\n4. [Frozen perception module] In OMG-LLaVA, the “universal perception module” (OMG-Seg) is frozen (Figure 2e) to preserve its learned knowledge. However, Table 3 shows that allowing the OMG decoder to be tuned can actually improve the referring segmentation performance. It is not explained why this module should be frozen in other tasks, or whether this practice would bring further gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. [Performance] Although this work claims OMG-LLaVA as a generalist model, it cannot outperform prior state-of-the-art models on any specific task.**\n\n**A**: We have updated the performance of OMG-LLaVA, as shown in Table R1. OMG-LLaVA outperforms LISA, PixelLM, and GSVA on the RES benchmarks. GLaMM used the GranD dataset for pretraining. The GranD pretrain dataset contains 11M images, far exceeding the GranD-f dataset used by OMG-LLaVA, which only includes 210K (0.21M) images. Nevertheless, OMG-LLaVA still surpasses GLaMM on the GCG benchmark by 0.6 CIDEr, 1.4 AP50, and 0.1 mIoU.\n\nAdditionally, OMG-LLaVA achieves substantial performance gains over GLaMM, LISA, and PixelLM on image-level benchmarks (Table R2). For instance, when jointly co-training with image-level and segmentation-based datasets, OMG-LLaVA achieves 1412 on MME benchmark, outperforming LISA, PixelLM and GLaMM with 1410, 968 and 1389, respectively. When only trained on the LLaVA dataset, OMG-LLaVA outperforms LLaVA-1.5 by impressive margins of 41, 3.0, 3.0, and 5.1 points on MME, SEED-Bench, POPE, and AI2D, respectively.\n\n---\n\n**Q2. [Technical contribution] Overall, OMG-LLaVA seems a direct integration of OMG-Seg and LLaVA.**\n\n**A**: Our most significant contribution is introducing an elegant MLLM architecture that comprises only a perception model and an LLM, capable of understanding and reasoning at the image, object, and pixel levels. The frozen perception decoder can simultaneously encode visual prompts and decode [SEG] tokens. However, our experiments and those in LISA have shown that the frozen perception decoder cannot effectively decode [SEG] tokens. The proposed perception prior embedding is the key component enabling our meta-architecture to work well. Thus, we have three main contributions.\n\n- First, we propose an elegant MLLM architecture with a single visual encoder, a perception head, and an LLM, which can simultaneously handle image, object, and pixel-level tasks.\n\n- Secondly, we introduce perception prior embedding to help the LLM better align pixel-centric visual tokens with object-centric visual tokens, allowing the LLM to generate object-centric tokens that the frozen perception decoder can decode based on the input embedded perception prior.\n\n- Thirdly, we construct OMG-LLaVA, which achieves SOTA performance on pixel-level tasks and comparable accuracy to SOTA on object-level tasks. Additionally, OMG-LLaVA significantly outperforms versatile MLLMs like LISA, PixelLM, and GLaMM in image-level tasks.\n\nFrom these aspects, we argue that our work is not the direct integration of OMG-Seg and LLaVA. \n\n---\n\n**Q3. [Image-level tasks] Although this work claims OMG-LLaVA can unify image-level, object-level, and pixel-level reasoning and understanding tasks, there is no quantitative evaluation in terms of image-level tasks such as VQA.**\n\n**A**: Table R2 presents our model's performance on various image-level benchmarks, including MME, MMBench, SEED-Bench, POPE, and AI2D. Jointly co-training on segmentation and image-level data can detrimentally impact an MLLM's performance on image-level benchmarks. However, OMG-LLaVA mitigates this negative impact thanks to its perception prior embedding. OMG-LLaVA significantly outperforms GLaMM, PixelLM, LISA, and LaSagnA on these image-level benchmarks. OMG-LLaVA scores 1412 on MME and 47.7 on MMBench, surpassing GLaMM with 1389 on MME and 11.1 on MMBench. By training solely on image-level data, OMG-LLaVA achieves superior performance over LLaVA 1.5 on benchmarks such as MME, SEED-Bench, POPE, and AI2D, attributed to its perception prior embedding.\n\n---\n\n**Q4. [Frozen perception module] In OMG-LLaVA, the “universal perception module” (OMG-Seg) is frozen (Figure 2e) to preserve its learned knowledge. However, Table 3 shows that allowing the OMG decoder to be tuned can actually improve the referring segmentation performance. It is not explained why this module should be frozen in other tasks, or whether this practice would bring further gains.**\n\n**A**: During finetuning, the OMG decoder is duplicated and unfrozen to better decode the [SEG] tokens generated by the LLM. The encoding of object-centric tokens and visual prompts continues to be handled by the original frozen OMG decoder. We also add the experiment of keeping the OMG decoder frozen when finetuning the model. As shown in the last two lines of Table R1, finetuning on the RES datasets while keeping the OMG decoder frozen results in a slight performance degradation compared to the case where an additional OMG decoder is duplicated and unfrozen (78.0 mIoU *vs.* 77.2 mIoU). However, the performance still significantly surpasses LISA and PixelLM. Moreover, we directly unfreeze the original OMG decoder without duplicating it. In that case, OMG-LLaVA cannot work well due to the reliance of object-centric tokens on a stable mask segmentation capability.\n\n---\n\n**Q5. What is the “pixel shuffle operator” (Line 171)?**\n\n**A**: This paper leverages the pixel shuffle operator to downsample image features. By flattening $S \\times S$ neighboring pixel features, the operator reduces the image size from $(H, W, C)$ to $(H/S, W/S, C \\times S^{2})$.\n\n---\n\n**Q6. To be consistent with Equation 2.**\n\n**A**: Thank you for the suggestion. We have modified $M \\in R^{N_{q}\\times HW}$ to $M \\in R^{HW \\times N_{q}}$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "117F6hxOyG", "reviewer": "Reviewer_ck7P", "summary": "This paper presents OMG-LLaVA, which facilitates pixel-level, object-level and image-level understanding tasks within a unified framework. Within the OMG-LLaVA, an OMG decoder and perception prior embedding approach are proposed to enhance object-centric comprehension. Comparison with state-of-the-art methods prove the effectiveness of the proposed method.", "questions": "Since comprehension capabilities ranging from image-level to object-level are established, does OMG-LLava possess composite capabilities such as visualizing the intention of LLM during a conventional conversation in the form of segmentation masks? This is very helpful of understanding the working mechanism of LLMs.", "limitations": "See weakness.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper integrates several segmentation tasks and object-level visual comprehension ability within a unified framework. Using a high-resolution image encoder and compressing the visual tokens to save computation resources is reasonable.\n2. The design of perception prior embedding is interesting which can integrate object-level cues with visual tokens.", "weaknesses": "1. The proposed method is not very elegant as too many designs customized for different tasks are introduced in the framework. For different tasks (e.g., image captioning, image segmentation and visual prompt based comprehension), different workflows should be conducted.\n2. Some important technical details are omitted. For example, to encourage different object queries to attend to different objects, hungarian matching algorithm should be applied to assign labels for each individual object query, especially for COCO segmentation where multiple objects are supposed to be simultaneously segmented. Besides, to generate masks from object queries, the mask decoder should be carefully designed to generate high-quality masks, however, this paper only mentioned a simple FFN is utilized. Is it reasonable? More details should be provided. \n3. Some important experimental results are not provided. For example, experimental results on prevalent VQA benchmarks like MMBench, MME, SEEDBench, etc. should be provided to illustrate the general-purpose conversational capability of OMG-LLava. Whether introducing these segmentation data can hurt the VQA capabilities of the model? Overall, only segmentation and simple captioning qualitative results are reported, which are much less than the supported functions of the proposed OMG-LLava.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The proposed method is not very elegant as too many designs customized for different tasks are introduced in the framework.** \n\n**A**: OMG-LLaVA boasts an elegant model architecture and more streamlined workflows across tasks than GLaMM. For instance, in GLaMM, visual prompt encoding and segmentation mask decoding employ entirely different workflows and models. However, in OMG LLaVA, visual prompt encoding and segmentation mask decoding are unified into a single OMG Decoder to generate object-centric queries. Maintaining a consistent workflow while unifying many tasks is extremely challenging and currently difficult. For example, mask decoding and visual prompt encoding are hard to implement solely with LLMs without relying on other modules. We believe that in the future, diverse tasks will be unified under elegant architectures and shared workflows, and OMG-LLaVA is a significant step toward this goal.\n\n---\n\n**Q2. Some important technical details are omitted. For example, to encourage different object queries to attend to different objects, a Hungarian matching algorithm should be applied to assign labels for each individual object query, especially for COCO segmentation, where multiple objects are supposed to be simultaneously segmented. Besides, to generate masks from object queries, the mask decoder should be carefully designed to generate high-quality masks, however, this paper only mentioned a simple FFN is utilized. Is it reasonable? More details should be provided.** \n\n**A**: There is a misunderstanding. The segmentation ability is derived from the frozen perception model (OMG-Seg), not the LLM's output. OMG-Seg is a component of OMG-LLaVA, so there is no need for the LLM to reiterate these perception results.\nSecondly, the mask decoder of OMG-LLaVA is not a simple FFN layer. The last layer's hidden states of the [SEG] token first pass through a text projector (an MLP layer) and then are fed into the frozen mask decoder (OMG-Seg head) to obtain segmentation masks. During training, the mask decoder remains frozen while the text projector is trained. Since the perception results have already been embedded into the LLM's input via perception prior embedding, generating segmentation queries is not difficult. As shown in Table R4 in our PDF file, we also experimented with adding extra cross-attention layers in the text projector, but the performance was similar to using a simple MLP.\n\n---\n\n**Q3. Some important experimental results are not provided. For example, experimental results on prevalent VQA benchmarks like MMBench, MME, SEEDBench, etc. should be provided to illustrate the general-purpose conversational capability of OMG-LLava.**\n\n**A**: We have evaluated the performance of OMG-LLaVA on multiple image-level benchmarks, including MMbench, MME, SEED-Bench, POPE, and AI2D. The results are presented in Table R2. When jointly co-trained with both image-level and segmentation-based data, OMG-LLaVA significantly outperformed LISA and GLaMM on all image-level benchmarks. However, co-training with segmentation-based data significantly decreases performance on image-level benchmarks, although OMG-LLaVA can mitigate this issue to a large extent through perception prior embedding. We also evaluate the performance of OMG-LLaVA trained solely on the LLaVA dataset on image-level benchmarks. OMG-LLaVA outperformed LLaVA 1.5 on MME, SEED-Bench, POPE, and AI2D by 41, 3.0, 3.0, and 5.1 points, respectively.\n\n---\n\n**Q4. Since comprehension capabilities ranging from image-level to object-level are established, does OMG-LLava possess composite capabilities such as visualizing the intention of LLM during a conventional conversation in the form of segmentation masks? This is very helpful of understanding the working mechanism of LLMs.**\n\n**A**: As shown in Figure R6, OMG-LLaVA can visualize LLM intentions through segmentation masks. While the ability to visualize LLM intentions in diverse conversations was not explicitly included in OMG-LLaVA's training data, the model learned this capability by training on referring expression segmentation data and image-level conversation data. This ability is crucial for AI assistants, as it significantly enhances the user experience.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6Hq5Mu2oE", "reviewer": "Reviewer_Timw", "summary": "This paper proposes OMG-LLaVA, an elegant MLLM framework. OMG-LLaVA achieves pixel-level, object-level, and image-level understanding and reasoning with only a perception model and a language model. OMG-LLaVA exhibits more comprehensive capabilities than current MLLMs, such as LLaVA, Osprey, and GLaMM. The experimental results on several benchmarks demonstrate that OMG-LLaVA achieves performance comparable to SOTA methods.", "questions": "1 The paper needs to include more ablation experiments to help readers better understand the work. For example, what would happen if the visual projector design shared an MLP for pixel- and object-centric visual tokens?\n2 Would the performance of OMG-LLaVA improve with a larger LLM?", "limitations": "The authors pointed out the limitations of OMG-LLaVA, including performance degradation on image-level tasks and lack of capability for part-level segmentation. The latter can be addressed by replacing the perception model with a more powerful one.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The motivation is very clear and easy to follow.\n2 The OMG-LLaVA is a simpler and more efficient MLLM framework compared to current MLLMs, and it has broader capabilities such as universal image segmentation and supporting comprehensive visual prompt inputs.\n3 The proposed perception prior embedding strategy is crucial and efficient for providing perception results for LLMs. This approach allows LLMs to achieve grounded outputs similar to selecting object tokens from inputs, which is more reasonable and performs better on segmentation tasks than LISA, PixelLM, and GLaMM.\n4 OMG-LLaVA unifies the visual prompt embeddings and segmentation embeddings as object-centric visual tokens, effectively shortening the training phases.", "weaknesses": "1 The paper needs to include more ablation experiments to help readers better understand the work. For example, what would happen if the visual projector design shared an MLP for pixel- and object-centric visual tokens?\n2 Would the performance of OMG-LLaVA improve with a larger LLM?\n3 The authors should include more MLLMs in Table 1 for a more comprehensive comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The paper needs to include more ablation experiments to help readers better understand the work. For example, what would happen if the visual projector design shared an MLP for pixel- and object-centric visual tokens?**\n\n**A:** Table R4 presents the results of ablation studies on the projector. We observed a substantial performance degradation when pixel-centric and object-centric tokens shared a projector. This phenomenon can be attributed to the distinct semantic and perception information encoded in these two types of tokens. Sharing a projector results in detrimental interference between them. Furthermore, our experiments with additional cross-attention layers in the projector did not yield any performance benefits. For the object-centric tokens, specifically visual prompt tokens and object queries, sharing a projector resulted in marginal performance gains for both segmentation and visual prompt understanding tasks.\n\n---\n\n**Q2. Would the performance of OMG-LLaVA improve with a larger LLM?**\n\n**A:** We have conducted experiments with OMG-LLaVA using various LLMs, and the results are presented in Table R3. Our findings indicate that the performance of OMG-LLaVA increases with the strength of the LLM. Due to time limitations, the performance of OMG-LLaVA models based on larger LLMs, such as Yi-34b, will be reported in the coming days.\n\n---\n**Q3. The authors should include more MLLMs in Table 1 for a more comprehensive comparison.**\n\n**A:** We appreciate your feedback. To further emphasize the unique capabilities of OMG-LLaVA, we have compared more MLLMs (about 25 MLLMs). The result demonstrates that OMG-LLaVA provides the most comprehensive capabilities while employing a single visual encoder.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t5fhq87ikl", "reviewer": "Reviewer_UauW", "summary": "This work presents a new multi-modal large language model, named OMG-LLaVA. This model builds on previous work OMG-Seg and combines a LLM (LLaVA-like) into one simple framework. Compared with previous MLLMs, this work unifies lots of image-level, object-level, and pixel-level segmentation and reasoning tasks in one shot.", "questions": "See the weakness, I would raise the score if all questions are well solved.", "limitations": "Please see weaknesses.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1, The motivation and goal are interesting and ambitious. The proposed architecture is novel and interesting to me. Although built from previous universal segmentation network, the authors show a simple but effective way to connect LLM and dense visual perception model.\n2, The proposed method only uses one image encoder and one image decoder, which is clean and easy to follow. Moreover, the authors propose a new connection module, named perception prior module, which also works effective in connecting object query into LLM for further processing.\n3, Compared to several works, including PixelLM and GLaMM, OMG-LLaVA has comprehensive functionality but better performance. Moreover, the entire design is simple and elegant. \n4, Overall writing is good and is easy to follow.", "weaknesses": "Overall, I think this paper is good and elegant, compared with previous combined approach. However, there are several details should be added for the better draft.\n1, More detailed designs or ablation should be carried out for perception prior embedding design. The current draft only shows the effectiveness of proposed approach.\n\n2, More detailed discussion on meta-architecture design should be \nOne question is why you fix the image decoder why not use another decoder with the same architecture but with the same pre-trained weights.\nAlthough this operation can increase the parameter costs, I wonder whether joint sharing one decoder can have mutual effects.\nThus, more detailed experiments on meta-architecture design should be added.\n3, No parameter and Gflops analysis. For example, the author claim they only use one image encoder, one decoder and one LLM. Compared with previous combined approach (GLaMM), this advantage is not well explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. More detailed designs or ablation should be carried out for perception prior embedding design.** \n\n**A**: Thanks for your suggestion. We have added more detailed ablation studies on perception prior embedding strategy. We have conducted ablation studies on different strategies for assigning perception embeddings to each pixel token, and the results are shown in Table R5. The best results were achieved by assigning the corresponding object query to each pixel as the perception embedding to be embedded, using the SoftMax method. The strategy of generating perception embeddings through L1 normalization led to performance degradation, but it still brought a significant performance improvement compared to not using the perception prior embedding strategy.\n\n---\n\n**Q2. More detailed discussion on meta-architecture design should be One question is why you fix the image decoder why not use another decoder with the same architecture but with the same pre-trained weights. Although this operation can increase the parameter costs, I wonder whether joint sharing one decoder can have mutual effects. Thus, more detailed experiments on meta-architecture design should be added.**\n\n**A**: Using another decoder with the same architecture and loaded with the same pre-trained weights to generate segmentation results in a marginal performance improvement, as shown in the table below. Undoubtedly, using an additional decoder can achieve better performance due to the introduction of more trainable parameters. However, the performance gain is marginal, so we chose to adopt a frozen OMG Decoder to achieve a balance between performance and parameters.\n\n| Decoder  |      | refCOCO |       |      | refCOCO+ |       | refCOCOg |      |        | GCG   |      |      |\n| -------- | ---- | ------- | ----- | ---- | -------- | ----- | -------- | ---- | ------ | ----- | ---- | ---- |\n|          | Val  | TestA   | TestB | Val  | TestA    | TestB | Val      | Test | METEOR | CIDEr | AP50 | mIOU |\n| unfrozen | 78.0 | 80.3    | 74.1  | 69.1 | 73.1     | 63.0  | 72.9     | 72.9 | 14.5   | 38.5  | 28.6 | 64.7 |\n| frozen   | 77.2 | 79.8    | 74.1  | 68.7 | 73.0     | 61.6  | 71.7     | 71.9 | 14.5   | 37.5  | 28.9 | 64.6 |\n\n---\n\n**Q3. No parameter and Gflops analysis. For example, the author claim they only use one image encoder, one decoder and one LLM. Compared with previous combined approach (GLaMM), this advantage is not well explored.**\n\n**A**: We have statistics on the parameters of the main components of OMG-LLaVA, such as the LLM and visual encoder, and the results are shown in below table. The visual encoder of OMG-LLaVA is only 0.2B, much smaller than the 0.9B of LISA and GLaMM. The number of vision tokens significantly impacts the computational cost of the LLM. Compared to GLaMM, which uses 576 visual tokens, OMG-LLaVA only requires about 276 visual tokens, which is only 50% of GLaMM.\n\n| Methods   | LLM  | Visual Encoder             | Image Size                 | Visual Tokens                            |\n| --------- | ---- | -------------------------- | -------------------------- | ---------------------------------------- |\n| LISA      | 7B   | VIT-L(0.3B) \\& VIT-H(0.6B) | (224, 224) \\& (1024, 1024) | 256                                      |\n| GLaMM     | 7B   | VIT-L(0.3B) \\& VIT-H(0.6B) | (336, 336) \\& (1024, 1024) | 576                                      |\n| OMG-LLaVA | 7B   | ConvNext-L(0.2B)           | (1024, 1024)               | 256(Pixel-centric) \\& 20(Object-centric) |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Responses

---
Dear Reviewers,

We thank all the reviewers for the detailed suggestions. All reviewers acknowledge the technical contributions of our work, including the new unified designs for MLLM and comprehensive benchmark evaluation. We listed additional important experiments and answered common questions here. We will respond individually to the corresponding reviewers for the detailed questions of each reviewer. 

---

## 1. Evaluation on image-level benchmarks (R#czBJ and R#ck7P)

We have evaluated the performance of OMG-LLaVA on multiple image-level benchmarks, including MMbench, MME, SEED-Bench, POPE, and AI2D. The results are presented in Table R2 (please refer to the submitted pdf). When jointly co-trained with both image-level and segmentation-based data, OMG-LLaVA significantly outperformed LISA, GLaMM, PixelLM, and LaSagnA on all image-level benchmarks. For example, OMG-LLaVA scores 1257 on MME and 45.7 on MMBench, surpassing GLaMM with 1234 on MME and 8.9 on MMBench. We list these results in the pdf file.

However, co-training with segmentation-based data significantly decreases performance on image-level benchmarks (This is also verified in previous work, PixelLLM (https://arxiv.org/pdf/2312.02228), see the last page of their arxiv.), although OMG-LLaVA can mitigate this issue to a large extent through perception prior embedding. We also evaluate the performance of OMG-LLaVA trained solely on the LLaVA-1.5 dataset on image-level benchmarks. When using the same LLM, OMG-LLaVA outperformed LLaVA 1.5 on MME, SEED-Bench, POPE, and AI2D by 41, 3.0, 3.0, and 5.1 points, respectively. 

---

## 2. Performance of OMG-LLaVA (R#czBJ and R#Timw)

We have fixed some bugs and updated the performance of OMG-LLaVA, as shown in Table R1 (please refer to the submitted pdf). OMG-LLaVA achieves 78.0, 69.1 and 72.9 mIoU on refCOCO, refCOCO+ and refCOCOg benchmarks, outperforming LISA, PixelLM, and GSVA on the RES benchmarks. GLaMM uses the GranD pre-train dataset for pretraining. The GranD pretrain dataset contains over 11M images, **far exceeding** the GranD-f dataset used by OMG-LLaVA, which **only** includes 210K (0.21M) images. Nevertheless, OMG-LLaVA still surpasses GLaMM on the GCG benchmark by 0.6 CIDEr, 1.4 AP50, and 0.1 mIoU.

We have conducted experiments with OMG-LLaVA using stronger LLMs, such as Phi3 and Qwen2, and the results are presented in Table R3. With the stronger LLM Qwen2-7B, OMG-LLaVA performs better on pixel- and image-level benchmarks.

In addition, we will consider adding more segmentation and caption datasets for co-training, which will serve as our future work.

---

## 3. The contributions of OMG-LLaVA (R#czBJ and R#ck7P)

Our most significant contribution is introducing an elegant MLLM architecture that comprises only a perception model and an LLM, capable of understanding and reasoning at the image, object, and pixel level. This is a systematic contribution. To the best of our knowledge, we are the first open-source system to fulfill that goal in the MLLM domain using one image encoder, one image decoder, and one LLM. The frozen perception decoder can simultaneously encode visual prompts and decode [SEG] tokens. However, our experiments and those in LISA have shown that the frozen perception decoder cannot effectively decode [SEG] tokens. To better adapt the segmentation model to LLM, we propose the perception prior embedding module, which is the key component enabling our meta-architecture to work well.

In addition, compared with existing methods, OMG-LLaVA boasts an elegant model architecture and more streamlined workflows. For instance, in GLaMM, visual prompt encoding and segmentation mask decoding employ entirely different workflows and models. In OMG LLaVA, visual prompt encoding and segmentation mask decoding are unified into a single OMG Decoder to generate object-centric queries. Maintaining a consistent workflow while unifying all tasks is extremely challenging and currently difficult. For example, mask decoding and visual prompt encoding are only possible with LLMs that rely on other modules.  OMG-LLaVA significantly outperforms versatile MLLMs like LISA, PixelLM, and GLaMM in image-level tasks.

---

## 4. More ablation studies (R#Timw and R#UauW)

Following the reviewer's suggestion, we add additional ablation studies for better clarity. We have included ablation studies on the projector and perception prior embedding. The results are shown in Tables R4 and R5.

Table R4 presents the results of ablation studies on the projector. We observed a substantial performance degradation when pixel-centric and object-centric tokens shared a projector. This phenomenon can be attributed to the distinct semantic and perception information encoded in these two types of tokens. Sharing a projector results in detrimental interference between them. Furthermore, our experiments with additional cross-attention layers in the projector did not yield any performance benefits. For the object-centric tokens, specifically visual prompt tokens and object queries, sharing a projector resulted in marginal performance gains for both segmentation and visual prompt understanding tasks.

We have conducted ablation studies on different strategies for assigning perception embeddings to each pixel token, and the results are shown in Table R5. The best results are achieved by assigning the corresponding object query to each pixel using the SoftMax operator. The strategy of generating perception embeddings through L1 normalization causes performance degradation, but it still brought a significant performance improvement compared to not using the perception prior embedding strategy.

---

We will fix all the possible issues and improve the manuscript. To address all your concerns and questions, we prepared a comprehensive response, including additional experiments where necessary. If you have any further questions, please don't hesitate to interact with us.

Best regards

Meta-review: This paper proposes OMG-LLaVA a framework for combining pixel-level vision understanding with reasoning abilities.  The paper received 1 strong accept, 1 accept, 1 borderline reject, and 1 borderline accept recommendations from reviewers. Positive points included the good motivation, clean and effective approach, generally clear writing, and good results. Negative points included lack of details for some approach components, limited technical novelty (especially by one reviewer), and missing ablation studies. Many of these concerns were adequately addressed by the rebuttal. There was an extensive post rebuttal discussion between the reviewers about the pros and cons of the paper. Overall, after carefully considering the paper, rebuttal, and discussions, the ACs feel that the paper makes a solid contribution, in which the positives outweigh the negatives, and recommend accept. It is recommended that the authors incorporate the rebuttal points into the final version.
Decision: Accept (poster)