OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

Authors: tao  zhang
xiangtai  li
hao  fei
haobo  yuan
shengqiong  wu
shunping  ji
chen change loy
shuicheng  yan
wuhan  university
skywork  ai
s-lab  ntu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:42:41

Abstract: Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

Main paper: 
 introduction: 
 With the development of transformer models [94; 6; 93; 40; 71; 92; 64; 126; 49; 87; 10; 19; 58] , recent works in both natural language processing (NLP) and computer vision raise one common trend: adopting one unified model to solve multiple tasks. For example, large language models (LLMs) [93; 40; 92] adopt scale-up models to solve multiple NLP tasks and achieve better results than previous expert models. In vision, we have also seen a similar trend [19; 58; 100; 99; 47; 112] , adopting one model to solve multiple tasks or sub-tasks, including detection, segmentation, video analysis, low-level vision, pose estimations, and more tasks. Different methods adopt different transformer designs, including visual-in-context learning [99; 100] , unified decoder [19; 58] , and unified tokenizer [86; 16; 58] . In summary, benefiting from the scalability and flexibility of the transformer, adopting one model for all tasks has made a great progress [19; 71; 72; 70; 126; 88; 87] . Please segment the train that is furthest from the camera.Please segment what helps the person to stand on the ocean.The image features a red double-decker bus. This bus is parked on a road. The bus is positioned in front of trees, with the sky overhead.A zebra is standing on the grass. The zebra is drinking from a puddle.A book is placed on top of a laptop. It is titled "Europe". The book is positioned on the right side of the laptop.Instruction: Please describe the <Region>. Answer: A man, dressed in a green shirt, is sitting on the floor. The image features a white pickup truck parked on a street. The truck is a large, full-size vehicle, and it is parked in front of a residential home. The truck is positioned on the street, and it appears to be parked in front of the home's driveway. The truck is also parked next to a curb, which is a common feature on streets in many cities.Instruction: What is the number on this man's shirt? Answer: The number on the man's shirt is 11.Instruction: What sport is this man playing? Answer: The man is playing baseball. Instruction: Why is there a red stain on this man's pants? Answer: The red stain on the man's pants could be due to several reasons. One possibility is that the stain is dirt or mud from the playing field, which can easily get on the player's pants when they slide or dive into the dirt while playing baseball.Segmentation and Captioning 1. A small, young lamb, sitting on the grass in a field 2. A large, adult sheep, standing in the grassy area of the field 3. A large grass filed.Instruction: Please describe the <Region> and <Region>.Answer: The man is swinging a bat.In the image, a woman is standing next to a zebra, holding its reins. She is also petting the zebra, showing affection towards the animal.The scene takes place in a zoo, with the woman standing in front of a tree. The zebra is standing on dirt, and the woman appears to be enjoying her interaction with the animal.Instruction: How many scissors on the table? Answer: Three. Instruction: What the color of the middle scissor? Answer: It is black. Several works adopt similar designs [3; 13; 64; 18; 25] , and all of them are termed Multi-modal Large Language Models (MLLMs). After that, most research focuses on improving MLLM benchmarks in various ways, including increasing data sizes [14; 18; 70] and enhancing the visual encoders [131; 24; 18] and visual resolutions [110; 18; 65; 25] . However, LLaVA-like models cannot output precise location information since they only carry out image-level analysis. Thus, recent works [126; 133; 11; 82; 13; 119; 128; 87; 67] try to fill this gaps by adding extra detection models for object level analysis, mask decoder for pixel-level analysis, visual prompts, and also propose task-specific instruction tuning with various datasets. By providing extra detection data and a decoder, the updated MLLMs can perform localization output. However, these models [135; 96; 49] are specifically tuned on specific tasks, losing the ability of LLaVA for image level analysis, such as caption and visual question answering. Meanwhile, several works [126; 49; 87; 78] adopt LLMs as agents to collaborate with various visual models or generation models. Despite the works being simple and effective, the inference and parameter costs are huge due to the multiple visual encoders and decoders. Moreover, there are no specific designs for task unification.Motivated by the previous analysis, we ask one essential question: Can we bridge image-level, object-level, and pixel-level tasks into one MLLM model with only one LLM, one visual encoder, and one visual decoder? Back to the universal perception models, we can leverage these models to help us build a stronger MLLM to unify three-level inputs, including image, object, and pixel levels.In particular, we adopt OMG-Seg [58] as our universal perception model due to its simplicity and effectiveness in various segmentation tasks.In this work, we present OMG-LLaVA, an elegant MLLM that bridges image-level, object-level, and pixel-level reasoning and understanding tasks in one model. We preserve the basic pixel-level segmentation ability of OMG-Seg by freezing the visual encoder and decoder, as shown in the bottom left of [49; 88] , etc., (d) MLLMs with both object-level and pixel-level capabilities but with a very complex system, such as [87] , (e) OMG-LLaVA's architecture, which possesses an elegant and simple design while having image-level, object-level, and pixel-level capabilities.image-level understanding as LLaVA, including caption and conversation, where most MLLMs for grounding lose such ability. In addition, OMG-LLaVA also supports the visual prompts as inputs, which results in object level understanding, such as visual prompt-based conversation and region-level captions. We achieve all these abilities using one LLM, one encoder, and one decoder.In particular, to better encode the visual segmentation outputs, we propose a perception prior embedding module to absorb the object queries into object-centric visual tokens, which are the inputs of LLMs. We present a unified instruction formation strategy, which lets the model accept visual images, texts, and visual prompts as inputs and generate the response of text, segmentation tokens, segmentation masks, and labels. Following the LLaVA [71] , we adopt pretraining and instruct tuning pipelines. Extensive experiments show the effectiveness of our components and training strategy. In addition to visual segmentation, OMG-LLaVA can also achieve good enough performance on 6 datasets, including COCO panoptic segmentation, VIPSeg video panoptic segmentation, refCOCO, refCOCO+, refCOCOg referring expression segmentation, GranDf grounded conversation generation, and refCOCOg region caption datasets. We hope our research can inspire the research on MLLM design in a more elegant way for the community. 
related work: 
 Multimodal Large Language Models. Early multimodal models [53] explore better fusion strategies, various feature extractors, and different meta-architectures. Most works focus on single tasks, such as caption and VQA. With the development of the large language models [6; 93; 40] , recent works [52; 3; 92; 71; 17] mainly explore building an instruction-tuning pipeline for multiple multimodal benchmarks [39; 74; 62; 32] . LLaVA [71; 70; 69; 106; 136; 30; 28] is one earlier work that treats visual features as tokens. After that, several works [126] explore visual cues to enhance the visual inputs of LLaVA. On the other hand, several works [129; 127; 88; 131; 24; 25; 66; 130; 83; 38; 49] add extra components to adapt LLaVA for visual grounding, detection, segmentation, and video analysis. In particular, several works explore language-driven grounding and segmentation. However, these works are all trained with a specific purpose. We aim to build the simplest model to unify segmentation, instruction tuning, and prompt-driven segmentation in one model. To the best of our knowledge, we are the first model to achieve this goal.Unified Segmentation Models. The vision transformers [10; 26; 79; 94] have led to research interest in universal segmentation. Recent works [95; 19; 123; 56; 21; 117; 115; 73; 91; 124; 122; 139; 111; 54; 141] have developed mask classification architectures with an end-to-end set prediction approach, outperforming previous specialized models [12; 46; 36; 57; 34; 60; 140] in both image, video and generalization segmentation tasks [45; 61; 59] . In particular, several works explore open-world segmentation, including entity segmentation [85; 84] , open-vocabulary segmentation [125; 105] . Meanwhile, several works [58; 41; 111; 112; 33; 2] adopt one model with shared parameters to perform various segmentation tasks. One recent work, OMG-Seg [58] , first unifies image, video, open-vocabulary, and interactive segmentation in one simple model. However, all of these works focus on visual segmentation and cannot generate interactive text and visual prompts, like MLLMs. Our work builds such a bridge to align MLLMs, visual segmentation, and prompt-driven segmentation models from joint co-training and model sharing, which serves as a new baseline for this field.Language-driven Location and Segmentation. Early works [120; 68; 44; 23; 104; 135] in this direction mainly define the various language-driven tasks, including referring segmentation and [126; 133; 11; 82; 13; 119] introduce region-level visual embeddings, allowing LLMs to achieve object-level understanding and reasoning tasks. However, these models rely on complex region embedding extraction designs. In addition, most cannot perform pixel-level understanding tasks. Thus, as shown in Fig. 2 (c), [49; 88; 35] introduce segmentation tokens, enabling LLMs to output segmentation masks and thus handle pixel-level understanding and reasoning tasks. Nonetheless, they require a large segmentation module, such as SAM [47] , making the system highly redundant. As shown in Fig. 2 (d), GLAMM [87] combines the above pipelines to handle object-level and pixel-level tasks. However, this significantly increases the system's complexity and redundancy. Additionally, GLAMM relies on explicit instructions from the user, losing the perception ability to handle basic pixel-level understanding tasks such as instance segmentation, semantic segmentation, panoptic segmentation, and interactive segmentation. In this paper, we focus on addressing all the challenges above in a more simple yet elegant way. Our OMG-LLaVA unifies image-level (such as image caption and image-based conversation), object-level (such as region caption and visual prompt-based conversation), and pixel-level (such as universal segmentation, referring segmentation, reasoning segmentation, and grounded conversation generation) visual understanding and reasoning tasks into token-to-token generation. The framework follows a simple and elegant system design, including only one visual perception module and one large language model.[71] 1 ✓ ✓ MiniGPT4 [142] 1 ✓ ✓ mPLUG-Owl [118] 1 ✓ ✓ LLaMA-Adapter [132] 1 ✓ ✓ Mini-Gemini [65] 2 ✓ ✓ InternVL 1.5 [18] 1 ✓ ✓ VisionLLM [97] 1 ✓ ✓ ✓ Shikra [13] 1 ✓ ✓ Point & Box ✓ ✓ Kosmos-2 [82] 1 ✓ ✓ Box ✓ ✓ GPT4RoI [133] 1 ✓ ✓ Box ✓ ✓ Ferret [119] 1 ✓ ✓ Point & Box & Mask ✓ ✓ Osprey [126] 1 ✓ ✓ Mask ✓ ✓ SPHINX-V [67] 1 ✓ ✓ Point & Box & Mask ✓ ✓ LISA [49] 2 ✓ ✓ ✓ ✓ GLAMM [87] 2 ✓ ✓ Box ✓ ✓ ✓ ✓ Groundhog [134] 4 ✓ ✓ Point & Box & Mask ✓ ✓ ✓ ✓ AnyRef [35] 2 ✓ ✓ Box ✓ ✓ ✓ PixelLM [88] 1 ✓ ✓ ✓ GSVA [109] 2 ✓ ✓ ✓ Groma [78] 1 ✓ ✓ Box ✓ ✓ VIP-LLaVA [8] 1 ✓ ✓ Point & Box & Mask ✓ ✓ PSALM [135] 1 Point & Box & Mask ✓ ✓ LaSagnA [102] 2 ✓ OMG-Seg [58] 1 Point ✓ OMG-LLaVA 1 ✓ ✓ Point & Box & Mask ✓ ✓ ✓ ✓Unified View of Different Tasks. We model various tasks as the token-to-token generation to bridge the gap between image-level, object-level, and pixel-level understanding and reasoning. To support these tasks, we define three types of tokens: text tokens T t , pixel-centric visual tokens T pv , and object-centric visual tokens T ov . Text tokens encode textual information. Pixel-centric visual tokens represent dense image features, providing the LLM with comprehensive image information. Object-centric visual tokens encode the features of specified objects, offering the LLM object-centric information, and can be easily decoded into segmentation masks.Then, all the tasks can be unified as:EQUATIONFor example, in the classic image-level understanding task, i.e., image caption, a text response T out t is generated based on text instruction T in t and image features T in pv . In the object-level understanding task, region captioning, the text response T out t is generated based on text instruction T in t , image features T in pv , and specified object-centric visual tokens T in ov . The pixel-level reasoning task, referring segmentation, involves generating object-centric visual tokens T out ov based on text instruction T in t and image features T in pv . Additionally, OMG-LLaVA can support various mixed-level tasks, such as providing grounded descriptions around specified objects.Pixel-centric visual tokens can be obtained by tokenizing images using a CLIP backbone as the tokenizer. However, object-centric visual tokens require encoding object information to be easily decoded into segmentation masks. Therefore, methods like mask pooling in Osprey [126] and ROI pooling in GLaMM [87] fail to meet these requirements. We found that a universal perception decoder can meet all the requirements. Thus, we chose the OMG-Seg decoder [58] as the object-centric tokenizer due to its comprehensive capabilities. 
omg-llava framework: 
 The framework of OMG-LLaVA is shown in Fig. 2 (e). OMG-LLaVA comprises a large language model (LLM) and a frozen universal perception module. The universal perception module encodes images and visual prompts from users into pixel-centric and object-centric visual tokens. It obtains object-centric visual tokens output by the LLM into explicit segmentation mask responses. The LLM accepts text instruction tokens and pixel-centric and object-centric visual tokens from the universal perception module as inputs and then outputs text responses along with object-centric visual tokens. The detailed architecture of OMG-LLaVA is illustrated in Fig. 3 . The universal perception module comprises an image encoder, an OMG decoder [58] , and a non-trainable perception prior embedding component.Image Encoder. To maximize the perception capabilities of the universal perception module, we use the ConvNeXt-L [75]-based CLIP [86] model as the image encoder and employ a high image resolution (1024×1024). However, the large image resolution results in excessive visual tokens input into the LLM, leading to significantly higher computational costs than using lower-resolution images (such as 224×224 or 336×336). We address this issue by utilizing the lowest resolution image features (32× downsampling). Additionally, we use the pixel shuffle operator to further reduce the image features' resolution. Ultimately, the downsampling factor for the image features used to generate visual tokens is 64, meaning that a 1024×1024 image produces 256 visual tokens.OMG Decoder. We utilize the OMG decoder [58] to generate object-centric visual tokens, furnishing the LLM with information regarding the primary objects in the image and those mentioned by the user's input visual prompts. As shown on the left side of Fig. 4 , the OMG decoder comprises masked cross-attention [19] and self-attention layers. The OMG decoder's input includes a set of learnable object queries [20; 19; 10] for automatically capturing all objects of interest and visual prompt queries derived from encoded input visual prompts [47] . The visual prompt queries and learnable object queries are collectively termed object queries. The OMG decoder probes feature for object queries from the image features by employing masked cross-attention and models relationships between objects through self-attention. The object queries can be decoded into segmentation masks and object categories via a simple FFN layer. With the OMG decoder, OMG-LLaVA can efficiently tokenize object information into object-centric visual tokens, thereby equipping the LLM with information about objects in the image and those referenced by the user.The OMG decoder can accept point prompts as input. While box and mask prompts can be easily converted into point prompts, this crude conversion significantly loses prompt information, complicating the explicit encoding of the user's intent. To address this, we can impose constraints on the attention masks of the masked cross-attention layers based on the visual prompt to precisely encode the object information referenced by the prompt. As depicted on the right side of Fig. 4 , we utilize the box coordinates to define attention masks for all pixel features outside the box for box prompts. Similarly, we directly employ the provided object mask to generate attention masks for mask prompts. With this straightforward attention mask modification strategy, OMG-LLaVA can accurately capture the user's visual prompts, encompassing point, box, and mask prompts.Object Queries Mask Score Perception Prior Embedding. We find that directly combining a frozen perception module with LLM doesn't perform well, as also observed in LISA [49] . To retain the full capabilities of the universal perception module, OMG-LLaVA doesn't fine-tune the perception module to adapt to the output of the large language model. Instead, we propose a perception prior embedding strategy to tackle this challenge. Fig. 5 illustrates the perception prior embedding strategy.First, we fuse the image features F ∈ R HW ×C outputted by the image encoder with the object queries Q ∈ R Nq×C outputted by the OMG decoder D. Specifically, we utilize the segmentation mask M ∈ R Nq×HW obtained from the object queries and the corresponding confidence score S ∈ R 1×Nq to derive a mask score M S ∈ R HW ×Nq for each pixel for the object queries:EQUATIONThen, we compute a weighted average of the object queries Q based on the mask score M S and obtain the corresponding weighted object queries for each pixel. Pixel-centric visual tokens T pv are obtained by adding the weighted object queries to the image features F:EQUATIONAdditionally, we treat the foreground object queries as object-centric visual tokens T ov . The objectcentric visual tokens T ov are concatenated with the pixel-centric visual tokens T pv to form the visual tokens T v = (T pv , T ov ), which are input to the LLM to provide rich perception prior information.Visual Projector and Text Projector. Following [71] , we use an MLP as the visual projector, which is responsible for mapping visual tokens to the LLM's text embedding space. Since our visual tokens are pixel-centric and object-centric tokens, the visual projector comprises two MLPs, each handling one type of visual token separately. Inspired by [49; 87] , we also use a simple MLP to map the LLM output's hidden states of the [SEG] token to the visual space.Instruction Formulation. OMG-LLaVA can accept visual input, text input, and visual prompt input and output text responses and segmentation token, segmentation masks and labels. Thus, it can handle tasks such as image captioning, image-based conversation, region captioning, visual prompt-based conversation, referring segmentation, reasoning segmentation, grounded conversation, etc. We use a unified instruction formulation to support these functionalities. As shown in Fig. 3 , there are three special tokens: <Image>, <Region>, and [SEG] . Before being fed into the LLM, the <Image> token is replaced by visual tokens T v , and the <Region> token can be replaced by any object-centric visual token encoded by the visual prompt. The [SEG] token in the LLM's output is sent to the frozen OMG decoder to be decoded into a segmentation mask.Training. Following LLaVA [71] , our OMG-LLaVA performs two-stage training: pretraining and instruction tuning. During the pretraining stage, the perception model and LLM are frozen, and only the visual and text projectors can be tuned. In addition to the text regression loss, we apply regularization penalties to the visual projector P v and text projector P t to preserve object-centric information as much as possible.EQUATIONDuring instruction tuning, in addition to finetuning the visual projector and text projector, we use LoRA [37] to finetune the LLM. Following [87; 58] , besides the text regression loss, we apply cross-entropy loss and dice loss [80] to supervise the segmentation mask decoded by the [SEG] token, as shown in following (We set α = 5 β = 2 by default):EQUATIONTesting. The image-level, object-level, and pixel-level understanding and reasoning tasks can all be encompassed within the Eq. 3.1 paradigm. During the inference stage, we encode the necessary task requirements, such as text prompts, visual prompts, and image features, into tokens to input into the LLM. The output tokens of LLM are then decoded into text responses and segmentation mask responses according to the task definition. We refer the readers to check the more details in the appendix. dataset. † indicates that the method used the GranD dataset [87] for pretraining. [126] and the object-level point-prompt data from the MDVP dataset [67] , which contain approximately 74K and 200K data, respectively. For pixel-level understanding and reasoning, we use the referring segmentation datasets, including refCOCO, refCOCO+ [42] , refCOCOg [121] , and refClef, totaling 74K data. Additionally, semantic segmentation datasets, including ADE20k [137] and COCO-stuff [7] , totaling 26K data, and the grounded conversation generation dataset GranDf [87] , containing 200K data, are used.Implementation Details. We use the pre-trained ConvNext-L [75] OMG-Seg [58] as the universal perception module and InterLM2-7B [9] as the LLM for OMG-LLaVA. We adopt xtuner codebase [22] to build our model and data pipeline. The image is resized to 1024×1024. During the pretraining stage, only the visual projector and text projector are trained, with an initial learning rate set to 1e-3. During the instruction tuning stage, the initial learning rate is set to 2e-4, with only the perception model kept frozen, and the LLM is fine-tuned using LoRA [37] . The maximum sequence length in the LLM is set to 2,048. All training is conducted on four NVIDIA A800 GPUs with 80GB of memory. The pretraining stage and instruction tuning stage took 7 hours and 48 hours, respectively.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All the code and model weights will be released upon paper acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
main results: 
 Comparison with MLLMs. OMG-LLaVA is comprehensively compared with current MLLMs with perception capabilities, and the results are shown in Tab. Ablation Study. We conduct ablation studies on referring expression segmentation and grounded conversation generation datasets, with all training and testing settings consistent with the main experiments. We use a simple combination of OMG-Seg [58] and LLaVA [71] as our baseline, similar to LISA [49] , where the [SEG] tokens output by the LLM were input into OMG-Seg to obtain segmentation masks, with OMG-Seg kept frozen.As shown in Tab. 5, the baseline performed poorly on the RES datasets. Similarly, it exhibited low segmentation quality on the GCG dataset. This is because the LLM did not acquire any segmentation priors and needed to generate segmentation queries based on image features and adapt them to the input of the frozen perception module, which is a challenging task.When using our proposed perception prior embedding strategy, OMG-LLaVA exhibits performance gains of 13.8 cIoU, 10.6 cIoU, and 11.7 cIoU on refCOCO, refCOCO+, and refCOCOg, respectively. Additionally, the perception prior embedding strategy also brings a performance improvement of 11.1 mIoU on the GCG dataset and a slight improvement in image description capability (0.4 METEOR). When foreground object queries were provided to the LLM, OMG-LLaVA further improved its performance by 1.9 cIoU on refCOCO and 1.5 mIoU on GCG.We conducted a visualization analysis of the proposed strategies. As shown in the left part of Fig. 6 , the simple baseline has poor capability in associating text and segmentation, which is the crucial reason for its poor performance on RES. When using our proposed perception prior embedding strategy, the object query and pixel features are explicitly integrated according to the perception prior, resulting in significantly enhanced text-segmentation association capability. By adopting the object query input strategy, the quality of some challenging segmentation cases, such as the lower right corner of the fence in Fig 6 , slightly improves.Qualitative Results. We provide visualization results of OMG-LLaVA on multiple image-level, object-level, and pixel-level tasks in Fig. 1 . Additional qualitative visualization results or comparable visual results for referring expression segmentation and grounded conversation generation are presented in the appendix. 
conclusion: 
 We present a new MLLM, OMG-LLaVA, which bridges image-level, object-level, and pixel-level understanding and reasoning in one model. Our method only contains one image encoder, one LLM, and one decoder. With proposed perception prior embedding and unified task instruction tuning, OMG-LLaVA can perform over 8 different multi-modal learning tasks, as well as preserving the visual perception ability of the OMG-Seg baseline. Our method can achieve comparable results compared with previous combined works with much fewer trainable parameters and computation costs. We hope our work can inspire the community to rethink the design of the MLLM meta-architecture to minimize the model components and maximize the MLLM's functionalities. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: All the experiments are conducted on 8 A800 80G GPUs.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: No potential societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: No such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. [49] 1/1 0.4 -0.0 0.0 PixelLM [88] 309/135 17.4 -0.0 0.0 LaSagnA [102] 0/0 0.0 -0.0 0.0 GLaMM [87] 14 Evaluation results on image-level benchmarks. We evaluate OMG-LLaVA on several image-level benchmarks, including MME [32] , MMBench [74] , SEED-Bench [50] , POPE [63] , and AI2D [43] benchmarks. The evaluation results are shown in Tab. format not only achieves better performance in the referring expression segmentation task compared to the fixed answer but also avoids the damage to the LLM's instruction-following ability.Segmentation embeddings. We conduct ablation experiments on the generation strategy of segmentation embedding, and the results are shown in Tab. 10. We explore whether the hidden states of the intermediate layers corresponding to the [SEG] token are helpful for segmentation. Compared to using the hidden states of the last layer of the [SEG] token as the segmentation embedding, using the mean of the hidden states from all layers as the segmentation embedding resulted in negligible improvement on refCOCO but led to a significant performance drop on the more challenging refCOCOg.Concatenating the hidden states from all layers of the [SEG] token as the segmentation embedding resulted in a significant performance drop across all RES tasks. Therefore, the hidden state of the last layer already contains sufficient features to generate the segmentation mask, and introducing hidden states from other intermediate layers does not yield better segmentation results. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We outlined the contributions in the abstract, specifically proposing an elegant MLLM architecture that achieves image-level, object-level, and pixel-level understanding and reasoning capabilities using only a single visual encoder. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discussed the limitations of the paper in the appendix, including the inability to segment part-level objects due to the constraints of OMG-Seg. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: Yes.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Qualitative comparison with SOTA methods. We conduct qualitative comparisons and analyses on various tasks, including referring expression segmentation, grounded conversation generation, and image-based conversation, against the SOTA methods LISA [49] and GLaMM [87] . Fig. 7 shows the visualization results of the RES task for LISA, GLaMM, and our proposed OMG-LLaVA. OMG-LLaVA demonstrates a more stable segmentation performance than LISA and GLaMM. Additionally, OMG-LLaVA exhibits better image and text understanding capabilities than LISA (13B) and GLaMM, as illustrated in the fourth column with the example of "the smallest chair".Fig. 8 shows the visualization results of the GCG task for GLaMM [87] and OMG-LLaVA. Our proposed OMG-LLaVA provides more detailed and accurate descriptions of the scene, such as "lighthouse" and "bear." Additionally, OMG-LLaVA demonstrates more stable segmentation capabilities, as seen in the "mountain" in the bottom-right corner image.Fig. 9 shows the visualization results of the visual prompt-based description task for GLaMM [87] and OMG-LLaVA. Compared to GLaMM, OMG-LLaVA supports more flexible visual prompts, including point, box, and mask prompts. Additionally, OMG-LLaVA can generate more detailed object captions and demonstrate a more accurate image understanding.Fig. 10 shows the visualization results of the image-based conversation task for LISA [49] , GLaMM [87] , and OMG-LLaVA. Compared to LISA and GLaMM, OMG-LLaVA has stronger instruction-following ability. For example, when answering the question, "What is the number on the jersey of the athlete squatting on the ground?" both LISA and GLaMM incorrectly segmented "the jersey of the athlete squatting on the ground." Compared to GLaMM, OMG-LLaVA can provide more detailed and accurate answers to user questions. Compared to LISA, OMG-LLaVA demonstrates stronger scene understanding and reasoning abilities. For instance, in question 3 of Fig. 10 , LISA gave an utterly incorrect answer despite using a larger LLM (13B).Visualization results of RES. We provide additional visualization results of OMG-LLaVA on the RES task in Fig. 11 . OMG-LLaVA demonstrates a strong understanding of spatial relationships and human actions, enabling it to accurately and reliably segment the specified objects based on these descriptions. Furthermore, even without training on any reasoning segmentation data, OMG-LLaVA exhibits the ability to perform reasoning segmentation. As shown in Fig. 12 , OMG-LLaVA can infer the target based on the question and accurately segment the corresponding object.Visualization results of GCG. As depicted in Fig. 13 , our method performs well on the grounded conversation generation task. OMG-LLaVA demonstrates strong scene understanding and object segmentation capabilities. Although some objects are overlooked, this is due to the omission of many objects in the image captions of the Grandf dataset. We believe that using higher-quality data for training would result in even better performance for OMG-LLaVA.Visualization results of visual prompts-based description. Fig. 14 shows more visualization results for the visual prompt-based description task. OMG-LLaVA supports input of point, box, and maskbased visual prompts and provides detailed descriptions. These descriptions include information about the objects and their relationships with other objects in the scene.Limitations of OMG-LLaVA. Although OMG-LLaVA achieves image-level, object-level, and pixellevel capabilities with a concise and elegant architecture, much room still exists for improvement. Firstly, joint training with pixel-level understanding data often leads to decreased image-level capability, a phenomenon widely observed in LISA [49] and GLaMM [87] . This challenge could be addressed by organizing the data to eliminate this conflict. Secondly, due to the lack of multi-granularity segmentation capability in OMG-Seg, OMG-LLaVA cannot perform part-level segmentation. This challenge could be addressed using a more powerful and universal perception module by adding part-level visual inputs.Future Works. Several future directions can be explored with our new meta-architecture. We list two potential directions, including video and more instruction-tuning data. Although OMG-Seg [58] can acquire the video inputs, OMG-LLaVA still cannot perform pixel-level spatial-temporal reasoning. This is due to the lack of such datasets. Moreover, more instruction-tuning data involve more localization outputs, and multiple round conversations can be used to build a stronger MLLM model. For example, we plan to use full GLaMM datasets [87] and more detection datasets [48; 89] for joint co-training as future work if more computation resources are available.A black truck is driving on a dirt road. The truck is also parked on the same dirt road. The sky is visible over the truck.A black Jeep is parked on a dirt road, surrounded by a vast sandy landscape. The scene is set in a desert environment with a clear blue sky visible.The image features a building standing on grass . The sky is overarching the building.The image features a lighthouse with a sky overhead. The sky is also seen over grass.The image showcases a living room with a chair and a potted plant placed on a rug . A window is situated beside the chair and a wall . The rug is positioned on the wooden floor.The image showcases a living room with a large window and a wooden floor with rug. A potted plant is placed on the wooden floor. A chair is positioned beside the potted plant. The chair is also situated on the wooden floor. The wooden floor is connected to the wall and the window and the door is embedded in the wall.The image showcases a book and a cabinet placed on a rug . The cabinet is situated beside a wall . A light is also visible, positioned beside the book.The image showcases a vintage, wooden cabinet with a built-in bookshelf. The cabine is positioned on a rug, and a light fixture is hanging from the ceiling . The cabinet is also situated beside a table, which is also placed on the rug. Book is on the table. The rug is on the wooden floor.The image features a dog standing on grass . The dog is positioned in front of a mountain . The grass is attached to the mountain.The image shows a bear standing on grass. The bear is positioned in front of a mountain. The sky is visible over the mountain.A woman stands in front of a waterfall in a mountainous region, surrounded by tall snowy cliffs and greenery, creating a beautiful and serene scene.A woman in a black shirt and black pants is walking in the mountains . A coffee cup is prominently positioned in the image. It's a large cup with a brown handle, and it's placed on a saucer. The cup is filled with a warm beverage, likely coffee, and it's the main focus of the scene.OursA small, round pastry is located in the middle of the plate. It's the second pastry from the left and is the second row from the bottom.a cup of coffee a macaroon on the left of the plate A small monkey is seen in the image, hanging from a branch. The monkey appears to be a baby, possibly a baby monkey, and is positioned towards the left side of the image. monkey hanging from tree branch A large, leafy tree is situated in the foreground of the image. It appears to be a mature tree, possibly a pine tree, and is located near a house. The tree is the main focus of this region, and it stands out against the backdrop of the house.A silver car is parked on the side of the road, occupying a significant portion of the image. The car's color is white, and it appears to be a sedan. The car is positioned in front of a blue house, which is also visible in the scene.a tree in front of a house a grey car parked on the side of the road 2023. 4 


Decision: This paper proposes OMG-LLaVA a framework for combining pixel-level vision understanding with reasoning abilities.  The paper received 1 strong accept, 1 accept, 1 borderline reject, and 1 borderline accept recommendations from reviewers. Positive points included the good motivation, clean and effective approach, generally clear writing, and good results. Negative points included lack of details for some approach components, limited technical novelty (especially by one reviewer), and missing ablation studies. Many of these concerns were adequately addressed by the rebuttal. There was an extensive post rebuttal discussion between the reviewers about the pros and cons of the paper. Overall, after carefully considering the paper, rebuttal, and discussions, the ACs feel that the paper makes a solid contribution, in which the positives outweigh the negatives, and recommend accept. It is recommended that the authors incorporate the rebuttal points into the final version.

Reviews: review: paper_summary: paper_summary: This work proposes, OMG-LLaVA, a unified framework for image-level, object-level, and pixel-level vision-language comprehension. In particular, OMG-Seg, a universal image segmentation model, is integrated with a LLaVA-like multimodal large language model (MLLM), so that various image-level (e.g., image captioning, visual question answering), object-level (e.g., promptable segmentation, region captioning), and pixel-level (e.g., referring expression segmentation, grounded conversation generation) vision-language tasks can be performed. By using one unified visual encoding and decoding module for objects and pixel masks, OMG-LLaVA poses a simple design and shows competitive performance compared with prior models.
strength_weakness: strength_weakness: 1. [Performance] Although this work claims OMG-LLaVA as a generalist model, it cannot outperform prior state-of-the-art models on any specific task. This performance difference would weaken the applicability. In Table 3, the performance should also be compared with GLaMM (which has significantly better performance than LISA).

2. [Technical contribution] Overall, OMG-LLaVA seems a direct integration of OMG-Seg and LLaVA. The most novel design in this work seems to be the perception prior embedding, which enables the connection between OMG-Seg and the object visual tokens in LLaVA. The technical contribution is thus somewhat limited. It would be great if the authors could better discuss the novel technical contributions of this work.

3. [Image-level tasks] Although this work claims OMG-LLaVA can unify image-level, object-level, and pixel-level reasoning and understanding tasks, there is no quantitative evaluation in terms of image-level tasks such as VQA.

4. [Frozen perception module] In OMG-LLaVA, the “universal perception module” (OMG-Seg) is frozen (Figure 2e) to preserve its learned knowledge. However, Table 3 shows that allowing the OMG decoder to be tuned can actually improve the referring segmentation performance. It is not explained why this module should be frozen in other tasks, or whether this practice would bring further gains.
questions: questions: In addition to the weaknesses mentioned above, there are two minor questions:

1. What is the “pixel shuffle operator” (Line 171)?

2. To be consistent with Equation 2, should the segmentation mask be written as $\mathcal{M}\in\mathbb{R}^{HW\times N_q}$ instead of $\mathcal{M}\in\mathbb{R}^{N_q\times HW}$?
limitations: limitations: The authors have discussed the technical limitations, but have not discussed potential societal impacts. There is no explanation about why this work has no societal impacts, either.
paper_summary: paper_summary: This paper presents OMG-LLaVA, which facilitates pixel-level, object-level and image-level understanding tasks within a unified framework. Within the OMG-LLaVA, an OMG decoder and perception prior embedding approach are proposed to enhance object-centric comprehension. Comparison with state-of-the-art methods prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The proposed method is not very elegant as too many designs customized for different tasks are introduced in the framework. For different tasks (e.g., image captioning, image segmentation and visual prompt based comprehension), different workflows should be conducted.
2. Some important technical details are omitted. For example, to encourage different object queries to attend to different objects, hungarian matching algorithm should be applied to assign labels for each individual object query, especially for COCO segmentation where multiple objects are supposed to be simultaneously segmented. Besides, to generate masks from object queries, the mask decoder should be carefully designed to generate high-quality masks, however, this paper only mentioned a simple FFN is utilized. Is it reasonable? More details should be provided. 
3. Some important experimental results are not provided. For example, experimental results on prevalent VQA benchmarks like MMBench, MME, SEEDBench, etc. should be provided to illustrate the general-purpose conversational capability of OMG-LLava. Whether introducing these segmentation data can hurt the VQA capabilities of the model? Overall, only segmentation and simple captioning qualitative results are reported, which are much less than the supported functions of the proposed OMG-LLava.
questions: questions: Since comprehension capabilities ranging from image-level to object-level are established, does OMG-LLava possess composite capabilities such as visualizing the intention of LLM during a conventional conversation in the form of segmentation masks? This is very helpful of understanding the working mechanism of LLMs.
limitations: limitations: See weakness.
paper_summary: paper_summary: This paper proposes OMG-LLaVA, an elegant MLLM framework. OMG-LLaVA achieves pixel-level, object-level, and image-level understanding and reasoning with only a perception model and a language model. OMG-LLaVA exhibits more comprehensive capabilities than current MLLMs, such as LLaVA, Osprey, and GLaMM. The experimental results on several benchmarks demonstrate that OMG-LLaVA achieves performance comparable to SOTA methods.
strength_weakness: strength_weakness: 1 The paper needs to include more ablation experiments to help readers better understand the work. For example, what would happen if the visual projector design shared an MLP for pixel- and object-centric visual tokens?
2 Would the performance of OMG-LLaVA improve with a larger LLM?
3 The authors should include more MLLMs in Table 1 for a more comprehensive comparison.
questions: questions: 1 The paper needs to include more ablation experiments to help readers better understand the work. For example, what would happen if the visual projector design shared an MLP for pixel- and object-centric visual tokens?
2 Would the performance of OMG-LLaVA improve with a larger LLM?
limitations: limitations: The authors pointed out the limitations of OMG-LLaVA, including performance degradation on image-level tasks and lack of capability for part-level segmentation. The latter can be addressed by replacing the perception model with a more powerful one.
paper_summary: paper_summary: This work presents a new multi-modal large language model, named OMG-LLaVA. This model builds on previous work OMG-Seg and combines a LLM (LLaVA-like) into one simple framework. Compared with previous MLLMs, this work unifies lots of image-level, object-level, and pixel-level segmentation and reasoning tasks in one shot.
strength_weakness: strength_weakness: Overall, I think this paper is good and elegant, compared with previous combined approach. However, there are several details should be added for the better draft.
1, More detailed designs or ablation should be carried out for perception prior embedding design. The current draft only shows the effectiveness of proposed approach.

2, More detailed discussion on meta-architecture design should be 
One question is why you fix the image decoder why not use another decoder with the same architecture but with the same pre-trained weights.
Although this operation can increase the parameter costs, I wonder whether joint sharing one decoder can have mutual effects.
Thus, more detailed experiments on meta-architecture design should be added.
3, No parameter and Gflops analysis. For example, the author claim they only use one image encoder, one decoder and one LLM. Compared with previous combined approach (GLaMM), this advantage is not well explored.
questions: questions: See the weakness, I would raise the score if all questions are well solved.
limitations: limitations: Please see weaknesses.

Comments: title: Reply to authors rebuttal
Please let us know whether we address all the issues
Response to Reviewer czBJ
Please let us know whether we address all the issues
Please let us know whether we address all the issues
Concerns Addressed
Thanks for the response.
Please let us know whether we address all the issues
Reply to authors rebuttal
comment: Thanks for authors further rebuttal.

While I acknowledge that learning masks as a sequence of coordinates is generally less effective than directly predicting them, it appears that the pixel-level and object-level understanding capabilities primarily stem from the robust vision foundation model, OMG-Seg, rather than from the MLLM itself. This integration adds an extra vision encoder and decoder from the foundation model, which significantly increases the computational load compared to a pure MLLM. Given the use of such a powerful visual foundation model, why not simply address downstream tasks like segmentation by utilizing a specialist model's API, as is done in HuggingGPT, if pursuing a higher performance?

Overall, I believe the contribution of the proposed model is diminished by the reliance on the powerful OMG-Seg model. I've raised my score to 4, pending further discussion with the PC and ACs.
Dear reviewer #ck7P,

We kindly ask again whether all your questions have been resolved, as the rebuttal deadline is approaching. If they have been resolved, please raise your score.

Thank you
Thanks for your reply. We will merge your comments and extended experiment results into our next draft.


Best regards!

Authors of OMG-LLaVA
Thanks to the authors for their detailed response. In their response, the performance of OMG-LLaVA is updated, including a more comprehensive evaluation on image-level benchmarks with or without segmentation data tuning. The technical contribution is sufficiently clarified. Therefore, I would like to raise the rating to 5.
Dear reviewer,

Please carefully read the rebuttal and the other reviews, then reply to the authors indicating whether your questions/concerns have been addressed. If not, please specify which questions/concerns remain unresolved so that the authors have a fair opportunity to address them before the author-reviewer discussion period ends in approximately 36 hours from now (August 13th, 11:59pm AoE).

Best,
AC
Dear reviewer,

Has our response addressed your questions? If you have any other questions or feel that some issues have not been adequately answered, please let us know. If not, we kindly ask you to consider raising your score.

Thank you
Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments and a PDF file. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you
Thanks for the further explanation. Please include the supplementary experiments in the final version of the submission. I'll keep my positive rating.
Thanks for your comments and acknowledgement to our work.

We will release the entire codebase, including training, testing, and demo model.

Best regards!

Authors of 8973
Dear reviewer,

Thank you for the comments on our paper.

We have submitted the response to your comments. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues.

Thank you
Thanks for your efforts and quick response.

> **R1:  Framework Design**

We apologize for the previous imprecise statement. We would like to rephrase it as "**Fine-grained** mask decoding and **flexible** visual prompt encoding are challenging to implement solely with LLMs without relying on additional modules." 

While we acknowledge that VisionLLM and VIP-LLaVA can respectively achieve mask decoding and visual prompt encoding without external modules, **these approaches have significant limitations**. For instance, VisionLLM decodes N contour points similar to PolarMask[1] to generate segmentation results, **but contour-based representations inherently suffer from inaccuracies** when compared to binary masks for precise object segmentation. This makes it hard to segment objects with hole. OMG-LLaVA adopt query-based baseline, OMG-Seg as baseline, which can avoid this issue. VIP-LLaVA encodes visual prompts by drawing them onto the image, **requiring all visual prompts to be provided before the dialogue**, limiting the flexibility to add new prompts during the conversation. OMG-LLaVA, on the other hand, leverages the OMG decoder to produce high-quality binary mask segmentation results and flexibly decode user-input visual prompts during the conversation.

[1] Xie E, Sun P, Song X, et al. Polarmask: Single shot instance segmentation with polar representation. CVPR 2020.

***

> **R2: Technical Details**

OMG-LLaVA employs a pre-trained OMG-Seg model as its visual encoder. Similar to DETR, OMG-Seg utilizes learnable queries and has been pre-trained on the COCO dataset. The training of OMG-Seg undoubtedly requires the Hungarian matching algorithm to assign labels akin to DETR. However, in the training of OMG-LLaVA, OMG-Seg is kept frozen, eliminating the need for training and Hungarian matching. Instead, it performs the inference process of panoptic segmentation. The dense instance perception capability is derived from the pre-trained OMG-Seg. 

The statement "the object queries can be decoded into segmentation masks and object categories via a simple FFN layer" is a detail describe of the OMG-Seg model and may lead to some misunderstandings. In OMG-LLaVA, the hidden states of the [SEG] token are transformed into initial object queries through an MLP projector. Subsequently, these initial queries are fed into the OMG decoder, where they interact with image features via self- and cross-attention layers to produce final object queries. Within the OMG decoder, the final object queries are passed through a simple FFN layer to obtain segmentation masks and object categories. We will rephase this part in the next draft.

***

> **R3: VQA Results**

When trained exclusively on the LLaVA dataset, the primary distinction between OMG-LLaVA and LLaVA 1.5 lies in the perception prior embedding. Despite employing the same LLM, an identical MLP projector, and similar CLIP backbones (ConvNext-L for OMG-LLaVA and ViT-L for LLaVA 1.5), the incorporation of perception results as input to the LLM enables a more profound understanding of images. Analogous findings have been reported in Vcoder [2] and SeTok [3].

[2] Jain J, Yang J, Shi H. Vcoder: Versatile vision encoders for multimodal large language models. CVPR 2024.

[3] Wu S, Fei H, Li X, et al. Towards Semantic Equivalence of Tokenization in Multimodal LLM. ArXiv, 2024.

***

> R4: Now that the OMG-LLaVA has object-level perception ability, can OMG-LLaVA tackle traditional task like object detection?

OMG-Seg is capable of handling both instance segmentation and panoptic segmentation, and bounding boxes can be easily extracted from the generated binary masks. As OMG-LLaVA incorporates a pre-trained OMG-Seg, it inherently possesses all of OMG-Seg's capabilities, such as object detection. OMG-LLaVA achieves 44.5 mAP on the COCO instance segmentation benchmark and 45.8 mAP on the COCO object detection benchmark.

In OMG-LLaVA, current dense segmentation tasks such as panoptic segmentation rely solely on OMG-Seg and do not utilize the LLM. We think that training the LLM to reproduce the perception results already output by OMG-Seg is unnecessary and meaningless, even though it could be achieved through simple training.
**R1: Framework Design**

I disagree with the author's claim that *"mask decoding and visual prompt encoding are hard to implement solely with LLMs without relying on other modules"*, because there are already related works that handle mask decoding and visual prompts using the LLM alone. Specifically, VisionLLM [1] transfers polygon masks into a sequence of points sampled from their contour as LLM's learning target. ViP-LLaVA [2] renders the visual prompts in the image, and feeds the image into the LLM without any visual prompt encoding techniques for training.

[1] Wang W, Chen Z, Chen X, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2] Cai M, Liu H, Mustikovela S K, et al. ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 12914-12923.

**R2: Technical Details**

My major problem is how the set of learnable object queries in the OMG decoder can automatically capture distinct object regions? As a comparison, each object query in the DETR should be assigned a unique GT box as the supervision. This is crucial in establishing the dense instances perception capability.

Since OMG-Seg, a powerful foundation model that can solve universal segmentation tasks, is adopted as the mask decoder, the expression that *"the object queries can be decoded into segmentation masks and object categories via a simple FFN layer"* is somewhat misleading.

**R3: VQA Results**

Thanks for authors' efforts, I have another follow-up question about the provided VQA results. 

1. What makes the OMG-LLaVA perform better than LLaVA-v1.5 using the same training data? 

**Follow-up questions:**

Now that the OMG-LLaVA has object-level perception ability, can OMG-LLaVA tackle traditional task like object detection?
After going through the feedback from other reviewers and the author's detailed replies, I'm more convinced this paper lives up to the NeurIPS standards. I’m especially impressed by the authors’ thorough and detailed response, with the solid experimental results they added. They addressed the key issues I brought up. So, I've bumped my score up to 8, with a clear accept.

Just to sum up my thoughts on this research: The paper introduces OMG-LLaVA, which is a trailblazer in merging three levels of perception into a single LLM system, using just one encoder, one decoder, and one LLM. The way the authors link object queries from a universal segmentation model to the LLM is really smart. It's not just about beating other models like Pixel-LLM and GLaMM—it’s about doing it more elegantly. During the rebuttal, the authors also showed that OMG-LLaVA performs even better with a stronger LLM, pointing to great potential for future work.

At last, the authors should open-source the entire codebase and models (including the stronger LLMs in the rebuttal) for the community soon.