Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
Authors: Yang Jiao
Shaoxiang Chen
ZEQUN JIE
Jingjing Chen
Lin Ma
Yu-Gang Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Multimodal Models
Vision-Centric Capabilities
Primary area: machine_vision
TL;DR: a large multimodal model with versatile vision-centric capabilities

Abstract: Large Multimodal Model (LMM) is a hot research topic in the computer vision area and has also demonstrated remarkable potential across multiple disciplinary fields. A recent trend is to further extend and enhance the perception capabilities of LMMs. The current methods follow the paradigm of adapting the visual task outputs to the format of the language model, which is the main component of a LMM. This adaptation leads to convenient development of such LMMs with minimal modifications, however, it overlooks the intrinsic characteristics of diverse visual tasks and hinders the learning of perception capabilities. To address this issue, we propose a novel LMM architecture named Lumen, a Large multimodal model with versatile vision-centric capability enhancement. We decouple the LMM's learning of perception capabilities into task-agnostic and task-specific stages. Lumen first promotes fine-grained vision-language concept alignment, which is the fundamental capability for various visual tasks. Thus the output of the task-agnostic stage is a shared representation for all the tasks we address in this paper. Then the task-specific decoding is carried out by flexibly routing the shared representation to lightweight task decoders with negligible training efforts. Comprehensive experimental results on a series of vision-centric and VQA benchmarks indicate that our Lumen model not only achieves or surpasses the performance of existing LMM-based approaches in a range of vision-centric tasks while maintaining general visual understanding and instruction following capabilities.

Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
Yang " Chen Zequn Jie Jingjing C.ent1.2 Lin Ma², Yu-Gang Jiang'.2
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University Shhnngaai Collaborative Innovation Center on Intelligent Visual Computing Meituan
Abstract
Large Multimodal Model (LMM) a hot research topic in the computer vision area and has also demonstrated remarkable potential across multiple disciplinary fields. A recent trend is to further extend and enhance the perception capabilities of LMMs. The current methods follow the paradigm of adapting the visual task outputs to language-orienter formats. This adaptation leads to the convenient development of such LMMs with minimal modifications, however, it overlooks the inductive biases within diverse visual tasks and hinders the learning of perception capabilities. To address this issue, we propose novel LMM architecture named Lumen, which decouples the learning of perception capabilities into lask-agnostic and task-specific stages. Firstly, Lumen promotes fine-grained vision-language concept alignment, which is the fundamental capability for various visual tasks. Thus the output of the task-agnostic stage a shared representation for all vision- centric tasks we address in this paper. Afterward, the task specifie decoding is carried out by flexibly routing the shared representation to lightweight task decoders with negligible training efforts. Comprehensive experimental results on series of vision-centric and VQA benchmarks indicate that our Lumen model not only achieves surpasses the performance of existing LMM- approaches in a range of vision-centric tasks while maintaining general visual understanding and instruction following capabilities.
Introduction
As the Large Language Models (LLMs) currently ignite the spark of Artificial General Intelligence (AGI), Large Multimodal Models (LMMs) [3, 4. integrating visual modalities with the linguistic prowess of L.MMs. With the instruction-following
6, 7 g 9 take step forward by
and content-reasoning capabilities inherited from LLMs, the LMM has successfully functioned as a versatile assistant across a wide range tasks, including visual question answering [00 [2].
image captioning 13 text-to-image generation [15], etc.
In pursuit of more convenient and efficient human-Al interaction, it is crucial further explore fundamental Aision-centrio capabilities encapsulated in the LMMs, which aid in detailed object referencing and dialogue responses. Early works, e.g., MiniGPT-v2 [16), Kosmos-2 [77 and Qwen- VI. [8], equip the LMM with the grounding ability by reformulating bounding boxes a sequence of coordinate tokens and adapting them to the language model's output space. Griffon [9 extends this design to object detection by meticulously curating language- to such language model-oriented reformulation, these methods can be implemented with minimal modifications to existing LMMs.
detection dataset. Owing
"This work is done when Yang Jiao is an intern at Meituan. t Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"Person- [232,49,134,33
person-[391 [391,42,153,,00
person-[833) (oxoon[85,11,109.60
Previous Methods
"The location is [LOC]. The task output [ET]][SEG]/[POINN
Lumen (Ours)
Figure 1: Comparison of our proposed Lumen with previous methods. Previous methods (e.g.. Griffon 19 serialize bounding box coordinates into discrete token sequences conform the language-oriented outputs format of LMMs, disregarding the unordered nature inherent in bounding boxes. Our Lumen first predicts unified heatmaps for various tasks. These heatmaps are further used for guiding simple decoding tools with the parsed task-type indicators to support versatile visual tasks. We omit the user instruction of referring all persons in the image for conciseness.
Although convenient, the aforementioned approaches encounter challenges when scaling up more intricate scenarios and vision-centré tasks. First, those LLMs highly relies on language models's auto-regressive sequence generation method, which leads to high uncertainty when multiple objects are concurrently addressed. As shown in the first row of Fig [ the three persons highlighted with bounding boxes of different colors lack an inherent order. And imposing a sequential order on them would exacerbate the confusion in the decoding process, as the model would be compelled to produce drastically different outputs after the same word "person". Although sequence augmentation augnede
techniques introduced in Pix2Seq (20] can alleviate this issue, they are tailored for the object detection task and can not be seamlessly transferred to other vision-centric tasks such as instance segmentation and pose estimation. On the other hand, in contrast to tasks within the NLP field, which often exhibit stronger inter-task correlations (2T) vision-centric tasks are inherently discrete due to the inductive [22]. Therefore, reformulating heterogeneous vision- centric task outputs into language-orienter formats tends to overemphasize the format consistency with language models, while the learning of the underlying visual perception capabilities and intrinsic characteristics of diverse vision tasks is overlooked.
bias introduced by their distinct task definitions
When delving into fundamental vision-centric tasks, namely object detection, instance segmentation, and pose estimation we observe that they can be decoupled into task-agnostic and lask-specific learning processes. Essentially, these tasks share common task-agnostic objective of identifying individual instances with an instruction like "finding the area addressed instructions". while their task-specific definitions introduce different decoding rules for generating diverse formats of outputs (c.g.. boxes, masks, or points). Compared with the general aask-agnostic objective, task- specific outputs relies less on semantics but is more difficult for the LMMs to learn output. Based on the above analysis, in this paper, we propose Lumen, a Large multimodal model with vision- centric capabilities enhancement, which decouples the task-agnostic and task-specific learning in two consecutive stages as shown in the Fig [ Concretely, in the first stage, the aforementioned visual tasks unified by reformulating them into the same matching problem. We start by feeding the user's instruction and image into a LMM for content comprehension. The obtained responses contain designed special token (i.e, [LOC] in Fig T that encapsulates the visual concepts conveyed in the provided instruction, regardless of the specific task. Subsequently. this special token interacts with image patches via transformer-base based aligner generate heatmap, wherein the response each location indicates the matching probability between the instruction and the corresponding image region. In the second stage, utilizing this heatmap as indicative guidance, task task-specific decoding
'e only investigate these three visual tasks in this paper, as LMMs inherently lack image generation capabilities.

---

processes are further managed by flexibly assembling predefined decoding rules and lightweight decoders generate the final outputs with different formats. Due to such decoupled learning behaviors, on the one hand, Lumen can concentrate on promoting fine-grained multimodal content comprehension, rather than being trapped in learning diverse specialized decoding rules lacking in semantics. On the other hand, Lumen can be affected less by the various inductive biases associated with vision-centric tasks within the LLM token space, thereby seamlessly maintaining general visual understanding and instruction following capabilities, demonstrated in Table
In summary, our contributions are three folds: (1) We introduce Lumen, a Large multimodal model with rision-centric capabilities enhancement, which unleashes the vision-centric potential of the LMM by decoupling the task-agnostic and task specifcc learning processes; (2) Our Lumen can seamlessly adapt to tasks such as object detection, instance segmentation, and pose estimation without requiring meticulously curated specialized dialogue datasets as done in the previous work [9] (3) Our Lumen not only matches or exceeds the performances of existing LMM-based basde approaches on a series of viston-centric tasks, but also maintains general visual understanding and instruction following capabilities.
2 Related Work
2.1 Large Multimodal Models (LMMs)
Benefiting from the remarkable reasoning capabilities of Large Language Models (LLMs), LMMs [23 24 25 transfer these abilities to the vision domain by aligning visual tokens with LLMs' input space. To achieve this, pioneering work, Flamingo (26) resamples the visual features and feeds them into attention-base based adapter layers inserted in the LLM. Aiming at more comprehensive vision- language aligmment, BLIP-2 (27) designs a Q-Former and jointly performs cross- modal representation learning and generative learning. Inspired by the instruction tuning technique [2128 in NLP field, Instruct-Ble II. 22)] LLLaV 4 and Mini-GPT4 3 curate high-quality multi-modal instruction data for enhanced instruction followwn abilities. Meanwhile, systematically evaluate their general-
35 have been explored. However, these methods focus on high-level visual content comprehension and reasoning, ignoring the fundamental visual perception functions, such as object detection, instance segmentation, pose estimation, etc.
purpose conversational capabilities, more advanced benchmarks [30 31 32
33
34
2.2 Vision Generalist Models
Generalist models in the vision domain aim at unifying a wide range of vision tasks using one model. Motivated by the success of
models NLP field [13, OFA (36] and GIT [37 unify various tasks in the sequence generation format. Following this trend, Unified- 10 38 Pix2Seq v2 [39] and UniTab [40] add discrete coordinate tokens into the vocabulary, so as to accommodating more tasks. Moreover, Gato 41 successfully unifies reinforcement learning tasks as the sequence generation format. Nevertheless, the sequence generation modeling can lead to low inference speeds and degraded performances. Toward this end, Uni- Perceiiees 42 43)
unify different
tasks as the maximum likelihood matching problem by calculating representation similarities of all targets and each input. With such an objective, generative and non-generative tasks can be unified by selecting corresponding input and target candidates. However, these generalist models are restricted to pre-defined tasks, failing to be flexibly instructed by natural languages like LMMs.
2.3 LMMs with Vision-Centric Capabilities
To endow LMMs with vision-centric capabilities, two research directions are investigated. On the one hand, a line of work regards LLMs/LMMs as intelligent planners, and allows them to trigger a wide range of task-specific Application Program Interfaces (APis) according to user's instructions. HuggingGPT [4] connect GPT with a suite of visual experts. AutoGPT [45] can further execute post- post-processing programs after detection. Moreover, BuboGPT [46] further combines visual grounding specialists with LMMs. On the other hand, an alternative approach explores the intrinsic localization potential of L.MSs with task-specific modifications. Kosmos-2 [17]. MiniGPT-v2 [16]. Qwen-VL. VI. [88 and Shikra 4 enlarge the vocabulary size of LMMs with discrete coordinate tokens to deal with the visual grounding task [48). LISA 49 merges the LMM with SAM [50) for enhanced
3

---

Truinable
V-L Dense Aligner
Heatmap A
fimg
Freeze
Mose
Routing
Untrunable
LOL
Peak Point
Large Language Model
LoRA
Selection
Box Decoder
Promptable
Adapter
Tokenizer
(light-
Mask Decoder
Vision Encoder
448x4484 Ximg Image
336x336
Object Det. Grounding
Instance Seg. Referring Seg.
Pose Est.
Text Instruction
Stage Task-Agnostic Matching
Stage 2: Task-Specific Decoding
Figure 2: Overall framework the proposed Lumen. Our Lumen consists of two stages. In the first stage, the imput image and the instruction with designed special tokens are embedded and fed into a large language model interact and comprehend visual and textual contents. Then, the [LOC] token output and high-resolttion image features are further aligned to produce heatmap denoting cross-modal matching probabilities. In the second stage, the heatmap can serve as strong indication for various vision tasks, and the task outputs can be obtained with lightweight task- decoders. The routing of the decoding pathway is determined by the task token (e.g., [DET] in image) output generated in the first stage.
reasoning capability the referring image segmentation scenario [11. However, these methods do
not address fundamental vision tasks like object detection and instance segmentation, where multiple objects should be detected or segmented simultaneously. To address this issue, VisionI_MM [52]
regards LLM as DETR-like task decoder and customizes structural prompts for the detection
task with Hungarian matching [53] for label assignment. Griffon [19] takes different approach by leveraging the inherent detection capabilities of the LMM, introducing
detection dataset for the instruction tuning. However, these methods leverage discrete coordinate tokens outputs for different vision tasks, while ignoring their inherent disparities. In this paper, disentangle the task-agnostic and task-specific learning of various vision-centric tasks to unlock the visual potential of LMMs.
3 Method
As shown in Fig 2 our proposed Lumen comprises two consecutive stages. In the first stage, we concentrate on promoting fine-grained multimodal content comprehension via densely aligning visual regions and language instructions, disregarding the discrepancies between various vision tasks. In the second stage, with the resulting alignment heatmap from the first stage, task-specific decoding performed with specialized operations or decoders. In the following parts, we will elaborate on the detailed designs within each stage.
3.1 Stage Task-Agnostic Matching
3.1.1 Conversation Reformulation
A preliminary step for adapting vision-centric tasks to LMMs is to reformulate the visual data into conversation formats. For different tasks, we employ a unified [IMG]. Please find the location of (description) Respond with {f (format). ASSISTANT: Sure, the location is [LOC]. The task outpul [DET] [SEG] Here, {description) and {format) can be customized according to specific tasks. For vision-centric tasks like object detection, instance segmentation and pose estimation, (description) a certain class name, and {format) can be boxes, masks, points. For vision-language tasks like visual grounding and referring segmentation,
template: "USER:

---

USER [1MG] Please fiod the location of all persons, Respond with bones.
USER: [TMG]. What is the location of player in white Respoed with but.
Incation [LOC], baok ouque [DET]
ASSISTANT: II Sure, the location [LOC]. The mask ongor [GROUND]]
Object Detection Data
Visual Grounding Data
USER [TMG] Please the Incation alf dogs in the image. Respon
USER: [ING] Find the lncation of banzua on the place of tosst Respond wile musé
with mosis.
ASSTSTANT. Sure, location or [LOC] baok comped [SEG]
[LOC]. oupprr [REFSEG].
Instance Segmentation Data
Referring Segmentation Data
USER [IMG] find fncution right eliow. Respoud with point
ASSTSTANT. Strre, location [LDCC] rusk outpor [POTNT]
USER; [TMG]. are colory of the bas in image?
ASSISTANT: bus image
white and sed.
USER: What fenture
of ASSISTANT:
Pose Estimation Data
Visual Question Answering Data
Figure 3: The illustration of reformulated conversation data examples of different tasks. For each reformulated data example, we sequentially present the original image (left), the heatmap generated from annotations (middle), and the task-specific ground truth (right) in the figure.
{description) is the referring sentence and {format) can be boxes masks. Apart from [IMG] that denotes image features, we also introduce a task-agnostic special token [LOC] and set task- specific special tokens, including (DET). [SEG], [POINT]. [GROUND] and [REFSEG]. As shown in Fig22 regardless of task types, [LOC] tasked to summarize the visual concepts in instructions and further densely align with image regions. Task-specific tokens merely serve routers for guiding the decoding pathway in the second stage without encoding task-specific inductive bias. Examples of reformulated conversation data of various tasks are illustrated in Fig
3.1.2 Model Architecture
With the given image Timg and reformulated instruction Stat inputs, first feed them into conventional LMM (LLaVA-1.5 [54] in our implementation) to generate language responses. We also extract high resolution image features fimg E 1g32x322266 following prior works [18 19]:
fimg Fu(ging)
where Fy denotes the vision encoder. Since the adopted vision encoder, CLIP-VIT 55 is pretrained
to accept image inputs of 336 x 336 pixels, we resize its positional embeddings to adapt to higher resolution, e.g., 448 448 pixels. Afterward, the feature of [LOC] token hicc R 1x26 from the generated response, together with high-resolution image features, are fed into the proposed V-L dense aligner to calculate their matching probabilities, resulting in a heatmap H E 322x32.
A FAllings Moc)
(2)
where FA represents the V-L. dense aligner. our implementation, we employ lightweight transformer-based architecture FA to interact high resolution image feature tokens and the [LOC] token. More discussions on the architecture design are included See 4.3 and Appendix A.I
3.1.3 Model Training
To enrich semantics during the dense alignment, we merge data from various tasks by reformulating them into uniform conversation formats shown in Fig 3 For vision-centric tasks, we hide diverse task-specific output format details and their learning targets are reformulated and unified as heatmaps, where each clement denotes the matching probability between the instruction and the corresponding
5

---

region. To construct the ground-truit truth heatmap H, we use Gaussian kernel to fill the probabilities into a blank map following prior works |6|577
Hay exp(- (x 1,) 2 + (y 1
(3)
20
where the (Lr, is the location of the one referred by the instruction, and a the standard deviation. For object detection and visual grounding. (Ix; is the center coordinate of each object, and o object size-adaptive following [56]. For instance segmentation and referring segmentation, we first convert their masks into bounding boxes by calculating enclosing rectangles of masks, and then calculate (lz, and a with the same rules as detection and visual grounding tasks, For pose estimation, (la, by)
is the coordinate of the annotated keypoint, is a hyper-parameter designated as
With the ground-trune heatmap H and predicted heatmap H, we apply Gaussian focal loss follow- ing [56] as:
O= Σ N
(1 - Hay) log( (1 - Hzy) (H log(1 Hay) otherwise.
if Hy =,
zy
a and are hyper-parameters of the focal loss, and is the number of the location where the ground-truth matching probability equals to 1. We set = 2 and 3 = 4 in our implementation. For supervising the language outputs, we use the standard cross-entropy loss L following previous LMM-based methods [19 18. The overall loss function L can be formulated as:
C An Xeha,
(5)
where λι and A are sapee-parameters for balancing two losses. During training, since the vision encoder, adapter, tokenizer, and large language model parts in Figeri the weights of LLaVA, we only finetune the large language model using the LoRA [58] technique and train the V-I. dense aligner under the supervision of the loss function L.
3.2 Stage 2: Task-Specitic Decoding
Decoding Modules. We first introduce the operations of the three key modules illustrated in Fig 2 (1) Peak point selection parses the heatmap into set of points, where each point indicates the center location of an identified object or keypoint. Specifically, we filter the locations with heatmap response greater than their 8 surrounding neighbors candidate peak points, and then retain the top K candidates as the selected results. The value of K can vary across different tasks, which will be elaborated on in Sec (2) Box decoder is used for further regressing the extent of the objects designated by the selected peak points. For efficiency, instead of introducing an additional network as the box decoder, we reuse the V-L dense aligner by appending two new learnable special tokens after the [LOC] token as additional text-side inputs. Accordingly, the V-L dense aligner will generate two additional 1-channel map predictions, which are used for regressing the height and width under the supervision of the L1 loss functions similar to [56]. (3) Promptable mask decoder can accept both points and boxes as visual instructions to generate the mask for the referred object. We directly use the SAM model (50) for executing the above process without finetuning. In general, these decoding modules introduce negligible training costs and can be easily implemented.
Decoding Pathways. On the basis of these three decoding modules, we customize their differ- ent cooperation patterns to perform diverse tasks, which will be introduced as follows: (1) Pose estimation requires to return the keypoint coordinates. We can easily obtain these coordinates by parsing the predicted heatmap with the peak point selection operation. (2) Object detection and visual grounding share the same output format of bounding box, therefore, we employ the identical decoding pathway for them. Specifically, as shown in Fig [ we feed the heatmap into the cascaded peak point selection and box decoder modules to generate bounding boxes. (3) Instance segmenta- tion and referring segmentation share a relationship analogous to the one between object detection and visual grounding, therefore, we also adopt the same decoding pathway for them. Concretely, we first parse both the center point and bounding box of the object, and then, we use them as visual instructions to guide the promptable mask decoder to generate the mask prediction.

---

4 Experiments
4.1 Experimental Setup
Datasets. Our training data consists of datasets from the following different tasks. (1) For object
detection, we use MSCOCO [59], Objects365 [60] and OpenImages [6]]. (2) For visual grounding, we use RefCOCO, RefCCO+ and RefCOCOg [62] (3) For pose estimation, we use MSCOCO employ subset of ShareGPT4V is worth noting that for instance segmentation and referring segmentation. since we first transform their masks into bounding boxes as mentioned in Sec 3.1.2] the constructed heatmaps are the same as the object detection. Additionally, given that we do not finetune the mask decoder in the second decoding stage, we actually do not use segmentation annotations throughout the entire training process.
keypoints [59] and AIC [63]. (4) For visual question answering,
dataset [33 with 665K samples.
Model Configurations. For the first task-agnostic matching stage, we utilize pre-trained CLIP ViT-L/14-336 (55) and LLaVA-7B-vI.5 4 as our vision encoder and large multimodal model, respectively. For the second decoding stage, since the different decoding modules and pathways have been introduced in See S.c we primarily specify the K value choices and corresponding post- processing details of different tasks here. (1) For object detection and instance segmentation, which generally includes multiple objects in a given image, we set K 100 to generate 100 box candidates. Then we apply regular NMS [64] to filter redundant boxes, and the remaining boxes are used for further prompt mask decoder to generate instance segmentation results. (2) For visual grounding and referring segmentation, given that only one object is referred by referring sentence, we set K == to only generate one predicted box/mask, and no post- proccessnss is required. (3) For pose estimation, we follow previous works [65] 20 to first crop the single object from the image using bounding boxes. Then, for each keypoint category, we set K = 1 to extract the point with the highest matching probability the prediction result.
Training Details. For the aggosticc stage, our training comprises two phases. (1) In Phase 1, we mix the object detection, visual grounding and pose estimation data with sampling rates of 0.69, 0.23, 0.08, respectively, for balanced data sampling. (2) In Phase 2, we mix the visual question-ans answerrnng.... object detection, visual grounding and pose estimation data with sample rates of 0.67, 0.23, 0.07, 0.03, respectively. We set the batch size to 160 and train the first step for 50,000 steps and the second step for 10,000 steps 8 NVIDIA 80G A100 GPUs The loss function balance terms X and X are both set to 1. For each phase, we use Adam W as the optimizer with an initial learning rate of 3 x 10-4 and weight decay of 0. During training, we do not calculate heatmap loss Ln. for visual question-answering data. We do not use any data augmentation techniques for all tasks.
Evaluation Metrics. We adopt evaluation metrics commonly used within each field of task. For object detection and instance segmentation, we use mAP based on box IoU and mask IoU, respectively. For pose estimation, we use mAP based on OKS (object keypoint similarity). For visual question- answering, we comply with the evaluation protocol of each individual benchmark.
4.2 Results on Versatile Tasks
We evaluate our method on vision-centric and vision-language tasks as shown in Table [nd 2 We categorize existing approaches into three groups, namely "task-specific specialists" " "vision generalists" and "LMM generalists". according to their functions and architectures. (1) Task-specific specialists are customized models in different fields. They have diverse architectural designs and are limited to a single task. (2) Vision generalists pursue handling multiple tasks with unified architecture. To excel in fundamental visual perception tasks, they typically utilize a powerful vision encoder or additional designs (c.g, data augmentations or decoding strategies) adaptive to iesioiccentric tasks. (3) LMM generalists aim to resolve every task in a conversational format. Focusing on improving the conversational quality, they adopt vision encoders proficient in multimodal content comprehension. In line with these methods, our Lumen pursues unleashing the inherent vision-centric capabilities of LMMs.
Object Detection & Instance Segmentation. Object detection and instance segmentation require the model to make dense predictions across the image, therefore they pose great challenges in capturing fine-grained object cues. COCO val set is used for evaluation. (1) Compared with other LMM generalists, our Lumen surpasses them by clear margins, achieving 15.6 AP50 boost over
7

---

Table I: Results fundamental vision-centric tasks and vision-language tasks. We use "-" and gray cell to indicate the result is not reported and the corresponding task is not supported by the method, respectively.
Method
Object Det.
Instance Seg.
Pose Est.
Grounding
Refer Seg.
50
AP75
50 AP75
AP AP75
AP50
cloU
spee-fp Specialists
Faster R-CNN 66
40.3
61.0
44.0
DETR 67
43.3
63.1
45.9
Mask R-CNN
41.0
61.7
44.9
37.1
58.4
40.1
PolurMask
69
30.5
52.0
31.1
CPM 69
62.7
86.2
70.9
RTMPose 70
68.2
88.3
75.9
MDFTR
SeqTR
82.7
64.7
LAVT 73
61.2
ReL.
74
65.0
Vision Generalists
Pix2Scq-v2 20
46.5
38.2
64.8
Uni-Perceiver-y2
43
58.6
50.6
mPIJUG-2
46.9
40.6
84.7
VisionLLLM
44.6
64.0
48.1
25.1
50.0
22.4
LMM Generalists
Shikra-7B 47
MiniGPT-v2-78
84.4
Griffon- 13B 19
23.2
37.6
23.4
81.5
InstructCV 76
48.5
Lumen-7B (Ours)
35.3
53.2
35.8
30.4
49.8
31.0
67.2
90.4
75.6
83.6
65.1
Table 2: Results prevalent VQA benchmarks. Here, we employ English MMBench dev, SEEDBench image, MME test, MMMU val and Math mini sets for evaluation.
Method
Param
MMBench
SEED
MME
MMMU
Math
InstructBLIP 29
7B
36.0
58.8
1213/292
32,9
25.3
MiniGPT-4 3
7B
24.3
47.4
582/144
23.1
Shikra +
7B
58.8
Qwen VL-Chat
60.6
58.2
1488/361
I.L.A-vv.5 (54
64.3
66.1
1511/296
35.6
23.5
Lumen (Ours)
7B
64.9
65.8
1426/332
35.2
24.6
Griffon [19] and a 4.7 AP50 boost over InstructCV 76 This indicates the effectiveness of our method in discovering dense object cues. (2) Compared with vision generalists and task-specific specialists, our method further approaches their performances. We analyze that the performance gap might originate from two major aspects. First, we use the input size of 448 x 448, which is much smaller than these competitors, for example, 1024 x 1024 20 75]. Second, we do not introduce
DETR-like object decoding techniques done in 12 751.
Pose Estimation. We employ COCO human 2D body keypoint val set for evaluation. Following the top-down paradigm employed by previous works [65 20), we first crop single object from the image using the bounding box. Therefore, the pose estimation simplified to discovering keypoints of single object. Since the LMM generalists do not perform this task, we only compare our method with vision generalists and task task-specific specialists. As illustrated in Table [[] our method outperforms the vision generalist model Pix2Seq-v2 v2 [20] with 2.4 AP. Mcanwhile, our Lumen also achieves comparable performances with task-specific specialists.
Visual Grounding & Referring Segmentation. Compared to object detection and instance seg- mentation, visual grounding and referring segmentation underscore the language comprehension ability. Here, we report the results on RefCOCOg val set for comparison because the referring expressions are more diverse and abundant than those RefCOCO and RefCOCO+. We also provide complete results on these three benchmarks in the Appendix B As illustrated in Table our method achieves better performances than Shikra [47] and Griffon on visual grounding task, with AP 83.6 82.3 and 83.6 81.5, respectively.

---

Table 3: Ablation studies on model designs.
(a) Effect of different "V-I. dense (b) Effect of the pretrained mask decoder. cloUs reported.
(c) Effect of LMM baseline.
aligner" design choices.
Baseline
AP
APso
APro
Arch.
AP
AP50
APrs
Mask Dec.
RC
RC+
RCg
I.aVA-vI.0
24.0
39.0
23.0
Conv.
18.4
30.2
TeansDec.
65.8
58.6
62.0
LLaVA-v1.00
26.8
43.2
26.0
Trans.
28.4
45.0
28.1
SAM
66.6
59.2
61.5
LLaVA-v1.5
28.4
45.0
28.1
Table 4: Effect of multi-task training. 50 RefCOCOg val set and COCO val set are reported.
Object Det.
Grounding
Pose Est.
VQA
RefCOCOg
COCO
1
72.3
26.3
2
24.6
41.2
3
75.8
45.7
4
V
75.0
44.8
5
v
74.8
45.0
Table 5: Effect of the training recipe on VQA performances. MMBench used for evaluation
Phase
Phase
AR
CP
FP-C
FP-S
LR
RR
Overall
69.3
74.3
54.5
66.6
28.0
55.7
62.5
67.3
80.0
49.0
67.9
28.8
58.3
63.7
V
67.3
77.0
58.7
67.2
33.0
56.5
64.2
Visual Question Answering. To examine general visual understanding and instruction following of our model, we follow common practices and employ MMBench [77]. MME (78), SEEDBench [79), MMMU (30] and Math Vista [80] for evaluation. As shown in Table 2 our Lumen achieves VQA results comparable with state-of-the-art LMMs while extending versatile vision-centric capabilities.
4.3 Ablation Studies
For efficiency, we default to training the model for 10,000 iterations in the first phase of task-agnostic learning. This protocol is standard for our ablation studies
Multi-task Training. We use different task combinations for training the model, and the results shown in Table Based on the results, we have the following observations: (I) Compared with adopting datasets from all tasks (#5), discarding object detection data (#1) will reduce visual grounding performances. Similarly, removing visual grounding data (#2) also results in performance degradation on object detection. This demonstrates that object detection and visual grounding can benefit from each other. (2) Excluding pose estimation data leads to performance enhancements on both visual grounding and object detection (#3 vs #5). This might be caused by the data conflict problem addressed many generalist models 52 23 (3) Excluding VQA data (i.e., discarding
Phase 2) does not significantly affect model's 's performances both detection and visual grounding, which is reasonable as VQA data do not provide explicit object-level vision- -aaiunne alignment cues.
Training Recipe. We ablate the effects of 2-phase training strategy to inspect the contribution of vision-centric data on VQA tasks. In Table 5 we list detailed scores on diverse aspects covered by MMBench. #1 is initialized from LLaVA's weights, without any tuning. By comparing #2 and #3, we observe that FP-C (i.e, Cross-instance Perception) improves remarkably, which indicates that incorporating vision-centric data in Phase can promote multi-instance perception in VQA (as well as the overall performance). Scores of Phase I-only are not reported as the model trained without rich instruction data tends to respond template answers like "Sure, the location is [LOC] .?
V-L Dense Aligner Architectures. We ablate different architecture design choices as shown in Table Bal where "Conv." indicates the operation of concatenating feature of [LOC] token with every image patch features and fusing them with convolutional layers, and "Trans." represents liigt transformer in our main method. Substituting our transformer based aligner with simple
2 Pleasse refer to the official benchmark for the specific meaning of each term.

---

Table 6: Generalization ability evaluation object detection. f indicates that VOC not included in training data, and thereby used for cross- dataset generalization evaluation.
Figure 4: Qualitiave results of Lumen when generalizing to the object counting task.
Instruction:
"What location of strunsberriess
Method
COCO
VOCt
Pred: // GT: 17
RetinaNet 81
77.3
Faster R-CNN
66
80.4
Instruction: "Pleuse fond the location of all fruits"
Pix2Seq-v2 v2
20
57.4
38.5
InstructCV 76
48.5
61.7
Pred: 15 GT: 15
Lumen (Ours)
53.2
77.9
convolutional layers incurs significant performance degradation. This result indicates that complete vision-language interaction is crucial in promoting explicit dense rision-languae alignment.
Pretrained Mask Decoder. We adopt the mask annotations from RefCOCO (RC), RefCOCO+ (RC+) and RefCOCOg (RCg) to train lightweight transformer-lic mask decoder from scratch (TransDec.) and report cloU their val set ablate the effect of leveraging pretrained SAM decoder in Table 3b Although SAM outperforms TransDec., the narrow performance gap suggests that the primary challenge advancing segmentation performances within our framework the need to improve the dense vision-language alignment in the learned heatmap.
LMM Baseline. We ablate the effect of different LMM baselines vision-centrie task shown
in Table 3c Since LLaVA-v1.5 employs CLIP ViT-L/14-336 as stronger vision backbone, we use substitute the original CLIP ViT-L/14-224 in LLaVA VLaVA-v1.0, denoted "LLaVA-v1.0* in Table 3c to respectively ablate the effects of stronger vision backbone and multimodal language model. I can be proved that both stronger vision backbone and multimodal language model can benefit dense vision- languaean alignment.
4.4 Generalization Evaluation
Generalization to Unseen Datasets. To evaluate the generalization ability of our method, we perform zero-shot evaluation on PASCAL VOC2007 val set 822. As illustrated in Table 6 our method demonstrates superior generalization ability than other generalist models. is worth noting that compared with InstructCV which also inherits the enormous word vocabulary of LLM, our method outperforms it on VOC zero-shot evaluation by 16.2 AP. Besides, even compared with specialist models trained on VOC dataset (i.e, RetinaNet and Faster R-CNN Table our method still achieves comparable performances.
Generalization to Unseen Tasks. To prove that the heatmap produced by our first-stage task- agnostic training is a powerful intermediate representation for scaling to more vision-centric tasks, apply simple decoding rules the heatmap make our Lumen support the object counting task [83]. As shown in Fig 4 our Lumen can make the correct prediction even without specifically training on the object counting task.
Conclusions
In this paper, we present Lumen, a Large multimodal model with versatile vision-centric capabilities enhancement. Within the Lumen, we employ a decoupled design to initially promote learning task-agnostic vision-language dense alignment, and subsequently collaborate with task-specific decoding operations to address diverse tasks. With the task-agnostic and task-specific design, Lumen significantly broadens the range of visual tasks that existing LMM generalists can tackle, and also maintains general visual understanding and instruction following capabilities.
10

---

6 Acknowledgements
This work was supported in part by National Natural Science Foundation of China Project (No. 623B2027) and National Natural Science Foundation of China Project (No. 62072116).
References
[1] Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, et al. Map-neo: Highly capable and transparent bilingual large language model series. arXiv preprini arXiv:2405. 19327, 2024.
[2] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. arXiv preprint arXiv:2403. 04652,, 2024.
[3] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision- languuge understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[4] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 238.8885 2023.
[5] Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context, arXiv preprint arXiv:24044 13046, 2024.
[6] Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409. 12959, 2024.
[7] Zekun Wang, King Zhu, Chumpu Xu, Wangchunshu Zhou, Jiaheng Liu, Yibo Zhang, Jiashuo Wang. Ning Shi, Siyu Li, Yizhi Li, et al. Mio: A foundation model on multimodal tokens. arXiv preprint arXiv:2409. 17692222...7 2024.
[8] Pengkun Jiao, Xinlan Wu, Bin Zhu, Jingjing Chen, Chong-Wor Wah Ngo, and Yugang Jiang. Rode: Linear rectified mixture of diverse experts for food large multi-modal models. arXiv preprint arXiv:2407.11 12730, 2024.
[9] Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Vidcompress: Memory-enhanced temporal compression for video understanding in large language models. arXiv preprint arXiv:2410.11417., 2024.
[10] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[11] Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A modtl-modal visual question answering benchmark for autonomous driving scenario. arXiv preprint arXiv:2305. 14836, 2023.
[12] Hao Shao, Shengju Qian, Han Xiao, Guanglu Song. Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing chain-of offtthougt reasoning in multi-modal language models. arXiv preprint arXiv:2403. 169999 2024.
[13] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[14] Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. More: Multi-order relation mining for dense captioning in 3d scenes. In European Conference on Computer Vision, pages 528-545. Springer, 2022.

---

[15] Bingqi Ma, Zhuofan Zong, Guanglu Song. Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models, 2024.
[16] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:23 10.09488, 2023.
[17] Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[18] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
[19] Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, and Jinqiao Wang. Griffon: Spelling out all object locations at any granularity with large language models. arXiv preprint arXiv:2311. 14552, 2023.
[20] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10833 2021.
[21] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. 2021.
[22] Lingxi Xie, Longhui Wei, Xiaopeng Zhang, Kaifeng Bi, Xiaotao Gu, Jianlong Chang, and Qi Tian. Towards agi in computer vision: Lessons learned from gpt and large language models. arXiv preprint arXiv:2306. 08641, 2023.
[23] Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint 261660040..16160, 2024.
[24] Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jic, Lin Ma, Guangrun Wang, and Xiaodan Liang. Making large language models better planners with reasoning-decision alignment. In European Conference on Computer Vision, pages 73-90. Springer, 2025.
[25] Yunxin Li, Baotian Hu, Xinyu Chen, Lin Ma, Yong Xu, and Min Zhang. Lmeye: An interactive perception network for large language models. IEEE Transactions on Multimedia, 2024.
[26] Sean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23753 2022.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 27.44,, 2022.
[29] W Dai, J Li, D Li, AMH Tiong, Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: towards general-purpose langunel models with instruction tuning. arxiv. Preprini posted online on June, 15:2023, 2023.
[30] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16500 2023,
12

---

[31] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: massive discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-95611 2024.
[32] Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, et al. Cmmmu: A chinese massive multi-discipline multimodal understanding benchmark. arXiv preprint arXiv:2401. 1944, 2024.
[33] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He. Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311. 12793, 2023.
[34] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Bin Lin. Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.0443 2024.
[35] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large rision-language models? arXiv preprint arXiv:2403.22 20330, 2024.
[36] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through learning framework. In International Conference on Machine
a simple requence-to-sequeen Learning, pages 23318-23340. PMLR, 2022.
[37] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.
2022.
[38] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916. 2022.
[39] Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 35:31 35:31333-31346, 2022.
[40] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In European Conference Computer Vision, pages 521-539 Springer, 2022.
[41] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Marth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. generalist agent. arXiv preprint arXiv:2205.061111111 2022.
[42] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perce Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16804-16815, 2022.
[43] Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang. Wenhai Wang, et al. Uni-pereeiver v2: A generalist model for large-scale vision and vision-languee tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2691-2700, 2023.
[44] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint 2333 2023.
[45] Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.0222 2023.
13

---

[46] Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang. Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding multi-moda llms. arXiv preprint arXiv:2307.08581, 2023.
[47] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023.
[48] Yang Jiao, Zequn Jie, Jingjing Chen, Lin Ma, and Gu-Gang Jiang. Suspected objects matter: Rethinking model's prediction for one-stage visual grounding. In Proceedings of the 31st ACM International Conference on Multimedia, pages 17-26, 2023.
[49] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li. Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint Stti:2308.00692 2023.
[50] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.02644 2023.
[51] Yang Jiao, Zequn Jie, Weixin Luo, Jingjing Chen, Yu-Gang Jiang, Xiaolin Wei, and Lin Ma. Two-stage visual cues enhancement network for referring image segmentation. In Proceedings of the 29th ACM International Conference on Multimedia, pages 1331-1340, 2021.
[52] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ender decoder for vision-centric tasks. arXiv preprint a1Xiv:2305.11175, 2023.
[53] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[54] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint arXiv:2310.03744, 2023.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[56] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. In arXiv preprint 17440 2019.
[57] Dongkai Wang and Shiliang Zhang, Contextual instance decoupling for robust multi-person pose estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 11060-11068 2022.
(58) Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
2021.
[59] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[60] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 0bjects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEF/CVF international conference on computer vision, pages 8430-8439, 2019.
[61] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 12867) 2020.
14

---

[62] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Esion-ECCV 2016: 14th European Conference, Amsterdum, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[63] Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, et al. Ai challenger: A large-scale dataset for going deeper in image understanding. arXiv preprint arXiv: 171.0677 2017.
[64] Jan Hosang, Rodrigo Benenson, and Bernt Schiele. Learning non-maximum suppression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4507-4515, 2017.
[65] Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Han Hu, Dong Chen, et al. Instructdiffusion: A generalist modeling interface for vision tasks. arXiv preprint arXiv:2309.03895, 2023.
[66] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[67] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[68] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[69] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12193-12202, 2020.
[70] Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. Rtmpose: Real-time multi-person pose estimation based on mmpose. arXiv preprins arXiv:2303.07339, 2023.
[71] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Adder-modulltee detection for end-to-end multi-modal understanding. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[72] Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao. Xiaoshuai Sun, and Rongrong Ji. Seqtr: A simple yet universal network for visual grounding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23 27, 2022, Proceedings, Part XXXV, pages 598 615. Springer, 2022.
[73] Zhao Yang, Jiaqi Wang, Yansong Tang. Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155 18155-18165, 2022.
[74] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmenta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23592-23600 2023.
[75] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. mplug-2: A modularized multi-modal foundation model across text, image and video. arXiv preprint arXiv:2302.00400 2023.
[76] Yulu Gan, Sungwoo Park, Alexander Schubert, Anthony Philippakis. and Ahmed M Alaa. Instructev: Instruction-tuned text-to-image diffusion models as vision generalists. arXiv preprint arXiv:2310.00390, 2023.
15

---

[77] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 19t.:99965.555 2019.
[78] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
[79] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal ilms with generative comprehension. arXiv preprint arXiv:2307. 16125, 2023.
[80] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning foundation models in visual contexts. In International Conference on Learning Representations (ICIR), 2024.
[81] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pages
2017.
[82] M. Everingham, L. Van Gool, C. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303-338, June 2010.
[83] Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning count everything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3394 3403, 2021.
[84] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325-15336, 2023.
[85] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2300..0 05.09999 2023.
[86] Gen Luo. Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 10034-10043, 2020.
[87] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15116-15127, 2023.
16

---

Appendix
A Implementation Details
A.1 Detailed Design of V-L Dense Aligner
Cross-Attention
Image Embeddings
Width
FFN
Cross- Attentioon
[LOC]
[M]
Self-Attentonn
Height
Prompt
Output
Hcatmap
Token
Tokens
Figure 5: Detailed designs of "V-L Dense Aligner". [LOC] special token output from the LMM. [M] is the special token used for predicting the matching probabilities (i.e., heatmaps) in the first class-agnostic matching stage in the main paper. [H] and [W] are special tokens used as a simple box decoder for predicting additional height and width in the second class-specific decoding stage in the main paper. We use dotted lines to illustrate that these tokens are added the second stage.
The detailed design of the transformerrbased V-L. dense aligner shown in Fig 5 Motivated by SAM (50), regard [LOC] token prompt token, which carries semantics conveyed in the instruction and used to guide different output tokens, namely [M], [H] and [W], to generate corresponding predictions (i.e., heatmap, height and width in Fig Specifically, the prompt token and output tokens are first interacted with a self-attention layer. Then, the resulting outputs are interacted with image embeddings through sequence of rross-attention layers and ferd-forward network as shown in Fig to perform dual attention. The above process is repeated twice. The final outputs are produced by performing an demennt-wsee multiplication of the features of output tokens with the features of the image embeddings. For the sake of clarity and conciseness, we have omitted channel compression steps from Fig
As for the output tokens, in the first stage, we only use [M] to generate the predicted heatmap. In the second stage, add [H] and [W] to predict the height and width maps under the supervision of L1 loss as done in [56].
Table 7: Performance comparison with state-of-the-art specialists and generalists the visual grounding task.
RefCOCO test-A
ReiCOCO+ test-A
RefCOCOg val test
Method
val
test-B
val
test-B
Specialists
UNINEXT
84
92.64 90.56
94.33
91.46
85.24
89.63
79.79
88.73
89.73
G-DINO-L
85
93.19
88.24
82.75
88.95
75.92
86.13
87.02
Generalists
VisionI.L 52
86.70
OFA-L 36
79.96
83.67
76.39
68.29
76.00
61.75
67.57
67.58
Shikra-7B 47
87.01
90.61
80.24
81.60
87.36
72.12
82.27
82.19
Lumen-7B (Ours)
88.59
92.06
83.55
80.35
87.59
72.20
83.62
84.44
17

---

B
Complete Results on Visual Grounding and Referring Segmentation
We provide the complete performance comparison with existing approaches on visual grounding and referring segmentation tasks as shown in Table and Table 8 We will analyze the results in the following parts.
B.1 Visual Grounding.
We report the visual grounding performances of our Lumen on RefCOCO, ReiCOCO+ and Ref- COCOg datasets in Table First, compared with VisionLLM-H, generalist that can also perform object detection and instance segmentation like us, our method outperforms on the RefCOCO val set even without using strong vision encoder, e.g., InternImage-B used in VisionLLM-H Besides, compared with OFA-L and Shikra-7B, which reformulate the box prediction as a sequence generation manner, method can achieve better comparable performances with them. It indicates that predicted heatmap can capture the correspondence between visual regions and complex language concepts.
B.2 Referring Segmentation.
We report the referring segmentation performances of our Lumen RefCOCO, RefCOCO+ and RefCOCOg datasets in T First, compared with other generalists, our Lumen surpasses them most benchmarks by clear margins. Besides, is worth noting that we do not any pixel- level supervision (i.e., segmentation masks) throughout our training process, but our method still achieves comparable performances with specialist models the challenging RefCOCOg benchmark.
Table 8: Performance comparison with state-of-the-art specialists and generalists the referring
image segmentation task.
ReiCOCO test-A
RefCOCO+ test-A
RefCOCOg val
Method
val
test-B
val
test-B
test
Specialists
MCN
86
62.4
64.2
59.7
50.6
55.0
44.7
49.2
49.4
LAVT
73
72.7
75.8
68.8
62.1
68.4
55.1
61.2
62.1
ReLA
74
73.8
76.5
70.2
66.0
71.0
57.7
65.0
66.0
LISA
49
74.1
76.5
71.1
62.4
67.4
56.5
66.4
68.5
Generalists
X-Decoder
87
64.6
Unified-1O
38
46.4
46.1
48.1
40.5
42.2
40.2
17.3
InstructDiff
665
61.7
65.2
60.2
46.6
52.3
39.0
67.4
Lumen (Ours)
70.7
75.5
66.2
62.8
69.6
55.4
65.1
67.0
C More Ablation Studies
C.1 Input Sizes
To examine the impact of various V-L dense aligner input resolutions, we resize the input image into three distinct sizes, concurrently adjusting the positional embeddings of the original CLIP ViT. As shown in Table 9a| enlarging the input size from 336 x 336 (default input size of CLIP ViT-L/14) to 448 x 448 can achieve nontrivial performance boosts. However, further increasing the size to 896 x 896 will harm the performances. This phenomenon indicates that excessively increasing the input size (to a scale dramatically different from its pretraining) will break the semantics of visual features, and thereby damaging the vision- languae alignment.
18

---

Table 9: Ablation studies on model designs.
(a) Effect of different input sizes for the dense alignment.
(b) Effect of different pre-trained vision encoders.
Vis Enc.
AP
APso
AP
Input Size
AP50 50
AP75
SAM VIT-H/16-1022
14.5
24.0
14.1
336x 336
26.1
40.2
25.6
CLIP VIT-L/14-336
28.4
45.0
28.1
448 448
28.4
45.0
28.1
x96x
24.5
36.4
22.5
(c) Effect of different K value on dense prediction (d) Effect of different numbers of training itera- tions.
task.
#
50
100
200
#lteration
10k
20k
30k
40k
50k
AP
26.3
27.6
28.4
28.5
AP
28.4
31.6
34.5
35.2
35.3
AP50
39.5
43.2
45.0
45.3
APso
45.0
49.2
52.4
53.2
53.2
AP73
27.2
28.0
AP75
31.5
34.7
35.5
35.8
C.2
Vision Encoders
Since the input size can be seamlessly scaled up as discussed above, it is worth further exploring to leverage vision encoders naturally adaptive to high-resolution image inputs. Toward this end, we employ SAM ViT-H/16 [50]. which takes 1024 1024 images as inputs, as an additional vision backbone to provide high-resolution visual features for V-L dense aligner. However, as shown in Table 9b] the utilization of SAM ViT-H/16 results significant performance deterioration. We posit that this phenomenon can be attributed to the inadequate alignment between the SAM visual encoder and the language modality. Building upon the aforementioned experimental findings and analysis, we think that embracing vision encoders capable of processing high-resolution image inputs without compromising semantic coherence can pave the way for further enhancing the vision-centric capabilities of L.MMs.
C.3 Number of K for Dense Prediction Tasks
We also investigate the impact selecting different values for K on the dense prediction tasks, e.g.- object detection. As shown in Table 9c when K increases from 20 to 100, the model's performance consistently improves, which can be attributed to the increased recall of true positives. increases from 100 to 200, the model's performance does not improve evidently, therefore, we set K as 100 in our implementation.
further
C.4 Training Epochs
We record the variations in detection performance as the number of training iterations increases in Table 9d can be observed that the model's performance slowly increases as the training proceeds. Due to the limitation of computational resources, did not train model for longer time. The convergence speed might limited by the optimization difficulty of utilizing a single [LOC] token to query the entire image regions. A feasible solution to mitigate this problem is to employ more special tokens like [LOC] serving as object queries in DETR [67] with the Hungarian matching for label assignment, which accelerate training and promote performances proved in VisionLLM (52) We do not adopt this design as it is unclear whether these tokens customized for object detection can generalize well to other tasks.
D Limitations
Due to the vision- language dense alignment manner of the introduced V-L connector, one heatmap generated by can only reflect areas densely activated by single instruction. generate individual heatmaps for each category in standard vision-centric tasks like object detection and pose estimation, the model must be run separately for each category. This limitation can be addressed in the future by reformulating vision-centric data into instructions that simultancously indicate multiple
19

---

categories. In our implementation, we use parallel programming techniques to expedite evaluation. As result, our Lumen requires -5 hours evaluate COCO val set on single A100, which faster than Griffon 19 (~10 hours).
Instruction: What the Iocation of persowr? Respond with boa.
Phot
locanon of all ambretias." Respond
location all traffic fight?
Respond
Instruction:
Iostruction: What the all boats? Respordl bax.
Instrnction: What
the location of persons? Respoud
bashetbalis? Repood with
Instruction:
Instruction: fiod of busey? Respond with mask.
Instruction:
Can find of all persour?
Can find of all Respond with mack:
Rerpand with much
Instruction:
Instruction:
Instruction:
Care you
Can you find location
Can you find
the location
the location backpackk
pazaoos?? Respooud with
Repoad with mask
Reypoed with
mast
Instruction:
Where the
Where is the locatin of nase?
Where location of right eye? Respudd
BUS
location of left car?
Reyooui point
Respond point
point
Instruction:
Instruction:
Instruction:
Where
Where the locatiou of left hip? Respond
Incutiuu of left boce? Respond with
location of right kip?
Regouu potnt
Instruction:
Instruction:
Instruction:
WPiat cluir
What is a brown
What ts a
shhar Booking
woodon ohair to teft ear? Respond
Aeft behind the
ear? Rerpand
Reguood with bat.
wuii bax.
sont bar.
Instruction:
Instruction:
Whar is zebra
What zehro
facing the left
rhat on the
twith shorter
skls
Respond
sails in this
Respond
Respona with
Instruction:
What bond foud with
What
white bovel ton the loft this umogee Resyond with mach.
unauge soup net
glass of been Respond
Respond with wand..
Instruction: Whar
Instruction: bodding an toragge bug? Regpond
Instruction:
Boldingg phoue the image? Respond wiith
ones holding her
Reyoouu seuar
Figure 6: Qualitative results of versatile vision-centric capabilities of proposed Lumen. We use different colors of dotted rectangles to denote performing different tasks.
20

---

E More Qualitative Results
We provide qualitative results demonstrate the versatile capabilities of our Lumen shown in Fig Overall, the abundant qualitative results prove that our method possesses versatile vision-centrid capabilities. It worth noting that the case within the second row, we instruct the model detect the non-existing object (i.e., "baskerball" in Fig(o). the maximum activation across the heatmap is relatively low, and filtered with a pre-defined threshold. Therefore, the model will not generate any output for this wrong instruction illustrated in Fig
F Broader Impacts
This research is expected to enhance the visual perception capabilities inherent in LMMs. Additionally, since our approach leverages open-source pre-trained LMMs and vision encoders, it requires low training resources, thus reducing the carbon footprint. Currently, we do not foresee any substantial negative ethical social impacts.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims and contributions clearly stated in the abstract and introduction, and are supported by experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Appendix D for discussion of limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Theoretical assumptions and proofs do not apply to this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Appendix and A. for detailed training process and architec- tural designs.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification: Due to the privacy policy of the institution the authors collaborated with, the code should be published only after obtaining authorization.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Sec Bec4.1am and Appendix 4.1 A.I for experimental details. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the standard evaluation protocol of each benchmark, and their evaluation results are stable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix 4.1 for compute resource details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix F for details.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper aims at equip the LMM with vision-centric capabilities, without introducing high-risk language data for training the model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and pretrained models used in this study are open-sourced and licensed for academic research purposes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Due to the privacy policy of the institution the authors collaborated with, new assets like code should be published only after obtaining authorization.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Crowdsourcing and research with human subjects are not involved in this
paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Human subjects are not involved this paper.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
27

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "wGwWfFf5yk", "reviewer": "Reviewer_AJBg", "summary": "The paper proposes to augment the output of a Large Multimodal Model (LMM) with three tasks, namely detection, segmentation, and pose estimation. To that end, Lumen first predicts heatmaps conditioned on the encoded input image and instruction. This heatmap is then used by task-specific decoders to predict grounded bounding boxes, segmentation, and poses.", "questions": "- L267 “...10.000 iterations in the first phase…”: it is not clear how it compares to the previously obtained results.\n- Why are the RC SAM results in Tab. 4 b) better than in Tab. 1?\n- How are predictions for Fig. 4 obtained? Is it done by counting the peak points, or does an LLM make the prediction? This should be stated clearly to avoid confusions.", "limitations": "Limitations of the work are not discussed. For example, what are the limitations of using the heatmap as a “powerful intermediate representation” (L313)?", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The analysis of pre-training tasks can be insightful and useful for the community. In general, the ablation section can be useful to the community, though it has many weaknesses (see below). \n- Augmenting the output of an LLM with dense vision outputs is a useful research direction to explore", "weaknesses": "- Some claims are not well-supported (L74-80)\n    - (1): it is not clear what it means to “unleash the vision-centric potential of [an] LLM…”. The method still uses the “V-L Dense Aligner” and trained box and semseg mask decoders to make the prediction. The value of an LLM in the proposed approach is not ablated properly. While Tab. 3 c) gives a hint towards that question, one improvement (1.0->1.0*) is actually a better vision encoder, and the other (1.0*->1.5) might be explained with more instruction-tuning data.\n    - (2): “Adapt to tasks …. w/o requiring specialized datasets”\n        - Fig. 3 suggests that specialized datasets for each task are used.\n    - (3): “exceeds existing llm-based approaches.”\n        - Most “LMM generalists” just don’t do most of the tasks\n        - Differences with [12, 59] are not discussed to make sense of the results. What exactly differs there? Why these are reasonable baselines, e.g., [59] is a diffusion-based one.\n        - Similarly, comparisons in Tab. 6 lack contextualization, e.g. [13, 59]. What makes the proposed model generalize to an unseen dataset? What is missing from the baseline from [41] that benefits from a similar heatmap representation?\n- The ablation section lacks details and proper discussion, albeit being one of the most useful sections of the paper:\n    - Training “phases” are first mentioned in L267 but never described properly. L267 suggests 1st phase is task-agnostic and 2nd is, presumably, task-specific. However, A.3 suggests that it’s only about training data sampling.\n    - The implication in L290 for Tab. 4 a) is unclear. What does it mean to have “complete vision-language interaction”, and why does the convolutional architecture not have it (or cannot have it)?\n    - Similarly, the conclusion in L296 is unclear. Is the proposed framework not already improving the “dense vision-language alignment”?\n    - In Tab. 5, it is not clear why FP-C for 2 is dropped compared to 3. According to A.3, all datasets are present during this stage. Why do the authors think separate stages are needed to achieve better performance? Is it longer training?\n- Minor: There should be a justification why these 3 tasks are “fundamental.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1-1: The value of the LLM**\n\n**A1-1**: In the workflow of Lumen, ***the LLM plays a crucial role in jointly comprehending the open-ended instruction and visual content (which is LLM's strength), and its understanding of the inputs is further condensed into the hidden state of the [LOC] token.*** The V-L dense aligner and task-specific decoder are responsible for ***efficiently*** translating such high-level information into the heatmap space and task output space. Experimentally, the LLM's rich knowledge and strong understanding of multimodal inputs are important for tackling vision-centric tasks as verified below:\n1. By comparing (1.0*->1.5) in Tab.3c you mentioned, it can be proved that enhanced comprehension ability of LLM can promote vision-centric task performances.\n2. As shown in Tab.6, LLM also facilitates our model to generalize to unseen categories well, where the general-purpose content comprehension capability of the LMM plays a crucial role.\n\n**Q1-2: The meaning of *`\"specialized datasets\"`***\n\n**A1-2:** We refer to the *\"specialized datasets\"* as the dataset meticuously curated in Griffon (L78), which uses GPT4-v as the data engine to generate dialogues customized for object detection. In contrast, we only sample template questions and answers from pre-defined templates as shown in Fig3, which requires much less efforts. \n\n**Q1-3: Comparison with existing LMMs**\n\n**A1-3:** For clarity, we respond your concerns as below:\n1. Existing LMM generalists can NOT be easily extended to versatile vision-centric tasks. Please refer to **Common Questions Q1** for more details.\n2. We use Griffon and InstructCV as comparable baselines because they share similar motivations with us in extending vision-centric capabilities for foundation models in the ***instruction-following manner***. To achieve this goal, they focus on curating specialized datasets with the aid of an external data engine (i.e, a powerful LLM). In contrast, we hide the inductive bias in vision tasks with the proposed decoupled learning paradigm during tuning the LLM. \n3. We use InstructCV and Pix2Seq-v2 as comparable baselines in Tab.6 as they are also generalist models similar to our Lumen. As discussed in **A1-1**, the multimodal comprehension capability of LLM facilitates generalization, and therefore the specialist model (i.e, [41] you mentioned) cannot generalize well even with a heatmap representation. \n\n**Q2-1: Explanation to the training phases**\n\n**A2-1:** There might be a misunderstanding. Both phases belong to task-agnostic stage as addressed in ***Appendix A.3 (see L582)***. In the phase 1, we focus on ***promoting dense instance perception ability*** by using vision-centric data only. In the phase 2, we further add VQA data to ***reconcile the learned dense perception ability and general-purpose conversational ability***. Since most parts of task-agnostic decoding don't need training, we only train the lightweight box decoder on COCO after the phase 1 and 2. Sorry for the confusion, we will clarify this in the revised version.\n\n**Q2-2: Explanation to the *`\"complete vision-language interaction\"`***\n\n**A2-2:** We use Tab.3a to prove comprehensive V-L interaction benefits heatmap generation. Since the V-L dense aligner is not our major contribution, we implement with a lightweight transformer although stacked convolutions can also promote such the interaction.  \n\n**Q2-3: Explanation to the conclusion in L296**\n\n**A2-3:** We use Tab.3b to prove that the pretrained mask decoder is not the major factor that influences the model performance. By further synthesizing the results of Tab.3a, where the performances decrease evidently due to heatmap degradation, we make such the conclusion that heatmap primarily influences the model performance more evidently than the mask decoder.\n\n**Q2-4: Explanation to *`\"FP-C\"`* changes and sparate stages**\n\n**A2-4:** By referring to the role of phase 1 stated in **A2-1**, *\"FP-C\"* drops without phase 1 is because of decreased dense instance perception ability. The motivation of sparate phase training has been addressed in **A2-1**.   \n\n**Q3: Explanation to the *`\"fundamental\"`***\n\n**A3:** These 3 tasks provide scene parsing capabilities from different granularities (i.e, object-level, part-level and pixel-level), which can serve as the foundation for downstream tasks (e.g, human tracking, robot planning, etc). Thus, these 3 tasks are widely explored in previous generalists like Pix2Seq-v2, InstructDiffusion, etc.\n\n**Q4: Effects of training less iterations**\n\n**A4:** We have discussed the effects of training iterations in ***Tab.9d of Appendix C.4 (see L636~L645)***, which indicates that results of training 50,000 steps are better than those for 10,000 steps. \n\n**Q5: Explanation to results comparison between Tab.3b(Maybe a wrong table id here as Tab.4 has no subtables) and Tab.1**\n\n**A5:** The results of *\"Refer Seg.\"* in Tab.1 are evaluated on RefCOCOg as mentioned in ***L257~L258***, which should be corresponded to the *\"RCg\"* in Tab.3b rather than *\"RC\"*. And since we reduce the training iterations for efficient ablations as mentioned in ***L267~L268***, the result of SAM RCg in Tab.3b is lower than the one in Tab.1. \n\n**Q6: Explanation to Fig.4**\n\n**A6:** We first render the peak points selected from heatmaps into the image, and then feed this image to the Lumen again with an instruction of *\"How many green points in the image?\"*, and the resulting answers are used as the prediction results. We will clarify this in the revised version.\n\n**Q7: Discussion of limitations**\n\n**A7:** We have discussed the limitations of heatmaps in the ***Appendix.D (see L646~L654)***, i.e., heatmaps for different categories should be generated by forwarding the model separately, which can affect inference speed when evaluating vision-centric tasks like object detection. Therefore, we accelerate this process using parallel techniques.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M0AtVT7HXM", "reviewer": "Reviewer_bt1Q", "summary": "The paper introduces Lumen, a novel architecture for Large Multimodal Models (LMMs) that enhances vision-centric capabilities by decoupling task-agnostic and task-specific learning stages, thereby improving performance across various vision tasks while maintaining general visual understanding.\nThis architecture allows Lumen to achieve or surpass the performance of existing LMM-based approaches on a range of vision-centric tasks, including object detection, instance segmentation, and pose estimation, while preserving its ability to follow instructions and understand visual content.", "questions": "See Weaknesses above.", "limitations": "Have been discussed in appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper suggests that the reformulation of visual tasks into a matching problem and the use of a heatmap for guiding decoding processes are effective strategies for enhancing LMMs.\n\nThe paper details comprehensive experimental results, ablation studies, and discussions on the model's generalization capabilities, demonstrating the effectiveness of Lumen in handling diverse visual tasks with minimal training efforts.\n\nThe proposed Lumen not only performs well on vision-centric tasks but also maintains general visual understanding and instruction-following capabilities.", "weaknesses": "The proposed method does show much stronger performance on VQA benchmarks than prior methods, as shown in Table2.\n\nHow is the proposed method of decoupling task-agnostic and task-specific learning stages better than one-stage methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: VQA performances compared to prior methods**\n\n**A1:** Since our major motivation lies in ***extending versatile vision-centric capabilities while keeping the general-purpose coversational capabilities of the LMM***, we do not meticously select the VQA data for training at present. Thanks for your suggestions, we will continue to improve the VQA capabilities of our model by upgrading the VQA training resources in the future work. \n\n**Q2: Comparison between the decoupled learning strategy and the one-stage one**\n\n**A2:** Decoupling task-agnostic and task-specific learnings facilitate the LMM to focus on learning fine-grained multimodal content comprehension, rather than being trapped in learning the diverse specialized decoding rules (e.g, the formulation of bounding boxes, instance masks and keypoints) introduced by vision-centric tasks. In contrast, the one-stage learning paradigm requires the LMM to simultaneously learn these decoding rules lacking in semantics and human conversations requiring semantic comprehension and logic reasoning, which is much more challenging than the decoupled one proposed in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zsFttoLPB", "reviewer": "Reviewer_kab5", "summary": "The paper proposes Lumen, a new LMM architecture that splits perception learning into task-agnostic and task-specific stages. Lumen finely aligns vision-language concepts, creating a shared representation for all visual tasks. This is then decoded with lightweight task-specific decoders requiring minimal training. Lumen outperforms existing models on vision-centric and VQA benchmarks, maintaining strong visual understanding and instruction-following abilities.", "questions": "1. Although Lumen is proposed as an LMM Generalist, its object detection performance lags significantly behind that of Task-specific Specialists or Vision Generalists. Therefore, is there a gap between using [H] and [W] tokens in the Aligner for regressing bounding boxes and using an additional decoder independently? Can the effectiveness of the [LOC] token guidance be demonstrated in a more advanced DETR-like object detection decoder to improve detection performance?\n\n2. Can the authors provide a more intuitive and specific explanation of the decoding paths in the model, such as the method for selecting peak points and decoding bounding boxes?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed Lumen model introduces an effective architecture that separates the learning of perception capabilities into task-agnostic and task-specific stages. This separation promotes fine-grained vision-language concept alignment, enabling Lumen to handle diverse vision-centric tasks more effectively.\n\n2. Lumen's ability to match or exceed the performance of existing approaches on a range of vision-centric tasks while maintaining general visual understanding and instruction-following capabilities demonstrates its potential for broad impact.", "weaknesses": "1. Limited Novelty of the Method. The main approach of the paper, which involves decoupling task-agnostic and task-specific learning stages and sending the [LOC] token output by the LLM into the Aligner as a guide, is relatively simple and lacks significant innovation. Similar concepts have already been explored in previous works such as Uni-Perceiver v2 and VisionLLM. The paper lacks more groundbreaking improvements.\n2. Insufficient Evaluation of Generalization to Unseen Tasks. The paper only demonstrates Lumen's generalization ability to the unseen object counting task, and only provides qualitative results. The limited evaluation is insufficient to showcase the model's ability to adapt to a broader range of unseen tasks, requiring more experiments for quantitative analysis. Moreover, a large portion of Table 1 is blank, which reduces the advantage of the proposed method compared to other baselines.\n3. Limited Benchmark Diversity. The experiments mainly focus on a few benchmarks such as COCO and RefCOCO, with the only unseen dataset being PASCAL VOC2007. Including a broader range of datasets, especially those involving different modalities and more complex tasks, would better demonstrate Lumen's capabilities in handling vision tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Comparison with UniPerceiver-v2 and VisionLLM**\n\n**A1:** There might be a misunderstanding. Our major contribution is that we ***use the heatmap as the effective intermediate representation to prevent the LMM being trapped by the inductive biases of different vision-centric tasks*** (i.e, the formulation of boxes, masks and keypoints), ***rather than the module designs themselves***. And the heatmap brings the following benefits that UniPerceiver-v2 and VisionLLM do not possess:\n1. Refelcting dense activation of the open-ended user instruction, heatmap can be seamlessly adapted to vision-centric tasks like object detection, pose estimation, with minimal training efforts. In contrast, Uni-Perceiver-v2 involves complex box and mask proposals generation during its visual encoding stage. VisionLLM uses a set of pre-defined query templates (100 in their implementation) like  `<cls><x1><y1><x2><y2>`, and conducts label assignment with the aid of Hungarian matching to handle the unordered nature of dense objects. And both UniPerceiver-v2 and VisionLLM cannot support point-level task like pose estimation.  \n2. Since heatmaps hide inductive bias of different vision tasks, the general-purpose conversational capability, which is the foundation of a LMM, is not affected when incorporating a large amount of vision-centric data during training as shown in Tab.2 of the main paper. As a comparsion, both UniPerceiver-v2 and VisionLLM do not comprehensively evaluate their VQA abilities.\n\n**Q2: Further generalization ability evaluation and explanation to Tab.1**\n\n**A2:** As you suggested, we first provide the quantitative results of our model on the FSCD-LVIS[1] test set for the object counting task below:\n| Model            |  MAE(&darr;)  |  RMSE(&darr;) |\n|------------------|:-------------:|:-------------:|\n| FamNet           | 60.53 | 84.00 |\n| Attention-RPN    | 64.31 | 64.10 |\n| **Lumen (Ours)** | **58.23** | **57.64** |\n\nBesides, we also evaluate our model on another unseen task ViPBench[2] to validate our model's generalization ability. Please refer to **A3** for more details.\n\nAs for Tab.1, the gray cells indicate the current method does not support corresponding tasks as addressed in the caption of Tab.1. Meanwhile, we have discussed in **Common Question Q1** that it is NOT easy to extend a LMM to versatile vision-centric tasks. On the other hand, by comparing the performances of our Lumen on each task with specialists and generalists along the same column, the capabilities of our model in solving versatile tasks can be validated.  \n\n\n**Q3: Further evaluation on more diversed benchmarks**\n\n**A3:** Thanks for your advice. to demonstrate our model's capabilities in handling vision tasks, we further evaluate our model on more benchmarks of vision-centric tasks, including Objects365-v1 val set for object detection, AIC[4] val set for pose estimation, and ReferIt[3] test set for visual grounding as shown in the table below:\n| Method           | Objects365 | AIC      | ReferIt |\n|------------------|:----------:|:--------:|:-------:|\n| Faster-RCNN (OD) | 19.6       | -        | -       |\n| HRNet (Pose)     | -          | **32.3** | -       |\n| TransVG (VG)     | -          | -        | 70.7    |\n| **Lumen (Ours)** | **20.4**      | 30.1     | **79.0**   |\n\nwhere *\"OD\"* and *\"VG\"* are short for object detection and visual grounding, respectively, representing the type of the specilist model. Since most of existing generalists (e.g, UniPerceiver-v2, mPLUG-v2, Griffon, etc.) only provide detection and grounding results on COCO and RefCOCO(+/g) benchmarks, respectively, we can only list available performances of specialists for comparison in the above table due to time limit. The results indicates that our model can also handle vision tasks on more diversed benchmarks. \n\nBesides, we also evaluate our model on ***another challenging task ViPBench***[2], which requires the model to ***densely comprehend instances highlighted by visual prompts*** before answering the questions.\n| Method           | Synthesized visual prompts | Visual prompts from human |\n|------------------|:--------------------------:|:--------------------:|\n| InstructBLIP     |            31.7            |         33.3         |\n| Qwen-VL          |            39.2            |         41.7         |\n| LLaVA-v1.5       |            41.6            |         40.2         |\n| **Lumen (Ours)** |          **44.5**          |       **45.2**       |\n\nAs demonstrated in table above shows, our model signficantly outperforms other methods due to the enhanced dense visual perception capabilities. \n\n- [1] Few-shot Object Counting and Detection\n- [2] ViP-LLaVA:Making Large Multimodal Models Understand Arbitrary Visual Prompts\n- [3] ReferItGame: Referring to Objects in Photographs of Natural Scenes\n- [4] AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding\n\n**Q4: Discussions on the DETR-like decoder**\n\n**A4:** We have discussed the DETR-like decoder selection in ***Appendix.C.4 (see L641~L645)***, where we acknowledged that a DETR-like decoder can promote the object detection performances because of its Hungarian matching-based label assignment. However, since our main contribution lies in ***extending versatile vision-centric capabilities for the LMM with less training efforts while keeping its general-purpose conversational capability***, rather than ***specifically optimizing the object detection task***, we do not select a DETR-like decoder in our model.\n\n**Q5: Explanation to decoding paths**\n\n**A5:** Regarding the peak point selection operation, the basic workflow can be referred to ***Sec 3.2 (see L186~L190) of the main paper***. For its code implementation, we use `torch.nn.MaxPool2d` with kernel size of 3 and stride of 1 to efficiently select peak points. \n\nRegarding the lightweight box decoder, we have illustrated its workflow in ***Fig.5 of Appendix.A.1***, as well as detailed descriptions in ***L555~L564***.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WlUXIo9Y9r", "reviewer": "Reviewer_1g9i", "summary": "1. This study proposes a new Large Multimodal Model architecture(Lumen) that prioritizes a vision-centric approach and addresses the limitations of current methods, which oversee the unique features of various visual tasks.\n\n2. It separates perception capability learning into task-specific decoders, leading to a shared representation for multiple tasks, significantly optimizing task-specific decoding.\n\n3. Comprehensive experiments prove their method outperforms existing LMM-based approaches in various vision-centric tasks and VQA benchmarks and maintains crucial visual understanding and instruction following capabilities.", "questions": "It appears that these perceptual tasks have merely been integrated into the MLLM in some fixed instruction formats. Are there any experiments or examples to demonstrate the new capabilities the MLLM has gained or the capabilities that have been enhanced by integrating these perceptual data?  such as stronger fine-grained perceptual abilities or reduced hallucinations?", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This study demonstrates the author's solid understanding of traditional perceptual tasks and their technical details and seamlessly integrates it with the latest MLLMs. While preserving the capability of the instruction following inherent to MLLMs, it incorporates a robust perceptual module.", "weaknesses": "1. The most significant issue of lumen is that it may impede the model's ability to scale up. Though get stronger perception performance by solving various visual perception tasks with extra task decoders, thus disrupting the emergence of a stronger capacity for visual reasoning and perception when it is trained with a lot of data. As validation, none of the top research teams developing Multimodal Large Language Models (MLLMs) meticulously design their visual decoders in this fashion.\n\n2. The proposed novel technical in the article, such as V-L Dense Aligner and Peak Point Selection, are commonplace in traditional perception tasks, mirroring the early fusion and language-guided query selections in [1]. This work appears to be an enhanced version of models like [2,3], with the introduction of dense query[4] to boost perceptual performance and the addition of more task decoders\n\n\n\n[1]. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection\n\n[2]. LISA: Reasoning Segmentation via Large Language Model\n\n[3]. NExT-Chat: An LMM for Chat, Detection and Segmentation\n\n[4]. Dense Distinct Query for End-to-End Object Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The design of Lumen**\n\n**A1:** As discussed in the **Common Question Q1**, it is nontrivial to extend the LMM with versatile vision-centric capabities. We further provide proofs below to further validate this claim:\n1. Compared with Griffon that refactorizes box coordinates into language answers, our Lumen surpasses it significantly even without meticulously curating high-quality conversations with the aid of powerful GPT-4v as training resources as demonstrated in ***Tab.1 of the main paper***.\n2. In the attached PDF file for the rebuttal, we show that even the most powerful LMM, GPT4-v, can not tackle versatile vision-centric tasks well.\n\n**Q2: Comparison with mentioned works**\n\n**A2:** There might be a misunderstanding. Our major contribution is that we ***use the heatmap as the effective intermediate representation to prevent the LMM being trapped by the inductive biases of different vision-centric tasks*** (i.e, the formulation of boxes, masks and keypoints), ***rather than the designs of specific modules***. And we discussed the benefits of our model in **A1 to Reviewer kab5**. \n\nBesides, both LISA and NExT-Chat do not prove their dense visual perception abilities like object detection and instance segmentation, while our Lumen can tackle these tasks benefitted from the heatmap representation. And the behavior of our heatmap is not similar to the mentioned Dense Distinct Query (DDQ). DDQ aims at selecting dense object queries distictive from each other from the image feature map to serve for better one-to-one label assignment process for object detection task. As the comparison, our heatmap serves as a unified representation for versatile vision tasks and is optimized without involving any label assignment process (e.g, Hungarian matching) as discussed in ***Appendix.C.4 (see L641~L645)***.\n\n**Q3: New capabilities enhanced by integrating perceptual data**\n\n**A3:** The integration of vision-centric data can promote fine-grained visual content comprehension ability in VQA task as demonstrated by comparing *\"FP-C\"* metric in ***#2 and #3 of Tab.5***, and relevant explanations can be referred to ***L281~L284 of the main paper***. Besides, following your suggestion, we also test our Lumen and the baseline method (i.e, reimplemented LLaVA-v1.5-7B) on HallusionBench[1] as shown below:\n|         |  qAcc  |  fAcc |  aAcc  |\n| :------ | :----: | :---: | :---: |\n| Baseline|  12.7  |  19.1 | 47.0  |\n| Lumen |  **14.9**  |  **21.7** | **48.7** |\n\nThe results prove that utilizing vision-centric data can also help our model reduce the hallucination.\n\n- [1] HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9cD7vSUvKp", "reviewer": "Reviewer_ecgp", "summary": "This paper introduces Lumen, a large multimodal model designed for versatile vision-centric tasks. Lumen features a decoupled architecture that initially learns a shared heatmap representation to address various vision-related tasks such as object detection and instance segmentation. Subsequently, this representation is decoded to perform these tasks seamlessly. Additionally, Lumen maintains robust capabilities in general-purpose visual question answering. Experimental results across various benchmarks demonstrate Lumen's superior performance compared to existing large multimodal models (LMMs), highlighting the effectiveness of the proposed method.", "questions": "Please refer to weakness", "limitations": "The authors have adequately addressed the limitations in the Appendix materials.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper has a good motivation of hiding inductive bias of each individual visual task introduced by specific output formats with the unified heatmap representation. This design is meaningful when using data from both vision-centric and VQA tasks to jointly finetune a LMM. Unlike the conventional instruction data (e.g., Pix2Seqv2, Griffon, etc.) converted from visual tasks, which reveal the detailed vision-centric task output formats (e.g., box coordinates for object detection, vertex coordinates of polygon for instance segmentation), the reformulated instruction data in this paper are less contaminated by massive in amount but lack in semantic visual task output knowledge. Consequently, the proposed method enhances visual perception capabilities without significantly compromising the LMM's general-purpose conversational abilities.\n2.\tThe two-phase training strategy is reasonable and effective. As the authors do not leverage decoding techniques specifically designed for dense perception visual tasks (e.g., setting multiple object queries and optimizing with Hungarian matching in object detection task), the convergence speed in learning the heatmap is relatively slow. Therefore, mixing the VQA data in this phase is uneconomic. And in the second phase, the authors reinforce the conversational capability by jointly tuning with VQA data and vision-centric data.\n3.\tThe experiments are sufficient and well-organized. The proposed Lumen achieves superior performances than exiting LMMs on vision-centric tasks, and exhibits comparable VQA abilities with LMMs proficient in conversations like LLaVA and Qwen-VL.", "weaknesses": "This paper has no major flaws, but I have some problems concerning the paper writing and model designs:\n\n1.\tThe organization of this paper should be reconsidered. The training recipe is crucial to reveal the development of conversational and dense visual perception capabilities, and ablations on the training phases are also discussed in the main paper. However, the detailed training recipe is included in the Appendix. Although the authors provide relevant pointer in the main paper, I think this arrangement of paper content is unsuitable. The detailed training recipe should be integrated into the main paper to ensure a clearer and more coherent presentation of the methods and results.\n\n2.\tThe inference speed is a concern. Since Lumen can only generate one heatmap by forwarding the model once, I am wondering the cost when evaluating the dense visual perception tasks as it needs to individually feed every class name into the model. \n\n3.\tI am wondering the accuracy of predicting task-specific tokens (e.g., [DET]/[SEG]) as routers? And if I modify the task prompts to more formats, can the model determines the correct task type? More discussions and proofs are suggested to be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The origanization of the paper**\n\n**A1:** Sorry for the confusion caused by the paper organization, we will reorganize the paper following your suggestion in the revised version.\n\n**Q2: The inference speed of the model**\n\n**A2:** We have discussed the inference speed of our model in the ***Appendix.D (see L646~L654)***. We acknowledged that the repetitive model forwarding required by generating heatmaps of different categories will increase the inference burden. Therefore, we parallize the inference procedure using the batch processing technique, and the inference time cost is lower than Griffon, another LMM-based method customized for solving object detection task as discussed in Appendix.D.\n\n**Q3: The prediction accuracy of routing tokens**\n\n**A3:** As you suggested, we first modify the task prompts as below. \n```\nOriginal Instruction: \"Please find the location of {description}. Respond with {format}.\"\n\nModified 1: \"What are {description} in the image. Please generate their {format}.\"\n\nModified 2: \"Generate {format} of {description} in the image.\"\n```\nAfterward, we radomly sample 100 data from each of five special token-related tasks, resulting a testing set with 500 samples in total. Then, we prompt the Lumen with the three types of instructions illustrated in the cell above, and calculate the accuracy of generated special tokens. The accuracy results are reported below: \n|     |  Original  |  Modified 1 |  Modified 2 |\n| :-- | :--------: | :---------: |  :--------: |\n| Acc.|     1.0    |     1.0     |     1.0     | \n\nThe results indicate that our model is not sensitive to the change of instruction and can generate the correct task routing token.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers for their efforts and valuable comments. We first address some common questions here, and then respond to each reviewer separately. We hope our responses can clarify the concerns of reviewers.

### Common Questions

**Q1: Comparison with LMM generalists**

**A1:** Compared with LMM generalists, Lumen accomplishes versatile dense visual perception tasks without compromising general-purpose conversational capabilities. This NOT easy because directly adapting outputs of various vision-centric tasks into language-oriented conversations to tune the LLM will face great challenges due to the inherent lack of ***orders*** and ***semantics*** in vision-centric tasks:
1. ***Lack of order:*** Vision-centric tasks like object detection and instance segmentation requires the model to densely generate boxes or masks for all instances of the same category. These boxes or masks are inherently unordered, making it challenging to reformulate them into sentences without confusing the model as discussed in ***L38~L41 of the main paper***. 
2. ***Lack of semantics:*** The output formats of versatile vision-centric tasks (i.e, boxes, masks and points) lack high-level semantics comparable to text words. Therefore, integrating conversation data reformulated from vision tasks can hurt the general-purpose conversation capability of LLM

To support the above claim, we also provide experimental comparison as proofs in **A-1 to Reviewer 1g9i** and the attached PDF file.
We will further clarify the above statements in the revised version.

Meta-review: The submission aims to train multimodal modes using a more " vision-centric" approach and proposes a method for it. After considering the submission and the rebuttal, all of the reviewers were positive about the submission and recommended acceptance. The AC agrees. However, several of the reviewers raised important questions and points of clarification on the writing, claims, and methodology (e.g., see the detailed comments by AJBg). They are important to be addressed in the final version. 

An additional piece of feedback that could improve the quality of the final version: similar to some of the reviewers, also the AC found the writing challenging. In particular, using phrases that are too ambiguous (below is a copied example) or more grandiose than they need to be. Also, the title appears quite general and doesn't prepare the reader for actually the technical content that is in the paper. The authors are strongly recommended to consider these when revising their writing for the final version. 

"it overlooks the intrinsic characteristics of diverse visual tasks and hinders the learning of perception capabilities."
Decision: Accept (poster)