Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing
Authors: Hao Fei
Shengqiong Wu
Hanwang Zhang
Tat-Seng Chua
Shuicheng YAN
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Large Language Model
Unified Large Language Model
Primary area: human-AI_interaction
TL;DR: We present a universal pixel-level vision LLM designed for comprehensive understanding, generating, segmenting, and editing of both static images and dynamic videos.

Abstract: Recent developments of vision large language models (LLMs) have seen remarkable progress, yet still encounter challenges towards multimodal generalists, such as coarse-grained instance-level understanding, lack of unified support for both images and videos, and insufficient coverage across various vision tasks. In this paper we present Vitron, a universal pixel-level vision LLM designed for comprehensive understanding, generating, segmenting, and editing of both static images and dynamic videos. Building on top of an LLM backbone, Vitron incorporates encoders for images, videos, and pixel-level regional visuals within its frontend modules, while employing state-of-the-art visual specialists as its backend, via which Vitron supports a spectrum of vision end tasks, spanning visual comprehension to visual generation, from low level to high level. To ensure an effective and precise message passing from LLM to backend modules for function invocation, we propose a novel hybrid method by simultaneously integrating discrete textual instructions and continuous signal embeddings. Further, we design various pixel-level spatiotemporal vision-language alignment learning for Vitron to reach the best fine-grained visual capability. Finally, a cross-task synergy module is advised to learn to maximize the task-invariant fine-grained visual features, enhancing the synergy between different visual tasks. Demonstrated over 12 visual tasks and evaluated across 22 datasets, Vitron showcases its extensive capabilities in the four main vision task clusters. Overall, this work illuminates the great potential of developing a more unified multimodal generalist.

VITRON: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing
Hao Feil.2
Shengqiong Wu
Hanwang Zhang Tat-Seng Chua² Shuicheng Yan
1, Skywork AI, Singapore 2 National University of Singapore 3 Nanyang Technological University
haofei37Cnus. edu. sg swu@u. nus edu hanwangzhangêntu. edu. sg dcacts@nus.edu.
shuicheng.
Bauichhngg.aankkullun www.yynnnnnniiinnnm -nnninc com
Project Homepage:
github. io/
Low-level Visual Semantics
High-level Visual Semantics
Vision Segmentation & Grounding
Pixel-level Vision Understanding
Panoptic
Instance
Semantic
Refentng
Image/Viden Captioning
Referiing Captoning
Image QA
Video QA
Fisual Understanding
Phrase Grounding
Video Grounding
EDWIRE 27 satte
O Whan ln becaid
Language Image Reineeal
1 Language V Video Rebrivaal
Object Segmentation (Tracking)
Video Temporal Grounding
Vision Synthesis Generation
Vision Editing Inpainting
VITRON
Adding
Removing
Text-to-Imag Image ruto gaz
Texb-to-V Video Gemeration
lou WiRSIn
Reg Replacing
Moving
Visual Genenating
Image- V--Vidl Video Generation
Style Changing
Color Changing
Figure I: VITRON supports four main task clusters of visions, spanning visual comprehension
visual generation, from low level to high level.
Abstract
Recent developments of vision large language models (LLMs) have seen remark- able progress, yet still encounter challenges towards multimodal generalists, such as coarse-grained instance-level understanding, lack of unified support for both im- ages and videos, and insufficient coverage across various vision tasks. In this paper,
"Shuicheng Yan is the corresponding author. This work was performed when Hao Fei was Associate Member, and Shengqiong Wu was Intem Skywork AI.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

we present VITRON, universal pixel-level vision LLM designed for comprehen- sive understanding, generating, segmenting, and editing of both static images and dynamic videos. Building on top of an LLM backbone, VITRON incorporates encoders for images, videos, and pixel-level regional visuals within its frontend modules, while employing state-of-the-an visual specialists as its backend, via which VITRON supports a spectrum of vision end tasks, spanning visual comprehen- sion to visual generation, from low level to high level. To ensure an effective and precise message passing from LLM to backend modules for function invocation, we propose novel hybrid method by simultaneously integrating discrete textual instructions and continuous signal embeddings. Further, we design various pixel- level spatiotemporal dision-langune alignment learning for VITRON to reach the best fine-grained visual capability. Finally, a cross-task synergy module is advised to learn to maximize the task-invariant fine-grained visual features, enhancing the synergy between different visual tasks. Demonstrated over 12 visual tasks and evaluated across 22 datasets, VITRON showcases its extensive capabilities in the four main vision task clusters. Overall, this work illuminates the great potential of developing a more unified multimodal generalist.
Introduction
Recently, the field of multimodal large language models (MLLMs) has witnessed rapid and flourishing development across multiple communities. Extensive research efforts have been directed towards augmenting powerful, purely language-based LLMs with modules capable of visual perception, thereby extending their applicability to MLLMs [1, 49, 63, 127, 77, 111, 27]. MILMS, such as BLIP- 2 [49], LLaVA [63], MiniGPT-4 [138] and GPT-4V [121] etc., demonstrate robust and exceptional capability in image understanding, paralleling the deep semantic comprehension of language. In the realm of vision, the ability to process and comprehend dynamic videos is equally critical. Concurrently, several MLLMs have emerged with a focus on video understanding, e.g., VideoChat [50] and Video-LLaMA [128), demonstrating significant advancements in video comprehension.
Subsequent studies have sought to further expand the capabilities of MLLMs, with efforts bifurcating into two primary dimensions. On one hand, there's deepening of MLLMs' understanding of vision, transitioning from coarse, instance-level comprehension towards a pixel-level, fine-fined understanding of images, thereby achieving visual regional grounding capabilities, seen in GLaMM [84), PixelL. [85], and MiniGPT-v2 [11], etc., alongside the counterparts in -ixel-grounding video LLMs [74]. On the other hand, there's 's expansion in the breadth of functionalities MLLMs can support within the vision field. A portion of the research has already ventured into enabling MLLMs not just comprehend input vision signals but also to support the generation and output of vision content, with systems like GILL [43], Emu [96], etc., flexibly generating image content, and GPT4Video [105] and NEXT-GPT [114] achieving video generation.
We posit that the future trend of vision LLMs necessarily involves the enhancement of their capabilities towards a high degree of unification, i.c., multimodal generalists. However, observations reveal that despite the diversity of existing vision LLMs developed by the community, there is still a clear lack of unification. First, almost all existing vision LLMs treat images and videos separate entities, either supporting only images or videos [1. 96, 138, 128]. We argue for a unified vision MILM framework that concurrently supports both images and videos, acknowledging that vision inherently comprises both static images and dynamic videos- - both core components of our world and largely interchangeable in most scenarios. Second, the current support for vision functionalities in MILMS is found wanting, with most models only capable of understanding [63, 138), or at most generating images or videos [20, 105]. We contend that future MLLMs should embrace a broader spectrum of vision tasks and functionalities, enabling unified support for all vision-related tasks and achieving an "one for all' capability, which is vital for real-world applications, especially in vision creation that often involves series of iterative and interactive operations. For example, users typically start by generating images from text, transforming an idea into visual content; and then refining this content through further fine-gn -me-grained editing to add more details; following, proceeding to create dynamic content by generating videos from the images; and finally, engaging in several rounds of iterative interaction, such video editing, to enhance and finalize their creation. Last but not the least, for generalist integrated with various multimodal functionalities, one key lies in how to ensure that all tasks achieve their best performance as much as possible. This includes both that, 1) the instructions
2

---

Image Segmentation
Image Generation
Image Editing
Video Segmentation
Video Generation
Video Editing
Textual resporse
Task Instruction
Task specific Feature
Task-invari ant Fine- grained Festure
Large Language Model
LoRA
Projectio
Projection
Projection
Embedding
Video Encoder
Image Encoder
Sketch Encoder
Video
Image
User Sketch
Text
Figure 2: Technical overview of the VITRON framework.
from the LLM precisely conveyed to the downstream decoders, and 2) different tasks do not undermine each other but rather cooperate.
To address all these gaps, this paper introduces VITRON, pioneering universal pixel-level vision LLM, as shown Fig. 2. First, VITRON leverages a backbone LLM for comprehending, reasoning, decision-making, and multi-round user interactions. To perceive both image and video modal signals and support fine-grained user visual inputs, VITRON incorporates encoders for images, videos, and regional box/sketch-speeiffe inputs. On the backend, several state-of-the-art (SoTA) image and video modules are integrated for decoding and executing a wide range of vision tasks, spanning from lower higher levels, such visual understanding (perceiving and reasoning), generating, segmenting (grounding and tracking), editing (inpainting). To ensure that VITRON precisely conveys the LLM's 's decisions various backend decoder modules for function invocation, we propose novel hybrid method of instruction passing. Specifically, we enable the LLM to output not only discrete textual instructions, but also continuous signal feature embeddings passed to the modules. Finally, to maximize the functionalities of different modules within VITRON, we further devise synergy module, where we fully maximize the task park-eesssteent fine-grained visual features to be shared among different visual tasks.
The overall training for VITRON aims to equip it with robust and powerful vision understanding and manipulation capabilities. We first imbue VITRON basic MLLM skills by carrying out 1) vision- language alignment learning between the frontend encoders and central LLM, also 2) invocation- oriented instruction tuning. and 3) embedding modules. Going beyond this, we further try to strengthen VITRON's capacities. On the one hand, introduce fine-grained spatiotemporal vision grounding instruction tuning, training LLM on grounding predictions and pixel-aware perception for images and videos, such that VITRON sufficiently gains pixel-level visual perception. On the other hand, we utilize adversarial training [29, 100) to decouple task-specific features from task-invariant fime-grained visual features in signal feature representations. thereby enhancing the synergy between different tasks.
alignment tuning between LLM and backend
Extensive experiments covering 12 tasks across 22 datasets are performed. Leveraging its advanced architecture as multimodal generalist, VITRON demonstrates proficiency in a comprehensive range of vision tasks. Notably, the unified system's performance is on par with or even surpasses singleton state-of-the tee-thhert specialists on specific tasks. Further analyses reveal the efficacy of each design of the system. Our overall contributions summarized as follows.
To our knowledge, we for the first time propose a grand unified vision MILM,, VITRON, capable of pixel-level understanding, generating, segmenting, editing of both images and vidcos. 2 We
3

---

x Vision Supporting
Pixel/Regional Understanding
Segmenting/ Grounding
Cross-fask
Model
Generating
Editing
Synergy
Image
Video
Flamingo [1]
X
X
BLIP-2 [49]
x
X
MiniGPT-4 [138]
X
X X X X X V X X X X
LLaVA [63]
X
GILII [43]
Emu
X
MiniGPT:5 [135]
X
V
DreamLLMM [20]
X
GPT4Rol [130]
X
V
NExT-Chat [126]
X
MiniGPT-v2 [11]
Shikra [12]
Kosmos-2 [78]
GLaMM [84]
Osprey [125]
PixeILM [85]
LlaVA-Plus [64]
VideoChal [50]
X
Vidco-LLaMA [128]
X
Video-LLaVA [59]
X
Video-ChatGPP [67]
X
GPT4Videu [105]
X
PG-Viaco-Ciaaa [74]
NEXT-GPT[114]
VITRON (Ours)
Table I: Comparisons of existing (partially. imperfect coverage) representative vision MLLM.
introduce a more effective L.---dedecooee instruction-passing mechanism over both discrete texts and continuous signal embeddings. 3 We propose carrying various pixel-level vision-language spatiotemporal alignment learning for MILMs to reach the best fine-grained visual capability. We devise synergy module maximize the task aakk-perssttent fine ene-ggained visual features shareable among all different visual tasks, via which VITRON surpasses existing SoTA specialists' performance.
Related Work
Achieving profound understanding and comprehensive operational capabilities in vision, ranging from low-level visual pixel understanding [7, 65, 120, 46, 55, 122, 102, 52-54) to high-level compre- hension of overall semantics [19, 45, 70, 23, 31, 38, 39, 48, 115, 26, 56, 24], represents significant topic. Recent years have seen the development of highly potent large-scale vision models, such as VIT [21] and CLIP [83]. which have achieved remarkable vision understanding capabilities; models like SAM [42] and SEEM [139] have solved vision segmentation tasks; and diffusion- based models [34, 82, 73, 28, 112. 86, 25] have reached unprecedented performance in vision generation. Yet these models might lack an LLM as central decision processor, unable to flexibly interpret user intent or execute tasks interactively [97, 47, 114]. The emergence of LLMs has exhibited unprecedented intelligence capability [76, 16, 99]. Extending the success of language understanding in LLMs, re- searchers have promptly investigated and developed various MILMs, enabling LLMs to comprehend vision. By integrating high-performane LLMs, these models have been made capable of understanding vision signals [77, 1, 49, 81, 63]. Going beyond vision understanding, further research has aimed to enhance MLLMs, for instance, by endowing them with vision generation capabilities [43, 96] or supporting pixel-level understanding and grounding [130, 125, 85, 132, 1 10]. In Table I we summarize some existing popular vision MLLMs in terms of the vision function support.
vision encoders of images videos into language based
However, we observe that current research on vision LLMs lacks depth in two critical aspects. Firstly, current vision LLMs tend separate images and videos, supporting either one or the other. The construction of unified MLLM is crucial, as vision inherently encompasses both static images and dynamic videos, both of which are core components of our visual world. Thus, covering both aspects simultaneously is essential for optimally adapting to practical applications. Although models like NEXT-GPT [114] have relatively well-supported unification across various modalities, they fall short in supporting pixel-level in-depth vision understanding and comprehensive support for vision operation tasks. The second issue is the incomplete support for vision tasks by existing MLLMs. Most current MLLMs primarily support understanding images videos [63, 138], with

---

only a few supporting generation [20, 105] or editing/inpaintngg [113]. Building a generalist that can handle (almost) all vision-related tasks and operations in an end-to-end architecture should be the next major trend for vision ML..Ms Yet simply integrating existing visual specialists into an L.L to form MILMs is not sufficient enough, as genuine human-level Al should possess universal intelligence with robust cross-task generalizability [72]. Thus, it is necessary to further consider how to enable synergy effects [20] among different task specialists within a generalist, for which goal, we have devised a synergy strategy in this work. Besides, compared to the multimodal comprehension capabilities of MLLM, endowing MLLM with strong multimodal generative abilities is even more challenging. The key lies in how to effectively and unbiasedly convey MLLM's 's semantic understanding signals to the backbone decoder modules. There are two mainstream approaches to LLM-to-decond message passing within the MLLM community. One is based on discrete textual instructions [106, 90, 104], and the other on continuous signal embeddings [43, 20, 11 114]. However, we find that these two methods are complementary. Specifically, the former allows the L.LM to efficiently convey task execution commands to the backend modules through simple text, but it struggles to provide modality- for tasks, but fails to accurately convey execution intention (especially for managing many modules). In this work, we propose a hybrid method by integrating them together.
signals; the latter can conveniently carry the features needed
3 Architecture of VITRON
VITRON takes most common encoder-LLMe LLM-decoder' architecture paradigm, as in existing popular MLLMs [63. 20, 1 114]. The overall framework is shown in Fig. 2, where three key blocks are included: 1) frontend lision&languaaee encoders, 2) central LLM for semantics understanding and text generation, and 3) backend decoder modules for user responding and vision manipulation.
3.1 Frontend VinguanLanguage Encoding
For both images and videos, we employ the CLIP WTT-L/14@336px [83] as the encoder, respectively. The video encoder independently processes each frame, further employing average pooling across the temporal dimension to yield overall temporal representation features. Then, we employ a regional pixel-aware visual extractor as the sketch encoder for user interaction, e.g., clicking, drawing boxes or polygons, and making scribbles. We mainly follow [125], and use the object-based representations of mask regions that come from user's inputs, which not only encode the pixel-level visual features but also gather the spatial position information of each region. The region features are pooled with also the binary mask of spatial geometry of the object region encoded, and the resulting embeddings are used. Then, the multimodal feature representations are passed to L.LM via linear projection.
3.2 Core LLM
In VITRON, an LLM serves as the pivotal agent. Following the most common practice [15, 94, 128], we utilize Vicuna (7B, version 1.5). The LLM processes inputs from both language and visual modalities to perform semantic understanding and reasoning, and then make decisions. For visual comprehension tasks, LLM directly outputs textual responses for users. On the other side, LLM also needs to transmit signals and instructions to backend modules, directing them to invocate more complex tasks that go beyond text generation, such as visual segmentation, generation, and editing. As emphasized earlier, the ability of L.MSs to effectively and precisely convey messages is crucial to the performance of complex multimodal tasks. To this end, we propose fully integrating the advantages of the two common message- pass-ng methods: discrete rextual instructions and continuous signal embeddings. The former aids in accurately invoking different backbone modules (thanks to the LLM's proficiency in task dispatching). while the latter supplements with richer modality-preserred visual features that cannot be directly described through discrete text. As depicted in Fig. 2, the LLM outputs 1) text responses for users, 2) text instructions for module invocation, and 3) feature embeddings of special tokens. The feature embeddings are split into the task-specifie features and the task-invariant fine- grained visual-language features. Both the text instructions and feature embeddings are passed to backbone modules.
3.3 Backend Visual Specialists
To enable our MLLM with various visual task abilities, we integrate an array of singleton vision specialists into LLM. For image generation and editing, we integrate the diffusion- based model GLIGEN [57]. For image and video segmentation, we opt for SEEM [139]. For video generation, ZeroScope [8] and I2VGen-XL. [131] are utilized for text-to-video and image-to-video tasks, respec- tively. Lastly, for video editing functionality, we incorporate Video [9]. The text instructions from LLM first determine which task module to invoke; simultaneously, feature embeddings are fed
5

---

into the corresponding module's feature encoder to assist with task execution. Specifically, we design structured invocation template, including 1) Module name, 2) Invocation command, and 3) Region (optional) specifying fine-grained vision feature needed for certain tasks. The feature embeddings include both task-specific (ea features and iok-iivvaiant tne-grained features. The purpose of this design is to achieve feature decoupling, during which we aim to have the task-invariant fine-grained features shared as widely as possible among all tasks to facilitate synergy between different tasks.
4 Pixel-aware Synergistic Vision-Language Understanding Tuning
With the VITRON framework, now train the model with three stages of targets. First, we try to endow it with basic multimodal capabilities, i.e., comprehension and generation. Then, we engage in fine-grained vision grounding instruction tuning to further enhance the model's pixel-level perception abilities. Finally, we carry out cross-task synergy learning, maximizing the shared fine-grained features among all tasks.
4.1 Basic Multimodal Comprehension and Generation Skill Training
In the first stage of training. the primary goal is to equip the MLLM with basic multimodal under- standing and generation abilities, including the frontend alignment of encoder-LLMN as well as the backend alignment of LLM-decoder. Appendix $B.1 details all the following three types of training.
Overall Vision- Langugge Alignment Learning. This to ensure the input vision and language are mapped to unified feature space. Following prior common practice, we utilize datasets comprising image-caption pairs (CC3M [89]), 'videoccapioon pairs (Webvid [4]), and 'region-caption pairs (RefCOCO [40]) drawn from existing established corpora and benchmarks. When provided with an image, video, or specific visual region, we engage the frozen LLM to generate a text description or caption that aligns with the reference caption.
Text Invocation Instruction Tuning. This step of training aims to equip the system with the precise capability to execute commands, allowing the LLM to generate appropriate and correct invocation text instructions. To accomplish this, we collect a total of 55,000+ instruction tuning samples.
fumbedding-orientt
Decoder Aligmment Tuning. Besides using explicit textual instruction to (from LLM) should also
invocate downstream modules, the signal feature embeedding/repreennttiin be fed to the modules. Following [114], we align the feature embedding with all the visual modules' input encoders via the decoding side projection layers, i.e., by minimizing their distances.
4.2 Fine-grained Spatiotemporal Vision Grounding Instruction Tuning
A visual generalist should require strong capability of pixel-aware vision understanding of both images and videos. Thus, we propose a fine-grained spatiotemporal vision grounding instruction tuning for VITRON. The core idea is to enable the LLM to ground the fine-grained spatiality of images and the detailed temporality of videos. Appendix $B.2 extends more detailed descriptions of the following three learning aspects.
Image Spatial Grounding. Considering that the LLM alone can only output text, we design it to respond with the corresponding bounding box areas. We focus on two types of tasks: grounded image captioning [133, 137] and referring image segmentation [40].
Video
Grounding. For videos, the LLM must identify spatial regions and
ground them within the temporal context of the video, essentially achieving video tracking, Similarly, we explore tasks such as grounded video captioning [136] and referring video tracking [107].
Grounding-awan Vision QA. The grounding tasks mentioned above only touch upon the low- level aspects of vision perception. However, in many scenarios, it's essential for the LLM to pos- sess high-level, in-depth vision reasoning capabili- ties, building upon the foundational low-level pixel grounding. Thus, we further introduce grounding- aware vision QA, including Image-QA [88, 37] and Video-QA [124], enabling LLM to undertake semantic-level QA tasks based on the grounded results.
Task Output
Task ID Label
Backhone Module of V isual Specilists
Discriminator
Task
Tasss
Task
Task invariant Fine- grained Feature
Task sesk-specific Feature
Large Language Model
LoRA
Figure 3: Illustration of the synergy module.
4.3 Cross-task Synergy Learning
As generalist, directly invoking different specialists leads to critical issue: how to ensure that the different modules (tasks) work together synergistically? Otherwise, without such collaboration.

---

integrating them into single compound system would be meaningless. To achieve this, here we propose decomposing the signal feature embeddings into task-specific features and task-invariant fine-grained features. Intuitively, since all the visual tasks we focus on are fine-grained, the more extensively the task-invariant fine-grained features are shared among different tasks, the more these tasks can benefit from each other, thus gaining greater synergy. Thereafter, we introduce cross-task synergy learning module, as shown Fig. 3. We employ adversarial training [3] to decouple task-specific from task-invariant features. We first let different backbone visual specialists make task predictions based on these two features (via concatenation). Meanwhile, we encourage a third-party discriminator (acts classifier) to determine which is the current task based solely the shared feature representation. Ideally, once the discriminator can no longer accurately identify the task, the shared feature can be considered the most purified and broadly applicable across tasks.
Experiments
Now we try to quantify the performance VITRON on the four vision task groups, covering 12 tasks across 22 datasets. All the training of VITRON is conducted on 10x A100 (80G) GPUs. To ensure fair comparison, all subsequent experiments adopt settings same/similar to those of baseline systems, with evaluations following established practices. See more implementation details in Appendix $C. Due to space limits, more experimental results are presented in Appendix $D.
5.1
Results on Vision Segmentation
RefCOCO [40]
ReiCOCO+ [123]
RefCOCOg [68]
Method
Val
TestA
TestB
Val
TestA
TestB
Val
Test
LAVT [120]
72.7
75.8
68.8
62.1
68.4
55.1
61.2
62.1
GRES [61]
73.8
76.5
70.2
66.0
71.0
57.7
65.0
66.0
LISA [46]
74.1
76.5
71.1
62.4
67.4
56.5
66.4
68.5
NExT-Chat [126]
74.7
78.9
69.5
65.1
71.9
56.7
67.0
67.0
VITRON
75.5
79.5
72.2
66.7
72.5
58.0
67.9
68.9
w/o syng.
-2.4
-2.0
-1.9
-1.7
-2.1
-1.5
-1.8
-1.6
Table 2: Results (cloU) of referring image segmentation. 'w/o syng.': without synergy learning.
Image Segmentation. Table 2 presents the results of referring image segmentation three datasets: RefCOCO [40]. RefCOCO+ [123] and RefCOCOg [68]. We compare with several significant models, including state-of-the-art non-MLLM approaches and the MLLM baseline, NExT-Chat. It is evident that our VITRON, while slightly underperforming compared to NExT--CMe on the RefCOCO
& datasets, achieves superior performance the remaining sets.
Method
VidSTG [134] HC-STVG [98]
Method
J&F
3
G-DINO [65]
25.3
19.5
RDE [51]
77.4
73.6
81.2
Video-LLaMA [128]
28.6
26.1
XMem [14]
81.0
77.4
84,5
Video-ChatGPT [67]
32.8
20.8
DeAOT [122]
80.7
76.9
84.5
VG-Viido-LLaaA [74]
34.2
28.3
ISVOS [102]
82.8
79.3
86.2
VITRON
39.5
31.4
VITRON
84,2
81.5
86.7
w/o syng.
-4.3
-3.7
w/o syng.
-2.1
-1.3
-1.0
Table 3: Results (mIoU) of video spatial grounding on two datasets.
Table 4: Results of video object segmen- tation on DAVIS 17 [80] Test-Dev set.
Video Segmentation. For video segmentation, we explore two tasks: video spatial grounding (with bounding box) and video object segmentation (aka., video tracking; with mask). Table 3 showcases the comparisons between VITRON and current state-of-the-art (SoTA) video MILMs in video spatial grounding. It is clear that VITRON significantly outperforms PG- Video-LLaVA Table 4 presents comparison of VITRON with some SoTA systems in video tracking, where our system continues to demonstrate superior performance.
Method
METEOR
CIREr
5.2 Results on Fine-grained Vision Understanding
GRIT [109]
15.2
71.6
Kosmos-2 [78]
14.1
62.3
Next, evaluate VITRON's capability in achieving fine- grained vision understanding, focusing mainly on region- level tasks for both images and videos.
NExT-Chat [126]
12.0
79.6
MiniGPT-v2 [11]
15.0
86.4
GLaMM [84]
16.2
106.0
Osprcy [125]
16.6
108.3
Region-level Image Understanding. We test VITRON on tasks including image referring expression comprehension and image regional captioning. The comparisons and results
VITRON
18.0
111.6
w/o syng.
-3.0
-8.6
Table 5: Performance of image re-
gional captioning on RefCOCOg [68].
7

---

shown in Tables 5 illustrate that VITRON surpasses the best baseline across various datasets and metries, proving its strong and accurate fine-graine aineggrinedd semantic understanding of images.
The above two tasks focus solely on the model's ability to recognize at the region level. Taking a step further, delve deeper into assessing the capability for image semantics understanding, particularly through image-based Visual Question Answering (VQA) tasks. These tasks effectively reflect the model's proficiency in comprehending the deeper semantic content of images. Table 6 displays the results across a series of six datasets for image-based VQA. We primarily compare two groups of models: those with and without pixel-wise vision grounding capabilities. The findings indicate that models equipped with fine- the-grained grounding abilities indeed show stronger task performance, suggesting that fine-ge teeeggga grounding contributes to a more profound understanding of semantics. Notably, our VITRON achieves the highest performance among the models evaluated.
Ground? ActivityNet-QAA [124]
Method
Ground?
OKVQA [88]
GQA [37]
Method
Flamingo [1]
44.7
Accuracy
Score
BLIP-2 [49]
45.9
41.0
VideoChat [50]
X
2.2
InstructBLIP [17]
x
49.5
LLaMA-Adaples [30]
X
34.2
2.7
MiniGPT-44 [138]
X
37.5
30.8
Vidco-LLaMA [128]
X
12.4
1.1
LLaVA [63]
54.4
41.3
Video-ChatGPP [67]
X
35.2
2.7
Shikra [12]
47.2
Video-LLaVA [59]
X
45.3
3.3
MiniGPT-v2I1 [1]]
57.8
60.1
PG- VG-VVide-LLLVA [74]
39.9
3.3
VITRON
59.4
62.1
VITRON
51.0
3.7
w/o syng.
-2.0
-1.7
w/o syng.
-4.4
-0.6
Table 6:
Results (accuracy) on image-based
Table 7:
Results (accuracy and confidence Score) on video QA.
VQA.
Region-level Video Understanding. Similarly, for videos, evaluate the Region-level Video Understanding capability. Building on observations from images, we now directly engage in video QA tasks. Table presents the results on video QA across four representative datasets. Interestingly, while PG-Video-LLaVA has video grounding capabilities, it does not show better results than Video-LLaVA, which lacks grounding. However, our VITRON achieves superior performance. This indirectly proves that our system possesses more accurate video grounding capabilities (as previously demonstrated in Table 8), aiding better video semantics understanding.
5.3 Results on Vision Generation
Method
FID
Method
FID (1)
CLIPSIM
GLIDE [75]
12.24
CogVideo [33]
23.59
0,2631
SD [86]
11.21
MakeVideo [92]
13.17
0,3049
Method
FVD (L)
(t)
NEXT-GPT [114]
11.28
Latent-VDMM [86]
14.25
0.2756
AnimateAny [18]
642.64
63.87
Emu [96]
11.66
Latent-Shift [2]
15.23
0.2773
DynamiCrafter [116]
404.50
41.97
DreamL_M [20)
GILL [43]
VITRON
12.20
7.57
8.46
NEXT-GPT [114]
CoDi [97]
13.04
0,2890
0.3085
SEINE [13]
VideoCrafteri [10)
297.62
306.49
50.88
$4.02
w/o syng.
+4.4
VITRON
10.11
0,3682
VITRON
175.46
56.89
Table 8: Text-to-Image
w/o syng.
+3.17
-0.5672
w/o syng.
+96.24
-5.03
generation on COCO-
Table 9: Text-to-Vicd generation on MSR-VTT [118].
Table 10: Image-to- Video gen-
caption data [60].
eration on UCF101 [93].
Next, we assess our system's capabilities in vision generation, focusing on three of the most representa- tive types of generation tasks: text-to-image generation, text-to-video generation, and image-to-video generation. These tasks broadly cover the spectrum of image generation requirements. Tables 8, and 10 showcase how our VITRON performs in comparison to other SoTA systems, including both MLLM and non-MLLM synthesizers. The results clearly demonstrate that VITRON outperforms on all three tasks. For instance, in both text-to-image and text-to-video generation tasks, VITRON shows more advanced performance compared NEXT-GPT. Similarly, in the image-to-video generation task, VITRON still outshines the SoTA baseline, VideoCrafterl, showcasing superior results.
5.4 Results on Vision Editing
Image Editing. We use the MagicBrush dataset [129], which challenges models with an editing query that demands series of complex edits to an image. These edits include removing, changing, inpainting, and adding elements. Since there are currently no MLLM systems that support image editing, our comparison is limited to non- non-LLM expert systems. In Table 11, we present the perfor- mance of different models across various metries. VITRON demonstrates stronger performance on all metrics, indicating its stable image editing capabilities.
8

---

Method
CLiP (1)
CLIP img
CLIP (f)
LT
Method
Target- Editing
NonTarget-
InstructPix2Pix [6]
0.837
0.245
0.093
Unediting
MagicBrush [129]
0.123
0.883
0.261
Tex12L.VE [5]
4.5
1.3
PnP [101]
0.025
0.568
0.101
0,280
Tune-A- Video [108]
7.8
4.6
NT-Inv [71]
0.121
0.752
0.263
0,077
Finu-Fdit [91]
0.135
0.897
0.261
0.052
VITRON
8,9
8.2
VITRON
0.142
0.910
0.274
0.047
w/o syng.
-2.2
-1.6
w/o syng.
-0.012
-0,104
-0.078
0.036
Table 12: Human evaluation on video
Table 11: Image editing results on MagicBrush [129].
editing.
Video Editing, For video editing, the community currently lacks standardized benchmark and evaluation method akin those for image editing. Therefore, we opted for manual evaluation approach. We asked different video editing systems to edit the same video based on the same query, after which five individuals were asked to score the edited videos. The evaluation focused success of target content modifications and 2) the faihhuulnsssfidellty of non-target content. Table 12 presents the manual evaluation results for video editing. clear that VITRON outperforms the two baseline systems in both respects, showcasing superior video editing capabilities. Following this, we visualized the process of video editing by VITRON.
the
Discussions
Above demonstrate the overall efficacy of VITRON via extensive quantitative comparison. Now we take one step further, exploring how and why the system advances via in-depth analyses.
Discrete Textual Instruction Continuous Signal Embedding, Which Better? Firstly, explore different message-passing passing mechanisms to determine whether discrete textual instruction more beneficial,
whether continuous signal embedding is better for building multi-modal generalist. Also, we validate the pros and cons of the proposed hybrid method of message passing. We conduct tests on 6 tasks, where we compare the task performance of VITRON using the hybrid method (default setting), without signal embedding and without text instruction, well as the successful execution rate of the backend task module. Fig. 4 presents the results. As can be observed, overall, the performance under scenarios utilizing both methods is consistently better, which confirms the effectiveness of our hybrid mode. Meanwhile, we find that the method of text instruction more conducive to the successful execution of backend modules, but soft feature embedding seems to be more useful in terms of specific task performances.
B Vitron w/o Continons Signal Emb
B3
Vitron w/o Disrete Txt Instruction
Vitron
"Tek (doU)
70
100
Elen: Rate (%)
Took (mlot)
(7))
Taak (METEOR;
19
100
Exsec Rate (%6)
80
17
20
Image Segnemtttion (ReiCloogg
Video Spatial Grounding (VidSIG)
Iuage Regional Captioning (RelCoCog)
Exar. Rate (%)
Eone. (%))
Task (CLIP...)
Exer. Hate (%)
30
80
"Tok [+ID
Took (PID
22
&+
Text-to-Tmapp (CoCo-Caps)
Text-to-Video (MSR-VIT)
Imago Editing (MagieBrush)
Figure 4: The influences of using different strategies for message passing.
How Much Does Each Fine-grained Visual Grounding Learning Contribute? Next, validate the specific contribution of the various fine-grained visual grounding learning strategies proposed 84.2. Fig. (the top 4 relate image tasks, and the bottom video tasks) shows the impact performance when a particular learning strategy is removed. Generally, all these 3 types of fine-grae ine-grained visual grounding learning strategies are vital for different downstream tasks. For instance, grounding and referring segmentation tasks directly influence fine-grained visual recognition tasks, whereas tuning for grounding- aware visual QA considerably boosts cognition level QA tasks. This verifies the efficacy of our proposed fine-grat fine-grained visual grounding tuning strategies.
Does VITRON Really Achieve Cross-task Synergy? Finally, we investigate if our system could adequately support cross-task synergy. Based on the results of the ablation item for the 'synergy

---

a8 Vitron
BB w/o Ground-Ing Can Cap 30 w/o Cround-Vid-Cap
B Sefer-Ing--eg
BnIm--QA
Ing. Region Captioon
B Bwyo Refer- Refer-Vid--Te Vid-Track
BC Vid-QA
Iuagee Seg (clol)
(METEOR)
Image Editing
Video (Aoc.) Image (Anc.)
(ReCCooog)
(TeCCoCog)
Vid. Olg. Seg. (TH.F) (CL)PPmx))
(rusie
(OKVQA
60
60
50
14
Vid Spatial Ground
TextstoViil Video (FID ()
(mlot)
40
(iiSSGG)
10 12
VIT)-VTT
(AetivityNet-QA)
(AciviyyAl--AA)
78
20
Figure 5: The impact of various fine-graie fine-grained visual grounding learning strategies.
Image Referring-
Image Regional Captioning
Video Spatial Grounding
Video Object Segmentation
Image QA
Video QA
Text-to-Image
Text-to-Video
Image-to-Vides
Image Editing-
Image Refering
Image Regional Captioting
Viden Spatial Grounding
Video Object Segnentation
Image Q4
Video Vi QA
Ret-to-Image Image
Rextto-Vidan
Image-to-Vide Video
Image Editing
Figure 6: The synergy correlation between each pair of visual tasks. The deeper the color of the cell, the more synergistic they are between.
module' in Table to Table 12, we can observe that the synergy learning mechanism indeed positively influences overall performance. In Fig. we further study whether there is synergy between different tasks and their collaborative relations. For ease of study, we considered one-to-one mapping relationship, studying the cooperation between pairs of tasks one at a time. It is evident that the cooperative effects vary between different tasks. Tasks or backbone modules that rely more heavily on fine tne-grained visual features gained more significant improvements. This also demonstrates that our synergy learning module can successfully facilitate cross-task synergy.
Conclusion
In this work, we present VITRON, a grand unified pixel-level vision LLM for seamlessly understanding (perceiving and reasoning), generating, segmenting (grounding and tracking), and editing (inpainting) both images and videos. We further introduce a novel hybrid method of message passing that combines discrete textual instructions with continuous signal embeddings to ensure precise function invocation. Furthermore, VITRON employs pixel-level spatiotemporal vision-language alignment to enhance its fine-grained visual capabilities. A cross-task synergy module is also developed optimize the use of task-invariant fine fine-graine grained visual features, boosting synergy across various visual tasks. On 12 visual tasks across 22 datasets. VITRON exhibits extensive capabilities in visual segmentation, fine-grate fine-grained vision understanding, generation, and editing. Overall, this research showcases the great potential to build vision-language generalist that can advance toward more unified AL.
Acknowledgements
This research is supported by Skywork AI, NExT++ Research Center.
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr. Yana Hasson, Karel Lenc. Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Scrkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadch, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot learning. In Proceedings the NeurIPS, 2022.
[2] Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. CoRR, abs/2304.08477 2023.
[3] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. In Proceedings of the IJCAI, pages 4312 4321, 2021.
[4] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the ICCV, pages 1708-1718, 2021.
[5] Omer Bar- Tal- Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In Proceedings of the ECCV, pages 707-723. Springer. 2022.
[6] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the CVPR, pages 18392-18402, 2023.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier. Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the ECCV, 2020.
[8] Cerspense. Zeroscope: Diffusion based text-to-video synthesis. 2023. URL https::/ co/ cerspense.
[9] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency-awald diffusion video editing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 23040-23050, 2023.
[10] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.191 19512, 2023.
[11] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for languaee multi-tast learning. arXiv preprint arXiv:2310.094: 2023.
[12] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
[13] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang. Xin Ma, Jiashuo Yu. Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In Proceedings of the ICLR, 2023.
[14] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In Proceedings of the ECCV, pages 440-658. Springer, 2022.
[15) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 902023.
[16] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finete CoRR, abs//210. 14116, 2022.
language models.
[17] Wealiang Dai, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. CoRR, abs/2305.0651 2023.
[18] Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Fine- grained open domain image animation with motion guidance. arXiv preprint 1rX.v 2023.
11

---

[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the CVPR, pages 248-255 leee, 2009.
[20] Runpei Dong, Chunrui Han, Yuang Peng. Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint ar3.v:2309.1144.40 13499, 2023.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold. Sylvain Gelly. Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 words: Transformers for image recognition at scale. In Proceedings of the ICLR, 2021.
[22] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the CVPR, pages 5374-53745 5383, 2019.
[23] Hao Fei, Qian Liu, Meishan Zhang, Min Zhang, and Tat-Seng Chua. Scene graph as pivoting: Inference- time image-free unsupervised multimodal machine translation with visual scene hallucination. In Proceedings of the ACL, pages 5980-5994, 2023.
[24] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, and Tat-Seng Chua. Dysen-vdm: Empowering dynamics-aware text-to-video diffusion with llms. In Proceedings of the CVPR, pages 7641-7653, 2024.
[25] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Mcishan Zhang, Mong-Li Loe, and Wynne Hsu. Video-of-though:: Step-by-step video reasoning from perception to cognition. In Proceedings of the ICML, 2024.
[26] Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing video-language representations with structural spatio-temporal alignment. IEFF Transactions on Pottern Analysis and Machine Intelligence, 2024.
[27] Hao Fei, Yuan Yao, Zhuosheng Zhang, Fuxiao Liu, Ao Zhang, and Tat-Seng Chua. From multimodal Ilm to human-level ai: Modality, instruction, reasoning, efficiency and beyond. In Proceedings of the COLING: Tutorial Summaries, pages 1-8, 2024.
[28] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Fric Wang, and William Yang Wang. Training-free free structured diffusion guidance for compositional text-to-image synthesis. CoRR, abs/2212.05032, 2022.
[29] Yaroslav Ganin, Eygeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarii training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
[30] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan La,, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.1500 2023.
[31] Xin Gu, Guang Chen, Yufci Wang, Libo Zhang, Tiejian Luo, and Longyin Wen. Text with knowledge graph augmented transformer for video captioning. In Proceedings of the CVPR, pages 18941-18951, 2023.
[32] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jieho Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018
[33] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jic Tang, Cogvideo: Large-scale pretraining for text-to-video generation via transformers. CoRR, abs/2205. 15868, 2022.
[34] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Towards non-autoregressivve language models. CORR, 2021.
[35] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proceedings of the ICLR, 2022.
[36] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high diversiyyy benchmark for generic object tracking in the wild, IEEE transactions on pattern analysis and machine intelligence, 43(5): 1662-1577 2019.
12

---

[37] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the CVPR, pages 6700-6709, 2019.
[38] Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun. Fuhai Chen, Gen Luo. Yongjian Wu, Yue Gao, and Rongrong Ji. Improving image captioning by leveraging intra-and inter-layer global representation in transformer network. In Proceedings of the AAAI, pages 1655-1663, 2021.
[39] Jiayi Ji, Yiwei Ma, Xiaoshuai Sun, Yiyi Zhou, Yongjian Wu, and Rongrong Ji. Knowing what to learn: a metric-oriented focal mechanism for image captioning. IEEE Transactions Image Processing, 31: 4321-4335, 2022.
[40] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects photographs natural scenes, In Proceedings of the EMNLP, pages 787-798, 2014.
[41] Douwe Kiela. Hamed Firooz. Aravind Mohan, Vedanuj Goswami. Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. 33: 261 1-2624, 2020.
[42] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2.344.026444 2023.
[43] Jing Yu Koh, Danicl Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. In Proceedings of the NeurlPS, 2023.
[44] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson. Kenji Hata. Joshua Kravitz, Stephanie Chen. Yannis Kalantidis, Li-Jia Li, David A. Shamma. Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-74 73, 2017.
[45] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[46] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308. 00692, 2023.
[47] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, and Yueting Zhuang, Fine-tuning multimodal Ilms to follow zero-shot demonstrative instructions. In Proceedings of the ICLR, 2023.
[48] Juncheng Li, Siliang Tang, Linchao Zhu, Wenqiao Zhang, Yi Yang. Tat-Seng Chua, and Fei Wu. Vari- ational cross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023.
[49] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2:
bootstrapping language -mmaee pre- pre-training with frozen image encoders and large language models. In Proceedings of the ICML, pages 19730-19742, 2023.
[50] Kunchang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. CoRR, abs/2305.0635.. 2023.
[51] Mingxing Li. Li Hu. Zhiwei Xiong, Bang Zhang, Pan Pan, and Dong Liu. Recurrent dynamic embedding for video object segmentation. In Proceedings of CVPR, pages 13411 2022.
[52] Xiangtai Li. Wenwei Zhang, Jiangmiao Pang. Kai Chen, Guangliang Cheng, Yunhai Tong, and Chen Change Loy. Video k-net: A simple, strong, and unified baseline video segmentation. In Proceedings of the CVPR, 2022.
[53] Xiangtai Li, Haobo Yuan, Wenwei Zhang, Guangliang Cheng, Jiangmiao Pang. and Chen Change Loy. Tube-link: A flexible cross tube bascline for universal video segmentation. In Proceedings of the ICCV, 2023.
[54] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmian, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-base visual segmentation: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[55] Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. Omg- seg: Is one model good enough for all segmentation? In Proceedings of the CVPR, 2024.
13

---

[56] Yicong Li. Xiang Wang. Junbin Xiao. Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In Proceedings the CVPR. pages 2918-2927, 2022.
[57] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the CVPR, pages 2251 1-22111, 2023.
[58] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jicbo Luo. Tgif: A new dataset and benchmark animated gif description. In Proceedings of the CVPR, pages 4641-46500 2016.
[59] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Yuan. Vidco-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
[60] Tsung-Y Yi Lin, Michael Maire. Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitmick. Microsoft COCO: common objects in context. In David J. Flect, Tomás Pajdla, Bernt Schicle, and Tinne Tuytelaars, editors, Proceedings of the ECCV, pages 740-755, 2014.
[61] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the CVPR, pages 23592 23601, 2023.
[62] Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635-651 6515, 2023.
[63] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. CoRR, abs/2304.08485, 2023.
[64] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang. Hang Su, Jun Zhu, al. Llava-plus: Learning to use tonls for creating multimodal agents. arXiv preprint arXiv:231 1. 100437, 2023.
[65] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chumyuan Li, Jianwei Yang. Hang Su, Jun Zhu et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.054499 2023.
[66] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2 13214, 2021.
[67] Muhammad Maaz, Hanoona Abdul Rasheed, Salman H. Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed vidco understanding via large vision and language models. CoRR, abs/2306.05 05444,, 2023.
[68] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension unambiguous object descriptions. In Proceedings of the CVPR, pages 11 11-20, 2016.
[69] Anton Milan, Laura Leal-Taixé, Ian Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark for multi-object tracking, arXiv preprint arXiv:1603 100.001,,,.,,31, 2016.
[70] Victor Siemen Janusz. Milewski, Maric-Francine Moens, and Iacer Calixto. Are scene graphs good enough improve image captioning? In Proceedings the AACL, pages 504-515, 2020.
[71] Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or Or. Null-text inversion for editing real images using guided diffusion models. Proceedings of the CVPR, pages 6038-6047, 2023.
[72] Meredith Ringel Morris, Jascha Sohl- dickkiics,,,n, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farahet, and Shane I Legg. Levels of agi: Operationalizing progress on the path to agi. arXiv preprint 1r 1.02462, 2023.
[73] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for eoxttto-mmae diffusion models, arXiv preprint arXiv:2302.0845 08453, 2023.
[74] Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-Ilave Pixel grounding large lide-languagee models. arXiv preprint arXiv:231..13335 2023.
14

---

[75] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. Proceedings of the ICML, pages 16784-16804, 2022.
[76] OpenAL. Introducing chatgpt. 2022.
[77] OpenAL. Gpt-4 technical report. 2022.
[78] Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma. and Furu Wei. Kosmos-
2: Grounding multimodal large language models to the world. arXiv preprint (iXiv22306.18244 2023.
[79] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to -hrasse correspondences for richer -ongeeto-sentence models. In Proceedings of the ICCV, pages 1641-2649 2015.
[80] Jordi Pont- Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbelácz, Alex Borkine-Horunng and Luc Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint arXiv: 170v:0704.00675. 2017.
[81] Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning. Proceedings of the ICML, 2024.
[82] Leigang Qu, Shenggiong Wu, Hao Fei, Liqiang Nie, and Tat-Seng Chua. Layoutlim-12i Eliciting layout guidance from LLM for text-to-image generation. In Proceedings of the ACM MM, pages 643-654, 2023.
[83] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πly Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the ICML, pages 8748-8763, 2021.
[84] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Hiug- Yang. and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint arXiv:2311.03333 2023.
[85] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. arXiv preprint arXiv:2312.02228, 2023.
[86] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. resouresotttion image synthesis with latent diffusion models. In Proceedings of the CVPR, pages
2022.
[87] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsey, and Aran Komatsuzaki. Laion-400m: Open datasct of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02//1 2021.
[88] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: benchmark for visual question answering using world knowledge. In Proceedings of the ECCV, pages 146-162, 2022.
[89] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed image alt-text dataset for automatic image captioning. In Proceedings of the ACL, pages 2556-2565, 2018.
[90] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li. Weiming Lu, and Yueting Zhuang. Huggingept: Solving Al tasks with chatgpt and its friends in huggingface. CoRR, Abs/2303.11788 2023.
[91] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprint arXiv:2311.10088 2023.
[92] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make generation without tixt-video data. CoRR, (asx/209.14499) 2022.
Make-a-video: a-video: Text-to-video
[93] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from vidcos in the wild. arXiv preprint arXiv: 1212.0022 2012.
[94] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu. Yan Wang, and Deng Cai. Pandagpt: One model to instruction-folle follow them all. CoRR, 10.22335.163555... 2023.
15

---

[95] Peize Sun. Jinkun Cao. Yi Jiang, Zehuan Yuan, Song Bai. Kris Kitani, and Ping Luo. Dancetrack: Multi-object tracking uniform appearance and diverse motion. In Proceedings the CVPR, pages 20993-21002, 2022.
[96] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang. Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.000 05222, 2023.
[97] Zineng Tang. Ziyi Yang. Chenguang Zhu, Michael Zeng, and Mohit Bansal, Any-to-any generation via composable diffusion. CoRR, abs/2305.
1846, 18846, 2023.
[98] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang. Qian Yu, and Dong Xu. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):22888844,, 2021.
[99] Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar. Aurélien Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
[100] Florian Tramèr, A Alexey Kurakin, Nicolas Paperot,, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint arXiv: arXiv: 1705007244, 2017.
[101] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the CVPR, pages 1921-1930, 2023.
[102] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Chuanxin Tang, Xiyang Dai. Yucheng Zhao, Yujia Xie, Lu Yuan. and Yu-Gang Jiang. Look before you match: Instance understanding malters in video object segmentation. In Proceedings of the CVPR, pages 2268 2268-2278, 2023.
[103] Peng Wang. An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li. Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence- to-sequence learning framework. In Proceedings of the ICMI., pages 23318-23340. PMIR, 2022.
[104] Xinyu Wang, Bohan Zhuang, and Qi Wu. Modaverse: Efficiently transforming modalities with llms. arXiv preprint arXiv:2401 00395, 2024.
[105] Zhanyu Wang, Longyue Wang. Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. Gpt4video: A unified multimodal large language model for Instlettion-olllowedd understanding and safety-aware generation. arXiv preprint urXiv:2311.16511 2023.
[106] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang. Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, Abs/2303.04671, 2023.
[107] Dongming Wu, Weneheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang. and Jianbing Shen. Referring multi-object tracking. Proceedings of the CVPR, pages 14633 14642, 2023.
[108] Jay Zhangjic Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie. and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffuusin models for text-to-video generation. CoRR, abs/2212.1 22..112211111 1565, 2022.
[109] Jialian Wu, Jianfeng Wang. Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-tex. transformer for object understanding. arXiv preprint arXiv:2212.00288 2022.
[110] Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang. Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, et al. Towards language-drive video inpainting via multimodal large language models. arXiv preprint ar 240i 10226, 2024.
[111] Mingrui Wu, Xinyue Cai, Jiayi Ji, Jiale Li, Oucheng Huang, Gen Luo, Hao Fei, Xiaoshuai Sun, and Rongrong Ji. ControlmlIm: Training-free visual prompt learning for multimodal large language models. arXiv preprint 24X2:22407.21534 2024.
[112] Shengqiong Wu, Hao Fei, Hanwang Zhang, and Tat-Seng Chua. Imagine that! abstract-to-intrice text-to-image synthesis with scene graph hallucination diffusion. In Proceedings of the NeurIPS, pages 79240-79259, 2023.
[113] Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji. Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization in multimodal 1mm. arXiv preprint arXiv:2406.05122 051000005122220 2024.
16

---

[114] Shengqiong Wu. Hao Fei, Leigang Qu. Wei Ji, and Tat-Seng Chua. NEXT-GPT: Any-to-any multimodal Ilm. In Proceedings the ICML, 2024.
[115] Junbin Xiao, Angela Yao, Zhiyuan Liu, Yicong Li, Wei Ji, and Tat-Seng Chua. Video as conditional graph hierarchy for multi-granular question answering. In Proceedings of the AAAI, pages 2804-2812, 2022.
[116] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310. 12/90, 2023.
[117] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yucting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the ACM MM, pages 1645-1653, 2017.
[118] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. MSR-VTT VTT: A large video description dataset for bridging video and language. In Proceedings of the CVPR, pages 5288-5296, 2016.
[119] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang. and Thomas Huang. Youtube-vos vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv: arXiv:1809.00 1809.03327, 2018.
[120] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-awar vision transformer for referring image segmentation. In Proceedings of the CVPR, pages 2022.
[121] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 174.2 9(1):
1, 2023.
[122] Zongxin Yang and Yi Yang. Decoupling fcatures in hicrarchical propagation for video object segmentation. 35:36324-36336, 2022.
[123] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L. Berg. Modeling context in referring expressions. In Proceedings of the ECCV, pages 69-85, 2016.
[124] Zhou Yu. Dejing Xu, Jun Yu. Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao, Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134, 2019.
[125] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv:2312.10033 2023.
[126] Ao Zhang, Liming Zhao, Chen-Wor Xic, Yun Zheng, Wei Ji, and Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXiv:231 I. 04498, 2023.
[127] Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Vpgtrans: Transfer visual prompt generator across lims. 36, 2024.
[128] Hang Zhang, Xin Li, and Lidong Bing. Iideo-Ilam:: An instruction-tume audio-visun language model for video understanding. CoRR, abs/2306.0288 022858 2023.
[129] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. 36, 2024.
[130] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wengi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-imterest arXiv preprint arXiv:2307.0360) 2023.
[131] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjic Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint 1xXiv2231..0411 2023.
[132] Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shenggiong Wu. Shunping Ji, Chen Change Loy, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. arXiv preprint arXiv:2406. (0..229966/389, /9389,
2024.
[133] Wenqiao Zhang, Haochen Shi, Siliang Tang, Jun Xiao, Qiang Yu. and Yueting Zhuang. Consensus graph representation learning for better grounded image captioning. In Proceedings of the AAAI, pages 3394-3402 2021.
17

---

[134] Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio- temporal video grounding for multi-form sentences. In Proceedings the CVPR, pages 10668-10677, 2020.
[135] Kaizhi Zheng, Xuchai He, and Xin Eric Wang. Minigpt-S: Interleaved vision-and-land and-aauugg--e generation via generative vokens. arXiv preprint arXiv:2310.02239, 2023.
[136] Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason Corso, and Marcus Rohrbach. Grounded video description. In Proceedings of the CVPR, pages 6578 6587, 2019.
[137] Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, and Hanwang Zhang. More grounded image captioning by distilling image-text matching model. In Proceedings of CVPR. pages 4776-4785, 2020.
[138] Deyao Zhu, Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-langue laiguag understanding with advanced large language models. CoRR, abs/2304. 105520 2023.
[139] Xueyan Zou, Jianwei Yang, Zhang. Feng Linjie Li, Jianfeng Wang. Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. In Proceedings of the NeurlPS, 2024.
18

---

A Details of Backbone Visual Modules/Specialists
To address the inability of text-based LLMs in handling various vision tasks, consider integrating off-the-shelf external modules. Once the LLM generates invocation details through understanding the input and recognizing the user's intent, the corresponding modules activated produce non-textual outputs. Technically, we employ a variety of current SoTA expert models for vision processing. For image generation and editing, For image and video segmentation, we opt for SEEM [139]. For video generation, ZeroScope [8] and 12VGen-XL [131] utilized for text-to-video and image-to-video tasks, respectively. Lastly, for video editing functionality, we incorporate StableVides [9]. In Table 13, we summarize the functionality of each backend module, along with a specification of the inputs and outputs.
integrate the diffusion-basee model GLIGEN [57].
No.
Function
Model
Input
Output
Text Generation
Image Generation
GLIGEN [57]
Image
Image Segmentation
SEEM [139]
Text, Image
Image. Mask BBox
4
Image Editing
GLIGEN [57]
Text, Image [BBox Mask]
Image
5
ZeroScope [8]
Text
Video
Vidco Generation
12VGen-XL [131]
Image
Video
Video Segmentation
SEEM [139]
Text, Video [BBox Mask)
Video, Mask BBox
8
Video Editing
StableVideo [9]
Text, Video
Video
Table 13: Summary of backend modules VITRON.
B Extensions of Pixel-aware Synergistic Vision-Language Understanding Learning
This section extends more details of the $4 in the main article.
B.1 Baise MILM Skill Training
Overall Vision-Language Alignment Learning. In line with the methodologies in current MILMs, our approach involves This step aims at mapping the input vision language features to unified feature space. This space creates representations that the central LLM can interpret, thereby enabling it to process incoming vision signals effectively. We utilize datasets of 'image-caption' pairs (CC3M [89], videooccaptinn pairs (Webvid [4]), and 'region-caption pairs (ReiCOCO [40]) from existing corpora. When provided with an image, video, specific visual region, we engage the frozen LLM to generate a text description or caption that aligns with the reference caption.
Invocation-oriented Instruction Tuning. The above phase of training endows both the LLM and the frontend encoders with the ability understand vision. This step, instruction tuning for invocation, aims to equip the system with the precise capability to execute commands, allowing the LLM generate appropriate and correct invocation text. This text is then used to trigger various backend task execution modules. Different terminal vision tasks might require distinct invocation commands. To unify this, we try to standardize the LLM's response output into a structured text format, which includes: 1) User response output, which directly replies to the user's imput. 2) Module name, indicating which function or task be executed. 3) Invocation command, a meta-instruction for triggering the task module. 4) Region (optional), specifying a fine-grained vision feature needed for certain tasks, such as in video tracking vision editing, where backend modules require this information. For the region, based on LLM's pixel-level understanding, bounding box described by the coordinates (X,, Y,, X,, Y₁) will be output. Following, we exemplify video tracking example for the structured LLM response for module invocation.
Structured LLM response for module invocation (with video tracking example):
Input:
Can you help me track the clock circled in the video?
A: [LLM response]
Sure! You provide video where there a female cop talking on walkie-talkie, responding to emergency calls, and crime prevention. Also in the scene, the clock green color. Following will
19

---

outline the clock in the video.
B: [Task Instruction] <Module> Video Segmentation </Module> <Instruction> segmentation: clock </Instruction> <Region> (0.23, 0.35, 0.11, 0.26) </Region>
Function
VITRON Input
Module Input
Data Source
Amount
Image Generation
Image-Caption
CC3M [89]
4,000
RefCOCO [40]
4,000
Image Segmentation
RefCOCO [40]
5,000
Object-Name
gRefCOCO [61]
2,028
COC02017 [60]
4,000
Image Editing
Bounding- -ox
MagicBrush [129]
5,000
Tideo-Caption
WebVid [4]
7,000
Video Generation
LAION-400M (87)
4,000
B
WebVid [4], [44]
5,000
Video Segmentation
Bounding- Box
WebVid
5,000
Edditing-Querg
WebVid [4]
5,000
Video Editing
Editing-Query
WebVid [4]
5,000
Table 14: Feature summary of the constructed data for text invocation instruction tuning. in image segmentation means the reference image provided by users. in video segmentation means the intermediate referred video keyframe interpreted within the system.
To teach the LLM produce the correct invocation responses, we need create data for instruction tuning. A key is ensuring that the data covers all possible scenarios. We must take into account different ways users might interact with the system for each functionality mentioned in Table 13 (except for text generation). For example, when requesting video creation, user might describe what they want purely text, or provide reference image as the basis for the desired video. Similarly, for editing images videos, users could express their editing requests either through text, or by using sketches, scribbles and other interactions. Thus, the LLM needs to versatile accepting various types of user inputs and generating accurate invocation response that matches the requirements of the backend modules. Technically, we make use the existing annotated datasets for various vision tasks included in this work. For each task under specific different user imput scenarios, with the corresponding data, we design various template dialogue-fore format examples. Based on these examples then prompt the GPT-4 generate more samples under various topics and enriched scenarios. Finally, we collect total of 55,000+ invocation-orientee instruction tuning samples. In Table we provide summary of these datasets.
Embeddingg Decoder Alignment Tuning. Besides using the explicit textual instruction to invocate downstream modules, also the signal feature embeediing/representtioo (from LLM) should also be fed to the modules. Denote the task-specifie features as v" and iiss-invrriant fine-gruined features as n² We will concatenate them as one unified feature embedding t [o?; w²) and then send v to the downstream module.
Following [ [1],, we align the feature embedding with all the visual module's input encoders via the decoding-side projection layers. We do this feature alignment learning by minimizing the distance between the projected feature embedding and the module's 's input encoder. For example for diffusion- based image video generation, we may directly use the textual condition encoder, while keeping all the other modules fixed. Technically, to endow the model to produce other modalities beyond text, we add signal tokens to the vocabulary the LLM. In the alignment training phase, we mainly take the captions from CC3M, WebVid, and AudioCaps as inputs and concatenate them with the special signal tokens as outputs. The loss function comprises three key components: 1) the negative ligleiilloood of producing signal tokens, and 2) the caption alignment loss: la-distance between the
20

---

hidden states of signal tokens produced by the LLM and the conditional text representations derived from the text encoder within diffusion models, and 3) conditional latent denoising loss [86].
B.2 Fine-grained Spatiotemporal Vision Grounding Instruction Tuning
We propose a fine-graines spatiotemporal vision grounding instruction tuning for VITRON. The core idea to enable the LLM to ground the fine-grained spatiality of images and the detailed temporality of videos. Technically, we leverage LoRA [35] to enable small subset of parameters within the L.M to be updated during the tuning.
Image Spatial Grounding. Considering that the LLM alone can only output text, design it to respond with the corresponding bounding box areas. We focus on two types of tasks: grounded image captioning and referring image segmentation. For grounded image captioning. we input an image and identify a specific object within it, prompting the LLM to describe the identified object. Conversely, for referring image segmentation (where we consider outputting bounding box, akin phrase grounding), the task involves inputting a complete image along with related phrase or sentence description, and the LLM is expected to output the object's spatial bounding box, represented by coordinate numbers (X,. Yt, X,, Yt). The X and Y coordinates are normalized real values within the range [0. 1]. where <X,> and <Y,> indicate the top-left corner, and <X,.> and <Ya> represent the aattom-right corner of the bounding box. We consider datasets such as Flickr30K Entities [79], RefCOCO [40], RefCOCO+ [123] and RefCOCOg [68]. Utilizing GPT-4, we preprocess and expand the original data into multi-turn QA format, thereby generating the most suitable instruction-tuning data.
Grounded Image Captioning:
Referring Image Segmentation:
Input:
Input:
Please generate detailed caption for the given image, and clearly link each part the caption to specific ob- jects or areas in the image which you can denote with a bounding box with "object: (X,, Y,, X,, Y,)" format. Output:
Please identify the target object from the given images based on the follow- ing text query: "A man black shirt holding a water bottle"..
Please output the bounding box (X,, Y,, X., Yo) of the target object.
A girl wearing a pink dress sitting long bench in the park reading a book.
Output: (X,, Y, X., Y6)
girl: (Xi, Y,, Xx, Ys).
pink dress: (X,, Ye, Xr, Yoh.
Video Spatial-Tempont Grounding. For videos, the LLM must not only identify spatial regions but also ground these within the temporal context of the video, essentially achieving video tracking. Similarly, we explore tasks such grounded video captioning and referring video tracking. While the approach is akin to that used for images, it requires not just the output of object bounding box coordinates in each frame but also the specification of the duration of the frames (F,, F,) in which these coordinates are to be outputted. Here <F,> denotes the starting frame number, and <F, means the ending frame number. We mainly construct the required instruction tuning data from several key video tracking datasets: LaSOT [22]. GOTIOK [36], Dance Track [95], and MOT16 [69].
Grounded Video Captioning:
Referring Video Tracking:
Input:
Input:
Please generate a caption for the given video, and link each part of the caption to specific objects in the video with its temporal presence duration. You should denote it with a bounding box with the start- ing and ending frame number in format "object: (X1, Y., X,, Y& F., F.)".
Please track the specified object throughout the video based on the following given de- scription, and mark its starting and ending position in each frame: "A man is skiing on a thick-snow-coveres Please output in the format of "(Xi, Y,, X.. Y, I F., Fe)".
slope."
Output:
A dog is ranning after a ball on the grass. dog: (X1, Y,, Xx, Yo F., Fe), ball: (Xt. Yo. X... Yo, 1 F.. F.).
Output:
(Xc. Y., X.. Y, F.. F.)
21

---

Grounding-aware Vision QA. The grounding tasks mentioned above only touch upon the low- level aspects of vision perception. However, in many scenarios, it's essential for the LLM to possess high-level, in-depth vision reasoning capabilities, building upon the foundational low-level pixel grounding. Therefore, we further introduce grounding-awan undertake semantic-level QA tasks based on the grounded results. Specifically, we consider both image-grounded QA and sadoo-grounded QA. The data sources for these tasks remain the same as mentioned above for the grounding tasks. And we again rely on GPT-4 to use its imaginative capabilities to design reasonable questions and answers based on the given grounding content the above grounding/segmenttiion datasets, formatting these into instruction-tune -uning format.
vision QA, enabling the L.LM to
Grounded Image QA:
Grounded Video QA:
Input:
Input:
Based on the given image, please se- lect the correct answer among all the candidates: Q: Where is the child sit- ting? A: arms, 2) ground, 3) chair, 4) bed.
Based on the provided vidco, answer the following ques- tion by choosing the most appropriate answer from the options given. Q: If the rope in the video breaks, what would happen to the person in the video? A: 1) rumning, 2) jumping up. 3) flying away, 4) fracture.
Please first identify and ground the tar- get object (in coordinates) mentioned in the question, and then proceed to answer the question.
First, analyze the spatial position and temporality of the target object mentioned in the question within the video, and based on this analysis, determine the answer to the question.
Output:
Output:
The target object mentioned in the question is "child," with the position given by (X,, Y, Xy. Vo). From this, it can be determined that the child is being held by woman, which means the child is sitting in the woman's 's arms. Therefore, the answer 1) arms.
The objects involved in the question are the rope (X,, Yo, X.. Y, F.. F.) and the person (Xr. Y,, X.. Y, F... FoJ. with their relationship being that the person is climbing with the assistance of the rope. Based on common sense, if the rope were to break, the person would fall to the ground from height. The most likely outcome is that the person would break bone.
B.3 Cross-task Synergy Learning
Let's denote the task-specific features as w² and task-invariant fine- fine-grained features as va, We will concatenate them one unified feature embedding [uP; vz] and then send v to the downstream module Mj for the k-th task prediction:
yk M(D).
(1)
Although we separate the two feature spaces into the shared one d and private one vP, there are still chances that the learned shared and the private features are closely entangled, weakening the refining of the shared task-invariant fine-grained feature. Therefore, we employ a third-party task discriminator with adversarial training refine the features. The discriminator D is classifier for predicting what the current task is, based merely on the task-invariant fine- mne-grained feature representation v². Ideally, once the discriminator cannot accurately identify the task ID L the task-invariant fine- fine-grained feature representation v" can be understood as the most purified one. Specifically, the discriminator a 4-layer 768-d Transformer (Trm) network, where we use a feedforward layer (FFN) with Softmax for the task prediction:
Trm (1, ". ,,,),,
(2)
Vk Softmax(FFN(())
where of is the predicted task ID.
The target of this adversarial training is to urge the shared features such that the discriminator cannot reliably predict the task ID:
sayn
Σ oh log(v()))
(4)
htttp:://opena..co///ntd
22

---

B.4 Overall Training Remarks
All our framework is trained through three main stages, in specific ordering of sub-steps:
Step-1:
Basic Multimodal Comprehension and Generation Skill Training, of 64.1.
Step-1.1: Aligning the encoder-LLM for overall language alignment learning.
Step-1.2: Doing text invocation instruction tuning such that the MLLM learns to output text instructions in the correct format.
Step-1.3: When the above step is converging, training the LLM with continuous soft muibedding-orientee LLM-decoder alignment, such that the LLM overall can convey the signal to the downstream modules.
Step-2:
Fine-grained Spatiotemporal Vision Grounding Instruction Tuning, cf 84.2.
Step-2.1: Starting by doing the Image Spatial Grounding training, on the grounded image captioning task and referring image segmentation task.
Step-2.2: When MLLM has the ability for fine-grained spatial understanding, doing the Video TemppallTempora Grounding training, on the grounded video captioning task and referring video tracking task.
Step-2.3: When the MLLM has learned to have the competent ability of both image and video spatiotemporal understanding at the perception level, doing the Grounding-awan Vision QA task at the cognition level.
Step-3: As the final step, when the overall system has learned have competitive ability in
various visual tasks, dining the cross-task synergy learning, ef $4.3. This should be done by combining both the adversarial training (C*v") with the end-task prediction (Lx). So the total loss of the step-3 is: L + ΣE Lik
C Extended Experimental Settings
We quantify the performance of VITRON a variety of standard benchmarks for downstream vision tasks and compare it against some of the currently strong-performin systems. Given the countless vision tasks within the community, our experiments focus only on 1-2 of the most representative tasks from each task category for validation. To ensure a fair comparison, all subsequent experiments adopt settings same similar to those of baseline systems, with evaluations following established practices. Before experiments, we perform targeted pre-training on all of VITRON's 's backend modules (such as GLIGEN and SEEM) on their respective datasets. This ensures system is optimized for the best possible performance during testing. Our approach centers on training the linear projection layers of all encoders and efficiently fine-tuning the language model using LoRA.
Our backbone LLM Vicuna?, 7B, version 1.5. The CLT encoders for both images and videos are with patch size of 14, and convert all images and video frames into 336px resolutions. The task discriminator in our synergy module with Transformer architecture, with 4 layers and each in 768-d representation. To train our model, we employ the Adam W optimizer along with a learning rate scheduler. The pre-training of VITRON unfolds in three phases, all conducted on 10~16 X A100 (80G) GPUs. Initially, we train the model using global batch size of 128 and maximum learning rate 3e-4, a process that takes approximately 40 hours. In the second tuning phase, we adjust the model with maximum learning rate of 1e-5, utilizing global batch size of 90. This stage of training lasts about 35 hours. The third phase of training employs a global batch size of 128 and maintains the maximum learning rate of le-5, completing in roughly 10 hours.
D More Experiment Results
D.1 Vision Segmentation
Video Segmentation. Table 15 presents the comprehensive comparison of VITRON with some
SoTA systems in video tracking on DAVIS 17 [80] Test-Dev and Youtube-VOS 2019 [119] Val sets.
httrps:/tugggingaacocoo/// http:://uuggggggooc.oco//my/y//a 7b-v1..
23

---

DAVIS 17 [80] Test-Dev
Youtube- VOS 2019 [11]] Val
Method
J&F
FF
J+
F.
J.
Fi
RDE [51]
77.4
73.6
81.2
76.2
84.8
XMem [14]
77.4
84.5
80.3
88.6
DeAOT [122]
80.7
76.9
84.5
84.6
89.4
80.8
88.9
ISVOS [102]
82.8
79.3
86.2
85.2
89.7
80.7
88.9
VITRON
84.2
81.5
86.7
86.5
90.4
81.9
90.2
Table 15: Results of video object segmentation.
D.2 Fine-grained Vision Understanding
Region-level Image Understanding. The comparisons of image-referring expression comprehen-
sion on three datasets are shown in Tables 16.
RefCOCO [40]
RefCOCO+ [123]
RefCOCOg [68]
Method
Val
TestA
TestB
Val
TestA
TestB
Val
Test
OFA [103]
80.0
83.7
76.4
68.3
76.0
61.8
67.6
67.6
Shikra [12]
87.0
90.6
80.2
81.6
87.4
72.1
82.3
82.2
MiniGPT-vv [11]
88.7
91.6
85.3
79.9
85.1
74.4
84.4
84.6
VITRON
90.9
93.2
89.3
83.7
89.1
76.9
86.4
87.0
Table 16: Results (accuracy) of image referring expression comprehension.
Table 17 displays the results across 6 datasets for image-based VQA.
Method
Grounding OKVQA [88]
GQA [37]
VSR [62]
Icon [66]
VizWiz [32]
HM [41]
Flamingo [1]
44.7
31.8
28.8
57.0
BL.P-2 [49]
45.9
41.0
50.9
40.6
19.6
53.7
InstructBLIP [17]
49.5
33.4
57.5
MiniGPT-4 [138]
37.5
30.8
41.6
37.6
LLaVA [63]
54.4
41.3
51.2
43.0
Shikra [12]
47.2
MiniGPT-v2 [11]
57.8
60.1
62.9
51.5
53.6
58.8
VITRON
59,4
62.1
63.9
52.2
54,7
60.2
Table 17: Results (accuracy) on image-based VQA.
Region-level Video Understanding. Table 18 presents the results of video QA across 4 represen- tative datasets. Interestingly, while PG-Video-LLaVA has video grounding capabilities, it does not show better results than Video-L.LaVA which lacks grounding. However, our VITRON achieves
superior performance.
MSVD-QA [117] IS-VV [118]
TGIF-QA [58]
Activity [124]
Method
Grounding
Accuracy
Score
Accuracy
Score
Accuracy
Score
Accuracy
Score
VideoChat [50]
56.3
2.8
45.0
34.4
2.2
LLaMA-Adapler [30]
43.8
2.7
34,2
2.7
Video-LLaMA [128]
51.6
2.5
29.6
1.8
12,4
Video-ChatGPT [67]
64.9
3.3
49.3
2.8
51.4
35,2
2.7
Video-LLaVA [59]
70.7
3.9
59.2
3.5
70.0
4.0
45.3
3.3
PG- PG-Vidco-LLaaA [74]
64.1
3.7
51.6
3.3
66.8
3.8
39,9
3.3
VITRON
74,9
4.0
62.0
3.8
72.4
4.1
51.0
3,7
Table 18: Results (accuracy and confidence Score) on video QA.
E
Qualitative Studies of Case Visualizations
E.1
Vision Segmentation
Fig. 7 further demonstrates an example of how our VITRON processes image segmentation tasks in interactive manner with the user. When sketch doodle outlines specific areas of an image, VITRON is capable of accurately identifying the corresponding objects within the image. Following this, precisely generates the bounding box and mask for the identified objects.
Fig. also demonstrates the process of video segmentation. VITRON successfully identifies, localizes, and tracks the target in the video based the provided queries. Our system demonstrates

---

remarkable ability to accurately and flexibly capture the user's intent, even when the desired object to be segmented is described in a highly implicit manner.
Could
you
help
you outline the grey hen the video?
segment this Item?
0:00 039
you're aiming
cut out grey pot.
Just checked out the video, it's Airight, proceed
forest scene for
I'm actually curious about the middle the twa pots, could you assist?
0:00 0:10/018
Absolutely! would be dog, Let's outline
can you focus
middle timeframe the video.
smart highlight the item pointed here, will you?
Then
0:00 0:29
there actually three penguins. I'll stay the midale one then.
Sure thing! Let's focus on the pot frant dog.
0:00 0
Figure 8: Example of video object segmentation.
Figure 7: Demonstration of image segmentation.
E.2
Fine-grained Vision Understanding
Fig. highlights VITRON's fine-grained visual understanding capabilities through regional image captioning, to accurately understand and appropriately describe image regions.
Further, we showcase visualization process illustrating how VITRON accomplishes fine-grained visual understanding through video grounding tasks in Fig. 10. VITRON's precise perception of video details ensures a more accurate understanding content semantics, leading to stronger video understanding capabilities.
25

---

Could you tell me what's happening in this photo?
what's
happening with the girl in the video?
Looks like the girl getting mini galf lesson thera. she's gripping golf stick, there's bend down, to teach the girl how to swing.
The picture features group of people riding small electric propuision boat on body There person sitting in the front, another in the middle, and the third person in the back. The boat appears to be motorboat with electric motor providing more emdronmentalll friendly made of transportation. The people seem to enjoying their time on the wates possibly engaging in recreational activitles or simply taking leisurely
visible in the
And how'd she marage hit the ball the floor?
She hits the ball the floor with a golf club, which piece equipment sport golf.
Figure 10: Demonstration of fine-grainer visual understanding via video grounding.
So what relationship persons according to the picture?
would like to see video where run past car on the street in the snow.
Absolutely. got just the video for you.
12
image, chair umpire tennis court. The relationship between the two persons is that the chair umpire responsible for overseeing the match and making decisions
holding
Himm, It's cool but not quite what in mind. Let paint picture your how about starting with adding white dog next to the car?
game, as calling balls and strikes, while the holding the tennis racket tennia player participating in the match.
Got it, I've placed a white deg right next the car, right predise.
why black siting such high chair?
person sitting high known rhair umpire's vantage point to observe the tannis match and make accurate judgments. The chair's olovated position umpire have including the lines and the players' mavements, This helps the umpire to make fair and impartial decisions during the match, such as calling balls strike, ansuring that the game played according rules.
Awesome, now could you convert that Image Into a dynamic video dog seen running past car?
have better
the
Sure thing, here's your video showcasing the dog making move past the car.
Figure 9: Demonstration of fine- fine-graine grained visual understanding via image grounding.
Figure 11: Demonstration of vision generation across different modalities.
26

---

typically E.3 Vision Generation
Fig. 11 illustrates the process of vision generation across different modalities, including text, image, and video. Initially, users start with a basic text command, and VITRON is capable of transforming simple idea into detailed video. However, if users are not satisfied with the video generated directly from text, they can first generate an image from text, then fine-tune or edit this image, and finally create a satisfying video based on the adjusted image. Our VITRON, thanks to robust interactive capability via multi-turn dialogue, enables users to perform a series of consecutive operations, ultimately facilitating smooth content creation. This fully helps meet the demands real-world application scenarios.
E.4
Vision Editing
Could
the background for snowy Scene. like bear strolling through winter forest.
Could you madify the chair marked to give it four legs?
0.00
Sure thing! The chair you pointed out sitting on a metal I've swapped that out for four iegs instear.
Great, no problem. I've made the adjustments to the background in the video, replacing directly with forest covered white snow. Here's the video with replacement.
And can clear off all the stuff on the table?
0.00
flower wine bottle? Gone. The table's all clear now.
wonderfu idea! Replacing with polar bear would indeed complement the snowy background better.
Nice. about adding white lounging an the floor?
frigid Arctic region, where the climate remains cold year-round, So they are olten found snowy landscapes or icy surfaces. Placing a polar bear snowy setting aligns
their habitat.
went ahead and added two cats lying
Check the updated image.
0:00
Figure 12: Demonstration of image editing.
Figure 13: Demonstration of video editing.
We showcase the specific process of this image editing, illustrated in Fig. 12. VITRON is capable of accepting different forms of user inputs (textual instruction or sketch) for precise image edits. It maintains contextual consistency throughout series of sequential editing operations, ultimately achieving satisfactory results that meet the user's expectations.
Fig. 13 illustrates this process. VITRON competently handles video editing tasks, including modifica- tions to the content's subject, and changes to the video's style, etc.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: in the Section &
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: in the Experiment part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
28

---

Justification:
in the Section 3 & 4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: in the Appendix A and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is a new model (e.g., large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: All the data utilized our experiments are publicly available. We will release the code upon the acceptance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/guides/ http://oooeSuuuueonnnnen
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httpps.ccppubiic//eec S ssi wwwinnsssonniiiiiiooeooooioiss sesionnllcy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: in the Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: in the Section and Appendix D, and All results are reported after the statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: the Section C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https /ttee///neeuieeeeeeeeeee cc/public/Bthicccuieeee dineee?
Answer: [Yes]
Justification: the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: [NA]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: the paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "G672wBkR9u", "reviewer": "Reviewer_FR5p", "summary": "This paper proposes a universal vision LLM, named Vitron, for comprehensive understanding, generating, segmenting, and editing of both images and videos. The model incorporates SOTA visual specialists as the backend to support various visual tasks. A cross-task synergy module is advised to learn to maximize the task-invariant visual features. Experiments demonstrate the effectiveness of Vitron on 12 visual tasks.", "questions": "Refer to weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ The paper is well-written and easy to follow.\n+ The proposed model integrates multiple visual tasks, including understanding, generation, segmentation, and editing.\n+ Experiments show that the performance of Vitron is comparable to SOTA methods of each visual task.", "weaknesses": "Some SOTA methods are not included in the experiments. For instance, CM3leon [1] 7B achieves 4.88 FID on COCO-Captions. MAGVIT-v2 [2] reaches 53 FVD on UCF.\n\n[1] Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning, 2023.\n\n[2] Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your time in writing comments, especially for your strong recognition of our paper. Below we provide our response.\n\n----\n\n**Q1. Some SOTA methods are not included in the experiments. For instance, CM3leon [1] 7B achieves 4.88 FID on COCO-Captions. MAGVIT-v2 [2] reaches 53 FVD on UCF.**\n\n**A**: Thank you for pointing this out.\n\nIn fact, CM3leon utilizes retrieval-augmented pretraining, which retrieves additional information to help achieve a 4.88 FID on COCO-Captions. For a fair comparison, we should consider the performance of CM3leon without retrieval, which is 10.82 FID on COCO-Captions, and this does not outperform our Vitron's performance (7.57 FID).\n\nAs for MAGVIT-v2, we noted that the settings in their paper are different from ours, even though both use the UCF-101 dataset for video generation. MAGVIT-v2 is focused on class-conditional video generation, i.e., label-to-video generation, whereas Vitron is focused on image(frame)-to-video generation, mainly following the baseline DynamiCrafter's setting. Therefore, these scores are not directly comparable. Since MAGVIT-v2 is not yet open-sourced, we are unable to run their model on image(frame)-to-video generation for direct comparison.\n\nHowever, we will for sure include these two works in the revision and will further cover the latest released relevant works that we failed to include when submitting to NeurIPS.\n\nAdditionally, compared to coarse-grained vision generation or understanding, our Vitron system has an advantage in fine-grained vision tasks (such as editing, grounding). Moreover, there is actually ample room for further performance improvement in various tasks, as Vitron has not yet been fully fine-tuned on the relevant datasets.\n\nLastly, we want to further emphasize the most core and valuable aspect of this paper: Vitron is not just about combining or invoking tools, but proposes a new idea: it must be capable of showing synergy. This is the core of Vitron that differentiates it from other similar Agent-based generalist models, which simply use LLMs to invoke external modules—this is meaningless, as the resultant generalist cannot surpass meta-specialists. To achieve a truly more powerful generalist, it is essential to realize cross-modal/cross-task synergy. This is akin to how ChatGPT achieves cross-task synergy in NLP tasks. Synergy is equally key to realizing the native multimodal emergence capabilities of MLLMs.\n\nSo, if you are affirming the value of our work, we hope you can consider improving the score slightly. Thank you very much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "91uWqilsOY", "reviewer": "Reviewer_BLKJ", "summary": "This paper proposed VITRON, a multimodal generalist that supports a wide range of vision tasks, treating images and videos as a unified entity.\nIt combines discrete textual instructions with continuous signal embeddings for effective function invocation.\nThe model is trained on fine-grained spatiotemporal vision-language alignment to enhance its pixel-level visual capabilities.\nA synergy module is developed to maximize the sharing of fine-grained visual features across different visual tasks, improving overall performance.\nOverall, VITRON aims to be a comprehensive system for vision-related tasks, demonstrating strong capabilities across various benchmarks.", "questions": "Please refer to **Weaknesses**.", "limitations": "Please refer to **Weaknesses**.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "VITRON introduces a novel hybrid method for message passing, combining discrete textual instructions with continuous signal embeddings.\nIt proposes pixel-level spatiotemporal vision-language alignment learning, advancing fine-grained visual capabilities.\nThe paper demonstrates extensive capabilities across 12 visual tasks evaluated on 22 datasets, showcasing high performance.\nIt includes a cross-task synergy module that optimizes task-invariant fine-grained visual features, enhancing task cooperation.\nVITRON represents a significant step towards a unified AI capable of understanding, generating, segmenting, and editing both images and videos.", "weaknesses": "The proposed system is complicated but the paper is not well-structured. A clearer exposition of the system’s components and their interactions would greatly benefit the reader’s comprehension.\n\n**Literature Review**: The paper would benefit from a more comprehensive literature review, particularly discussing related works such as Visual ChatGPT [1], HuggingGPT [2], InternGPT [3], ControlLLM [4], GPT4Tools [5], LLaVA-Plus [6], etc. A comparative analysis with these methods in the experimental section would provide valuable context and benchmarking.\n\n**Technical Clarifications**: There are several areas in the paper where further clarification is needed.\n* The necessity of both Module name and Invocation command (Line 874) could be better justified. Using Invocation command is enough?\n* Why does Video Segmentation module need Region results? In my opinion, the region should be predicted by the backend models?  The role of the Region results in the Video Segmentation module warrants further explanation, as it seems the backend model should predict these. \n* The alignment of Task-specific and Task-invariant Fine-grained Features with the inputs of Backend Visual Specialists is unclear, especially since these specialists are frozen during training. Specifically, how to connect Task-specific Feature and Task-invariant Fine-grained Feature to the backend models? Or the Backend Visual Specialists can work without any input limitation?\n\n**Data Format Consistency**: The data format inconsistency between Line 880-881 and page 23 should be addressed to avoid confusion about task requirements and tool usage. Does it mean \bthat the tasks in page 23 do not require using tools?\n\n**Impact of Features**: More insight into how the Task-specific and Task-invariant Fine-grained Features affect the overall system performance would be beneficial.\n\n**Baseline Comparisons**: Including training-free baselines that utilize GPT-4v with in-context learning and ReACT [7] for tool invocation would provide essential benchmarks for comparison. \n\n**Model Results**: As this paper claims that “VITRON surpasses existing SoTA specialists’ performance”, It is recommended to present the results of specialist models in the experimental tables as reference performances. Furthermore, As VITRON is positioned as a generalist, comparisons with other generalist methods would be fair enough.\n\n**Tool-use Accuracy**: An exploration of tool-use accuracy across the 12 tasks covered by VITRON is necessary to validate the method’s efficacy.\n\n**Execution of Backend Modules**: Clarification is needed on what constitutes the successful execution of backend modules in Figure 4.  In addition, why text instruction is more conducive to the successful execution of backend modules while soft feature embedding seems to be more useful in terms of specific task performances?\n\nGenerally, the paper’s contributions are incremental. To my knowledge, the single tool invocation is not complicated, especially with the limited scale of toolset. There are several papers can solve tasks by chaining multiple tools.\n\n[1] Wu C, Yin S, Qi W, et al. Visual chatgpt: Talking, drawing and editing with visual foundation models[J]. arXiv preprint arXiv:2303.04671, 2023.\n\n[2] Shen Y, Song K, Tan X, et al. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[3] Liu Z, He Y, Wang W, et al. Interngpt: Solving vision-centric tasks by interacting with chatgpt beyond language[J]. arXiv preprint arXiv:2305.05662, 2023.\n\n[4] Liu Z, Lai Z, Gao Z, et al. Controlllm: Augment language models with tools by searching on graphs[J]. arXiv preprint arXiv:2310.17796, 2023.\n\n[5] Yang R, Song L, Li Y, et al. Gpt4tools: Teaching large language model to use tools via self-instruction[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[6] Liu S, Cheng H, Liu H, et al. Llava-plus: Learning to use tools for creating multimodal agents[J]. arXiv preprint arXiv:2311.05437, 2023.\n\n[7] Yao S, Zhao J, Yu D, et al. React: Synergizing reasoning and acting in language models[J]. arXiv preprint arXiv:2210.03629, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We feel honored for your many constructive comments, and appreciate you went through our paper so carefully. Below we try to address your concerns or misunderstandings. If you find our response effective, please consider generously increasing rating.\n\n----\n\n**Q1: The system is complicated, but paper is not well-structured**\n\n**A**: Sec. 3 mainly introduces the framework components, and the three subsections of Sec. 4 further explain three important tuning mechanisms. We will further polish the paper's structure.\n\n---\n\n**Q2: needs a more comprehensive literature review, such as Visual ChatGPT, HuggingGPT, InternGPT, ControlLLM, GPT4Tools, LLaVA-Plus, etc**\n\n**A**: Actually, HuggingGPT and LLaVA-Plus are already included. Other works you listed, as well as the latest works will all be covered in the revision.\n\n---\n\n\n**Q3: several clarification is needed**  \n**Q3.1: The necessity of both Module name and Invocation command (Line 874)? Using Invocation command is enough?**\n\n**A**: It is necessary to consider both the Module Name and Invocation Command. The module name determines which module to invoke, while the command serves as input to that module. Another benefit is that it allows the LLM to explicitly recognize which task module should accurately process the user's input demand.\n\n---\n\n**Q3.2: Why does the Video Segmentation module need Region results?**\n\n**A**: Users may specify a particular area of interest to segment/track. For example, if a user specifies an area on a sketch pad, this area can be directly passed to the backend module. If the user specifies a region of interest using natural language, the LLM itself needs to determine and output this region result for the backend module.\n\n---\n\n**Q3.3: how to connect Task-specific and Task-invariant Fine-grained Features to the backend models?**\n\n**A**: Task-specific Feature and Task-invariant Fine-grained Feature embeddings will be concatenated as one, which is then fed to the feature encoder of the backend module. For example, for the video generation module of diffusion-based ZeroScope, the concatenated feature representation is fed directly to the UNet. But before connecting this feature vector to the backend module, we do alignment between them for sure.\n\n---\n\n**Q4: The data format inconsistency between Line 880-881 and page 23**\n\n**A**: Correct. The prompt in Line 880-881 is used to tune the invocation of tools, while the prompts on page 23 do not involve tool invocation. Therefore, these prompts are not necessarily in the same format.\n\n---\n\n**Q5: how the Task-specific and Task-invariant Fine-grained Features affect the overall performance**\n\n**A**: In Sec G.3, the Cross-task Synergy Study, we provided a general overview of how different tasks utilize the Task-invariant Fine-grained Features and their impacts. We will consider conducting a more detailed study of the separate impacts of these features in revision. \n\n---\n\n**Q6: should include training-free baselines that utilize GPT-4v with in-context learning and ReACT [7] for tool invocation.**\n\n**A**: Thanks. We implemented this experiment. Since GPT-4v doesn’t support video, and also time constraints in rebuttal period, we only implemented one image processing task, __referring image segmentation__ on RefCOCOg.\n\n|                     | Val  | Test |\n|-----------------------|------|------|\n| NExT-Chat             | 67.0 | 67.0 |\n| SEEM         | 65.7 | 65.8 |\n| GPT-4v + SEEM         | 64.9 | 65.1 |\n| GPT-4v + ReACT + SEEM | 65.4 | 65.4 |\n| Vitron                | 67.9 | 68.9 |\n\nCompared to two pipeline systems, due to our Vitron being trained fully jointly, it performs better than the pipeline—even though the backbone GPT-4v is stronger. As seen, the Pipeline systems did not surpass the backend specialist (SEEM).\n\n---\n\n**Q7: the paper claims “VITRON surpasses existing SoTA specialists”, should present the results of specialists**\n\n**A**: Actually our results (Table 1-11) cover some important SoTA specialists, such as G-DINO for video grounding, and MakeVideo for video generation, where Vitron achieved better performance. But thanks; we will consider covering more specialists and generalists in revision.\n\n---\n\n**Q8: how is tool-use accuracy?**\n\n**A**: Actually this is already shown in Fig 4, specifically in the right-hand of the bar chart (i.e., Execution Rate).\n\n---\n\n**Q9.1: what constitutes the successful backend execution in Fig 4?**\n\n**A**: For successful execution, it is crucial to accurately determine the module name and provide accurate input information, which includes both commands and some necessary features.\n\n---\n\n**Q9.2: why is text instruction more conducive to the successful execution; soft embedding more useful in task performances?**\n\n**A**: The reason is straightforward: Text instruction can only help ensure that the backbone module can select the downstream tasks correctly, but some critical information might not be conveyed through pure text alone. Soft feature embedding, on the other hand, provides very detailed additional information, giving the backbone module a more complete set of input features, which helps to achieve better results for the output tasks.\n\n\n---\n\n**Q10: contributions are incremental. the single tool invocation is not complicated**\n\n**A**: We believe the most core and valuable aspect of our paper is that Vitron is not simply about combining or invoking tools, but it proposes a new idea—necessitating synergy. This is what distinguishes Vitron from other agent-based generalist works (including Visual ChatGPT, HuggingGPT, and LLaVA-Plus).\n\nThese existing systems operate exactly similarly to an agent, which can be quite meaningless, as the resultant generalist cannot surpass meta specialists. To achieve a truly more powerful generalist, it is key to realize cross-modal/cross-task synergy. This is akin to how ChatGPT achieves cross-task synergy in NLP tasks. Synergy is pivotal to unlocking the native multimodal emergent capabilities of MLLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kr8kUKLU1o", "reviewer": "Reviewer_d2kF", "summary": "This paper introduces a universal pixel-level vision LLM designed for comprehensive understanding, generating, segmenting, and editing both static images and dynamic videos. Technically, the authors propose using the LLM as the core brain, incorporating different encoders for images, videos, and pixel-level regions to extend the comprehensive capabilities of LLMs, and various decoders for images, videos, and segmentations to extend the generative capabilities of LLMs. Additionally, a well-rounded hybrid message-passer, incorporating discrete textual instructions and continuous signal embeddings, is designed to pass messages from the LLM to backend decoders. A novel cross-task synergy module is introduced to enhance the synergy between different tasks. To demonstrate the efficacy of the proposed method, the authors conduct experiments on 12 visual tasks and evaluate across 22 datasets. Extensive experiments show that the proposed method can handle multiple vision-language tasks, spanning from visual comprehension to visual generation, and from low-level pixel understanding to high-level semantic understanding.", "questions": "1. I have a question that I would like to discuss with the authors: What are the current limitations of the unified model? Given that the performance of specialists is still not very satisfactory, what is the significance of a unified model? In the future, if the form of specialist tasks changes (e.g., the emergence of vision LMs rather than LLMs being at the core), what modifications would the current unified model need to undergo?\n2. If there are better specialists available, how can they be replaced, and what would be the cost of doing so?\n3. Are other modalities, such as audio or 3D, being considered for inclusion with fine-grained capabilities?", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The motivation of this work is clear and reasonable. Building a universal generalist is a trending topic, achieving an \"one for all\" capability. Images and videos are two core vision interfaces through which humans understand the world; thus, powerful vision-language generalists should be capable of comprehending and generating images and videos.\n2. This work presents a well-designed unified framework, integrating the SoTA specialists and enabling performance across various vision-language task groups from vision comprehension to generation, and from low-level pixel understanding to high-level semantic comprehension.\n3. This work proposes a hybrid instruction passing method by combining discrete textual instructions and continuous signal feature embeddings to ensure effective and precise information passing to the modules.\n4. The authors design a fine-grained spatial-temporal vision grounding instruction tuning method, enabling sufficient pixel-level visual perception. Furthermore, they apply adversarial training to enhance synergy between different tasks, achieving overall performance improvement.\n5. Extensive experiments are solid, and conducted on various datasets and tasks, including vision segmentation, fine-grained vision understanding, and vision generation. The experimental results show that after integrating cross-task synergy learning, the framework consistently improves performance across the majority of tasks.\n6. this work builds a text invocation instruction tuning dataset. If it is publicly available, it will be beneficial to the development of related research.\n\nOverall, I like this paper. I think the proposed idea is novel and interesting, and also intuitive. I tend to believe the contribution, novelty and quality of the paper to be highly aimed.", "weaknesses": "1. The authors may need to clarify the advantages of their work over existing generalists like HuggingGPT and Visual ChatGPT, which also aim to be generalist models to some extent.\n2. The authors propose a hybrid message-passing mechanism and provide a corresponding ablation study. But it would be better to have an intuitive demonstration of the information complementation between discrete textual instructions and continuous feature embeddings. For example, which types of information are passed by textual instructions and which are conveyed by the embeddings.\n3. GPT-4 is employed to construct the text invocation-oriented instruction tuning datasets. The detailed prompt templates and some examples should be provided.\n4. Some symbols need clarification, such as the J and F in Table 3.\n5. A value bar should be added to intuitively show the synergy across different tasks. Additionally, the calculation of the synergy between tasks should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work and providing insightful comments that significantly enhance the quality of our paper. Here are our responses to your concerns and questions, and we hope to gain your further support.\n\n---\n\n**Q1: The authors may need to clarify the advantages of their work over existing generalists like HuggingGPT and Visual ChatGPT, which also aim to be generalist models to some extent.**\n\n**A**: Vitron is not just a combination or tool invocation; it introduces a new concept: the necessity of synergy. This is what differentiates Vitron from other agent-based generalist works, including Visual ChatGPT, HuggingGPT, and LLaVA-Plus. These methods operate like agents, merely invoking external modules through LLM, which is ultimately ineffective because the resultant generalist cannot surpass meta-specialists. To achieve a truly stronger generalist, cross-modal/cross-task synergy is essential, similar to how ChatGPT achieves cross-task synergy in NLP tasks. Synergy is crucial for unlocking the native multimodal emergent capabilities of MLLM.\n\n-----\n\n\n**Q2: The authors propose a hybrid message-passing mechanism and provide a corresponding ablation study. It would be beneficial to have an intuitive demonstration of how discrete textual instructions complement continuous feature embeddings.**\n\n**A**: This concept is straightforward. For simple user queries, textual instructions can successfully invoke the backend module to produce the correct result. However, for more complex user intents or queries, additional supplementary information and features help the backend module understand how to execute the task accurately. For instance, in an image caption task, textual instructions suffice by simply instructing the backend module to \"describe the given image\". But for a text-to-video generation task, instructions like \"generate a video based on the text 'a dog walking in the park'\" may need further visual information in the form of feature embeddings, such as details describing the dog or the park.\n\n-----\n\n\n**Q3: GPT-4 is employed to construct text invocation-oriented instruction tuning datasets. Detailed prompt templates and examples should be provided.**\n\n**A**: Thank you for pointing this out. We have exemplified a prompt for a video tracking task in Appendix E.1. We will provide complete prompt templates for all tasks in the revision, and our data will be fully open-sourced.\n\n-----\n\n\n**Q4: Some symbols need clarification, such as the J and F in Table 3.**\n\n**A**: Apologies for not explaining these symbols. For video object segmentation on DAVIS 17, we use the mean Jaccard $J$ index and mean boundary $F$ score, along with mean $J$ & $F$, to evaluate segmentation accuracy.\n\n-----\n\n\n**Q5: A value bar should be added to intuitively show the synergy across different tasks. Additionally, the calculation of the synergy between tasks should be clarified.**\n\n**A**: We conducted this analysis in Appendix G.3 Cross-task Synergy Study, and Figure 6 provides a visualization of the result. We calculate cross-task synergy by recording Vitron's performance on individual tasks without Cross-task Synergy Learning and after incorporating it, then comparing the performance improvements for the same tasks. Figure 6 shows the normalized improvement visualization.\n\n-----\n\n\n**Q6: What are the current limitations of the unified model? What is the significance of a unified model if the performance of specialists is still not very satisfactory?**\n\n**A**: As emphasized before, achieving true generalist capabilities towards human-level AI necessitates cross-modal and cross-task synergy. We view synergy as a form of generalization ability. Only when MLLM achieves this across different modalities and tasks can we speak of advancing to the next level of MLLM towards human-level AI, similar to ChatGPT's synergy in NLP tasks. However, we have yet to see such generalization capabilities in MLLM (or unified multimodal models). The Cross-task Synergy Learning mechanism proposed in Vitron is a key starting point towards this goal.\n\n-----\n\n\n**Q7: If better specialists are available, how can they be replaced, and what would be the cost?**\n\n**A**: Vitron supports replacing backend specialist modules. However, the entire system might need corresponding retraining and alignment learning. Fortunately, Vitron currently uses the state-of-the-art specialists.\n\n-----\n\n**Q8: Are other modalities, such as audio or 3D, being considered for inclusion with fine-grained capabilities?**\n\n**A**: While unifying various modalities, including sound, images, and 3D information, is an important trend (as demonstrated by NExT-GPT), the main focus of this work remains on the vision modality. Nonetheless, achieving more modality unification and synergy is also a significant goal of this work. We hope the proposed Cross-task Synergy Learning mechanism will receive more attention and study in subsequent research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7OXz2BpyDl", "reviewer": "Reviewer_vBVg", "summary": "This paper proposes a unified vision LLM for various downstream visual tasks, including understanding, generating, segmenting, and editing. The proposed framework addresses the problems with respect to images, videos, text, and human interactions. The authors also propose a hybrid method to integrate discrete textual instructions and continuous signal embeddings.", "questions": "1. To what extent, can the \"unification\" be used for the multimodal tasks?\n2. The authors proposed the statements about instructions and cooperation in Line 57-59. How did they reflect these aspects in the designed modal and experiments?\n3. How did the model distinct the features from task-specific and task-invariant parts? In other word, did the authors do something to assign the aims during the model training?", "limitations": "As the most parts are clear for me, I am looking forward to the answers to the above questions. I suggest to add more details of the proposed module and the training scheme in the appendix.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "From low-level to high-level semantics, the proposed method address the vision problems in a unified framework. In addition, the generation and editing tasks are also included in this framework. The proposed instruction-passing mechanism over discreet texts and continuous embeddings help to address the different downstream tasks. The experiments are comprehensive.", "weaknesses": "There is no obvious technical weaknesses from my side. However, I do have some questions about this paper.\n1. The authors put \"pixel-level\" in a highly key position of this paper. From my side, as the proposed model supports different visual tasks, among them, many downstream tasks are not pixel-level such grounding. Why did the authors highlight the concept?\n2. The implemental details are missing, especially the synergy module. How can the readers know the re-training and designing process in detail?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We take your opinion “**There is no obvious technical weaknesses from my side**” as the greatest acknowledgment of the value of our work. Thank you very much! Your affirmation provides us with great motivation to further improve our research.\n\n-----\n\n**Q1: The authors put \"pixel-level\" in a highly key position in this paper. From my side, as the proposed model supports different visual tasks, among them, many downstream tasks are not pixel-level such as grounding**\n\n**A**: From a broader perspective, the research on MLLM is very vibrant and rapid. And future research on MLLM will inevitably develop towards a more unified approach, covering more modalities in breadth and achieving more powerful task processing capabilities in depth. This work focuses more on the latter. For visual MLLM, the initial visual MLLMs could only support coarse-grained, instance-level understanding or generation, but a key problem here is object hallucination due to the lack of pixel-grounding capabilities. To solve these issues, the community has gradually developed more advanced visual MLLMs that can support fine-grained, pixel-level visual understanding and generation, including visual grounding, segmentation, editing, etc. It's reasonable because to achieve stronger task performance, it is necessary to have stronger visual processing capabilities. In fact, if a model has fine-grained visual processing capabilities, it will definitely have stronger capabilities in coarse-grained tasks. Therefore, we emphasize achieving pixel-level visual capability in this paper.\n\n-----\n\n**Q2: The implemental details are missing, especially the synergy module**\n\n**A**: Apologies for the lack of necessary technical descriptions due to the 8-page limit. Please refer to Appendix E.3, 'Cross-task Synergy Learning', where we provide an extended introduction about this module. Also, in Appendix F, 'Extended Details of Experimental Settings', we extended the introduction to the overall framework's implementation. Moreover, our provided code includes detailed implementation information. Our system, including the code and all data, will be open-sourced. We will continue to improve the open-source code and provide a detailed manual.\n\n-----\n\n**Q3: To what extent can the \"unification\" be used for multimodal tasks?**\n\n**A**: Unification refers to using a single model or system to execute a variety of tasks, just as using one ChatGPT can perform all NLP tasks. For vision multimodal tasks, we have categorized all visual tasks into four orthogonal grand categories. This way, we use a separate backend module to support each category. Specifically, for categories like visual comprehension, our system will cover almost all image/video-to-text scenario tasks.\n\nGoing beyond the unification of pixel-level visual MLLM, in this paper, we also emphasize achieving synergy across multiple modalities and tasks to realize a true generalist towards human-level AI. We can understand synergy as a type of generalization ability, only when MLLM achieves generalization across different modalities and tasks can we say we have reached the next level of MLLM towards human-level AI, similar to how ChatGPT achieves cross-task synergy. However, we have not yet seen such generalization capabilities in MLLM (or multimodal unified models). The Cross-task Synergy Learning mechanism proposed in Vitron is a key starting point towards that goal.\n\n-----\n\n**Q4: authors proposed statements about instructions and cooperation. How did they reflect these aspects in the designed model and experiments?**\n\n**A**: We propose a hybrid mechanism of message passing, where we combine both discrete textual instructions and continuous signal embedding methods. The former aids in accurately invoking different backbone modules (thanks to the LLM’s proficiency in task dispatching), while the latter supplements with richer modality-preserved visual features that cannot be directly described through discrete text. As depicted in Fig. 2, the LLM outputs 1) text responses for users, 2) text instructions for module invocation, and 3) feature embeddings of special tokens. Both text instructions and feature embeddings are passed to backbone modules. In Figure 4, we explored these two different message-passing mechanisms to determine whether discrete textual instruction is more beneficial or whether continuous signal embedding is better for building a multi-modal generalist. Also, there we validated the pros and cons of the proposed hybrid method of message passing.\n\n-----\n\n**Q5: How did the model distinguish the features from task-specific and task-invariant parts?**\n\n**A**: Technically, we employ adversarial training to decouple task-specific from task-invariant features. We first let different backbone visual specialists make task predictions based on these two features (via concatenation). Meanwhile, we encourage a third-party discriminator (acting as a classifier) to determine which is the current task based solely on the shared feature representation. Ideally, once the discriminator can no longer accurately identify the task, the shared feature can be considered the most purified and broadly applicable across tasks. We further detailed formal task modeling in Appendix E.3, 'Cross-task Synergy Learning'.\n\nDuring the Embedding-oriented Decoder Alignment Tuning stage, we align the feature embedding with all the visual module’s input encoders via the decoding-side projection layers. We do this feature alignment learning by minimizing the distance between the projected feature embedding and the module’s input encoder. For example, for diffusion-based image or video generation, we may directly use the textual condition encoder, while keeping all other modules fixed.\n\n-----\n\n**Q6: I suggest adding more details of the proposed module and the training scheme in the appendix.**\n\n**A**: Thank you for your suggestion, we will adopt it and reflect it in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response to All Reviewers

---

Dear Reviewers,

We sincerely appreciate the detailed and constructive comments you have provided on our work. We are fully committed to integrating your suggestions into our revision process. We feel very encouraged that the reviewers find our work novel, interesting, and intuitive. Your support is deeply appreciated.

Here we would like to re-emphasize the significant and distinctive contributions of this work. We believe three keywords can well represent this project: **Unification**, **Fine-grained Vision**, **Synergy**.

1. **Unification.** The research on MLLMs is currently very vibrant and rapid. Therefore, future research on MLLMs will inevitably develop towards a more unified direction, i.e., to cover more modalities in breadth and achieving more powerful task processing capabilities in depth. This work focuses more on the latter and specifically on vision MLLM. For the first time, this paper proposes a grand unified vision MLLM, Vitron, handling understanding, generating, segmenting, and editing of both images and videos in a "one-for-all" manner.

2. **Fine-grained Vision.** In the visual MLLM community, initially the visual MLLMs could only support coarse-grained, instance-level understanding or generation. However, a key problem here is object hallucination due to the lack of pixel-grounding capabilities. Subsequently, more advanced visual MLLMs that support fine-grained, pixel-level visual understanding and generation (visual grounding, segmentation, editing, and more) are emerging. We devises a hybrid instruction-passing mechanism and various pixel-level vision-language spatiotemporal alignment learning, enabling the proposed Vitron model to support comprehensive and powerful fine-grained visual processing capabilities.

3. **Synergy.** Vitron achieves visual generalist, but not simply through tool invocation (like an agent); rather, it emphasizes achieving synergy across multiple modalities and tasks. We can understand synergy as a type of generalization ability, where only when an MLLM achieves generalization across different modalities and tasks can we say we have reached the next level of MLLM towards human-level AI. This is similar to how ChatGPT achieves cross-task synergy in NLP tasks. However, we have not yet seen such generalization capabilities in existing MLLMs. The Cross-task Synergy Learning mechanism proposed in Vitron is a key starting point toward that goal. This is also the most core and valuable aspect of this paper. We hope this work will inspire more related research to further explore this mechanism.

We will fix all the possible issues and improve the manuscript. To address all your concerns and questions, we have prepared a comprehensive response, including additional experiments where necessary. If you have any further questions or feedback, please don't hesitate to interact with us. We are more than willing to provide clarifications and address any additional concerns you may have.

Best regards

Meta-review: ### Overview

The submission introduces a unified framework to achieve a "one-for-all" multimodal model capable of understanding, generating, segmenting, and editing both images and videos at various levels of granularity, from image to pixel level. During the discussion, the authors emphasized three core aspects of their work: Unification, Fine-grained Vision, and Synergy. According to the reviews, the paper is well-supported by comprehensive experiments demonstrating the efficacy of the proposed method.

According to the discussions between authors and reviewers, there are several strenghts that are commonly shared by reviewers:

Novel concept: The idea of unifying a range of visual tasks under a single MLLM framework is innovative and timely, given the rapid advancements in multimodal models. The introduction of a cross-task synergy learning mechanism to enhance generalization across tasks is particularly noteworthy.

Comprehensive framework: The proposed framework integrates multiple backend visual specialists and combines discrete textual instructions with continuous signal embeddings for effective task execution. This hybrid instruction-passing mechanism is highlighted as a key technical contribution.

Extensive experiments: The authors conducted thorough experiments across a wide range of tasks, demonstrating that Vitron achieves comparable or superior performance to state-of-the-art (SOTA) methods. The inclusion of adversarial training to decouple task-specific and task-invariant features further strengthens the framework's robustness.

However, there are also a few weaknesses that make the reviewers not reach a great consensus on acceptance:

Paper presentation: Several reviewers noted that the paper's structure could be improved for better clarity, particularly regarding the detailed explanation of the system's components and the alignment strategies for embedding-oriented decoder alignment tuning. The authors’ rebuttal did clarify these aspects, but the initial presentation was lacking in some areas.

Comparison with previous works: There were concerns about the novelty of the work, particularly in comparison to existing generalist models like HuggingGPT, Visual ChatGPT, LlaVA-Plus, etc. While the authors argue that their focus on synergy sets their work apart, some reviewers felt this distinction was not sufficiently emphasized or validated in the original submission.

Novelty: A recurring theme in the reviews was the question of novelty. Despite the authors' detailed rebuttal, some reviewers still felt that the contributions were incremental, particularly when compared to related works. This led to a slight decrease in ratings by some reviewers.

### Final Recommendations

Overall, the main strengths of the paper lie in its novel approach to unifying visual tasks and the extensive experimental validation. However, the concerns regarding clarity, presentation, and the novelty of the work remain. Given the innovative nature of the work and its potential contribution to the field, it is recommended for acceptance, provided that the authors significantly improve the presentation, especially by clarifying the unique aspects of their contributions in comparison to existing models. The authors should include more detailed discussions and illustrations regarding the significance of the work compared with previous related works in the final version to address the raised concerns fully.
Decision: Accept (poster)