VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Authors: jiannan  wu
muyan  zhong
sen  xing
zeqiang  lai
zhaoyang  liu
zhe  chen
wenhai  wang
xizhou  zhu
lewei  lu
tong  lu
ping  luo
yu  qiao
jifeng  dai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 12:12:24

Abstract: We present VisionLLM v2, an end-to-end generalist multimodal large model (MLLM) that unifies visual perception, understanding, and generation within a single framework. Unlike traditional MLLMs limited to text output, VisionLLM v2 significantly broadens its application scope. It excels not only in conventional visual question answering (VQA) but also in open-ended, cross-domain vision tasks such as object localization, pose estimation, and image generation and editing. To this end, we propose a new information transmission mechanism termed ``super link'', as a medium to connect MLLM with task-specific decoders. It not only allows flexible transmission of task information and gradient feedback between the MLLM and multiple downstream decoders but also effectively resolves training conflicts in multi-tasking scenarios. In addition, to support the diverse range of tasks, we carefully collected and combed training data from hundreds of public vision and vision-language tasks. In this way, our model can be joint-trained end-to-end on hundreds of vision language tasks and generalize to these tasks using a set of shared parameters through different user prompts, achieving performance comparable to task-specific models. We believe VisionLLM v2 will offer a new perspective on the generalization of MLLMs.

Main paper: 
 introduction: 
 Multimodal large language models (MLLMs) [8, 97, 223, 107, 105, 140, 14, 169, 34, 33] have recently made significant progress, demonstrating outstanding performance across various visionlanguage tasks, even in scenarios requiring complex understanding and reasoning. However, a notable limitation is that current MLLM outputs are in text form, which significantly constrains their capacity to represent structured or visual information. Some researchers [140, 182, 181, 180] have expanded the text-based output formats of MLLMs to better align with downstream tasks. While these efforts have shown promise, they have not fully addressed practical needs such as dense object detection, pose estimation, and image generation.To overcome this limitation, a line of research [116, 187, 166, 111, 117, 47] enhances the capabilities of MLLMs by transmitting task information to tools via text messages, as illustrated in Figure 1(a) . Despite these advances, these text-based methods are restricted by the information that text can convey. They are not end-to-end, and the feedback gradient from the tools cannot be relayed back The embedding-based method displays a connection using learnable embeddings, which facilitates efficient information transfer but lacks support for multitasking. (c) Our method employs a "super link" technique, where a unified MLLM interfaces with multiple task decoders through super links, supporting over 100 diverse tasks.to the MLLM. This limitation has spurred another research direction [89, 148, 193, 44, 83, 164] that employs learnable embeddings as intermediaries to connect MLLM with one specific task decoder (see Figure 1(b) ). However, the naive embedding connection is difficult to scale to multi-task scenarios. A routing mechanism is needed to ensure the correct selection of tools, and the issue of task conflicts [224] arising from joint multi-task training is also a problem that needs to be considered. Therefore, developing an end-to-end MLLM generalist for various vision and vision-language tasks beyond text output remains a significant challenge.Given these challenges, developing an end-to-end generalist MLLM requires a more effective information transmission method than conventional text messages and naive embeddings. This method should ensure that task information and feedback gradients are accurately and flexibly communicated between the central MLLM and multi-task decoders while preventing task conflicts across various visual domains and input/output formats. In addition, multi-task datasets for generalist MLLMs need to be well-prepared. Despite the abundance of annotations in the community, the diverse and inconsistent formats of these annotations across different tasks make it challenging to develop a unified dataset that effectively supports multi-task learning.In this work, we introduce VisionLLM v2, an end-to-end generalist MLLM designed for a wide array of vision and vision-language tasks. This model not only performs typical visual question answering but also extends to image generation, image editing, and open-ended object detection/instance segmentation/pose estimation across diverse image domains. To facilitate information transmission between the MLLM and multiple downstream task decoders, we introduce the super link technique, which consists of two components: (1) Routing Token: special tokens (e.g., [DET], [POSE] , and [GEN]) added to the MLLM's vocabulary. Whenever the MLLM predicts a specific routing token, it triggers the selection of the appropriate decoder. (2) Super-Link Queries randomly initialized learnable weights bound to the routing tokens. These queries are appended after the routing tokens and processed by the MLLM to extract task-specific information, which is then sent to the target decoder. This method enables flexible task information transmission, allows decoder gradients to backpropagate to the MLLM, and avoids task conflicts by ensuring the queries are bound to routing tokens and not shared across tasks.Furthermore, we carefully collected and curated training data from hundreds of public vision and vision-language tasks to support various tasks. The data includes high-quality examples of visual question answering, visual perception, recognition, and understanding tasks from various sources such as natural scenes, remote sensing images, medical images, and industrial images. To ensure effective training with these extensive datasets, we also implemented a multi-stage joint training strategy, integrating new abilities and reaching a performance comparable to the expert models while maintaining the MLLM's foundational VQA capabilities.These designs endow VisionLLM v2 with three distinct characteristics: (1) Generality. With one suit of parameters, our model can be generalized to different tasks using different text and visual prompts.To our knowledge, it is the first end-to-end model to support hundreds of vision-language tasks while achieving performance comparable to expert models. (2) Openness. By employing open-ended decoders, our model allows users to freely define tasks through multimodal prompts, breaking away from the constraints of closed-set models limited to predefined tasks or categories. Furthermore, users can flexibly combine various tasks into more complex ones through multi-round dialogue.(3) Multimodal In-Context Ability. With multimodal inputs and outputs, our model demonstrates extensive versatility and exhibits superiority over the previous in-context models with single-modal outputs [184, 8] . These features distinguish our model from previous approaches, and establish a leading foundational MLLM for various vision and vision-language applications.In summary, our main contributions are listed as follows:(1) We propose VisionLLM v2, the first end-to-end generalist MLLM model to accomplish hundreds of vision and vision-language tasks1 , covering visual perception, understanding, and generation. It not only addresses the limitation of LLMs being confined to text outputs but also supports using textual, visual, and in-context instructions to flexibly combine tasks for real-world applications. 
vision generalist model: 
 Unified Vision Model. The unified model approach integrates multiple visual tasks into a single framework, enhancing efficiency and reducing the complexity of deploying separate models for each task. Works such as Pix2Seq-D [29] , SEEM [230] , and Semantic-SAM [93] focus on unifying the segmentation interface, achieving promising results. Grounding-DINO [112] and VisionLLM [182] explore open-set detection grounded by language, while UniPose [198] excels in pose estimation. Additionally, pioneering works [227, 224, 95, 121, 229, 189] aim to design a unified model capable of solving multiple tasks, including detection, segmentation, captioning, etc. Their results demonstrate the feasibility of a single model performing diverse tasks.Visual Prompting. Visual prompting has emerged as a novel paradigm by providing visual marks in the input instruction. It requires the model to pay attention to the specific region on the image when answering the question. Techniques like red circle [157] , SoM [196] , AutoVP [173] , ILM-VP [24] , and PIVOT [131] significantly reduce the need for textual prompt engineering, assisting models in focusing on relevant visual content. Similar to in-context learning in LLMs, Painter [183] , DINO v2 [91] , and SegGPT [184] leverage visual context to improve learning efficiency and adaptability, enabling models to adapt to new tasks with minimal input.Diffusion Model as Interface. Diffusion models are a flexible interface between users and visual tasks, facilitating a more intuitive interaction paradigm. InstructCV [51] and InstructDiffusion [56] exemplify using of natural language instructions to guide visual generation and manipulation. Pix2Seq v2 [30] showcases the potential of diffusion models in generating sequences of visual tokens, bridging the gap between vision and language.Different from these works, our VisionLLM v2 integrating LLMs extends vision generalist to support a broader range of vision-language tasks and explore various visual prompting paradigms, thereby significantly broadening the scope of application.3 VisionLLM v2 
model design: 
 The overall architecture of VisionLLM v2 is depicted in Figure 2 . It mainly consists of four parts: (1) an image encoder and a region encoder that encode the image-level and region-level information; (2) a large language model (LLM) that models the multimodal inputs and generates satisfactory textual responses;(3) a series of task-specific decoders for performing downstream tasks; (4) a super link that uses routing tokens and super-link queries for efficient and conflict-free information transmission. We detail each component in the following.Tokenization. VisionLLM v2 is flexible for handling multimodal input. ( 1) For text prompts, we employ the text tokenizer to tokenize them into distinct vocabulary indices, which can be further processed by LLM and result in the text features F text ∈ R L×C , where L denotes the length of input text, and C is the channel dimension of LLM.(2) For an image input, we utilize a pre-trained vision foundation model, such as CLIP [144] , to extract image features. Recognizing that current vision models operate the images at a low resolution, we adopt the dynamic resolution approach [33] to process the input images. Specifically, the input image is first automatically matched to an optimal aspect ratio from a predefined ratio set. Subsequently, the image is scaled up to a higher resolution based on the selected aspect ratio and divided into P square patches, each whose resolutions are 336×336. These local patches, along with a 336×336 global image I global , are processed by the image encoder to capture both holistic scenes and fine-grained details, resulting in image features F img ∈ R 576(P +1)×C .(3) For a visual prompt, we employ binary masks to flexibly represent the visual prompts, such as point, box, scribble, and mask. To extract the region embedding, we first concatenate the binary mask with the input image along the channel dimension and then process it with three convolutional layers to downsample by a factor of 14 (see appendix for more details). We further augment this feature map by adding the feature map of the global image I global . Finally, grid sampling is used to extract features within the masked regions, and these features are averaged to form the features of the visual promptF vprt ∈ R 1×C .Large Language Model. Following previous works [107, 213, 60] , both the images and visual prompts are projected to the feature space of the LLM. The LLM plays a central role in our model Besides outputting the plain text, LLM can also output the special routing token such as [DET] when needed. The super-link queries would be automatically appended after the routing token embeddings and further processed by LLM. They play as the bridge for connecting LLM and task-specific decoders. In this way, our generalist model can support hundreds of visual tasks. The detailed architecture about connecting the LLM with task-specific decoders can be found in Figure A13 .and is used to model multimodal inputs, parse user instructions, and generate appropriate responses. In this work, we adopt the commonly used Vicuna-7B [219] as the LLM in our network.Task-specific Decoders. To enhance the capacities of MLLM, we equip our model with several task-specific decoders. Specifically, we use Grounding DINO [112] for object-level localization. We additionally add a mask decoder upon it to obtain the segmentation ability. For pose estimation, we adopt UniPose [198] as the keypoint decoder. Moreover, we incorporate Stable Diffusion [152] and InstructPix2Pix [20] as the image decoders, endowing our model with the capability to generate and edit images. We discard these decoders' text encoders and link them with MLLM via the super link technique, which will be detailed explained in Section 3.2. In this way, the decoders can be trained end-to-end with the entire network, ensuring the effective transmission of task information and increasing the openness of these decoders.For the text-only output tasks, such as image-level and region-level VQA, we directly take the plain text generated by LLM as the final output. For visual perception and visual generation tasks, we propose the super link technique to tackle the challenge of selecting the appropriate decoder, avoiding task conflicts, and facilitating effective information transmission between the LLM and the decoders. The super link comprises two parts:(1) Routing Token. We add the routing tokens, e.g., [DET], [POSE], [SEG], [GEN], [EDIT], as special tokens to the original LLM vocabulary. When the model intends to complete the downstream task using one of the decoders, LLM would include the corresponding routing token in its textual response. To enable the model to discern which tasks to perform and which routing tokens to output, we construct a series of instruction templates for different tasks using ChatGPT [4] .(2) Super-Link Queries. For each decoder, we define the super-link queries as a fixed set of embeddings denoted as Q link ∈ R N ×C , where N is the number of queries. They are randomly initialized and serve as the bridge between LLM and task-specific decoders. Whenever the LLM predicts the routing token, the super-link queries would be automatically appended after the input embeddings of the routing token. We then extract their corresponding last-layer hidden states H link and apply an MLP projection to obtain Ĥlink . Finally, Ĥlink is sent into the specific decoders as a condition to perform the downstream tasks. In the following, we illustrate how to integrate Ĥlink into decoders for visual perception and generation, respectively.Visual Perception covers a wide range of visual tasks, such as open-ended/closed-set object detection, instance segmentation, pose estimation, etc. VisionLLM v2 supports using both text and visual prompts to define these tasks. We list an example in the following figure. <image> and <region> are the placeholders that will be replaced by image and region embeddings before being fed into the LLM. Here, we take Example 1 of interactive segmentation for clarification. The user prompts the model to segment specific regions within a question. MLLM sequentially lists the region names followed by a routing token [SEG] in the response. Remember that the proposed method would automatically append the super-link queries after the routing token. In that way, we can obtain the per-region representations by extracting the output hidden states of MLLM from corresponding superlink queries and pooling them into one embedding. These embeddings are fed into a segmentation decoder as the conditional feature, requiring only a single forward to produce segmentation results for all regions. In the following, we show a template example for interactive segmentation.Example1: Text Prompt + Visual Prompt for Interactive Segmentation. USER: <image> Could you please segment all the corresponding objects according to the visual prompts as region1 <region>, region2 <region>? ASSISTANT: Sure, these objects are region1 [SEG], region2 [SEG] .Visual Generation is also a wide topic covering a number of different tasks, such as generation, editing, variation, personalization, etc. In VisionLLM v2, we focus on two fundamental tasks, i.e., text-to-image generation and instruction-based image editing. We use Stable Diffusion v1.5 (SD) as our tool in the text-to-image generation task. We abandon its text encoder and use the output hidden states of the MLLM as the image generation condition for SD. Image editing task [82] can also be accomplished in the same paradigm by using both image and text prompts as inputs. In the following, we list a template example for text-to-image generation. Discussion. Some previous works have used the special token or learnable queries independently. InstructBLIP [42] , ep-ALM [158] , and MAPL [125] use learnable queries (i.e., soft prompts) to connect the modality encoders and LLM. FROMAGe [84] uses a special token for image-text retrieval so as to handle multimodal outputs, where the images are not generated from the network end-to-end. However, these works still remain constrained to text-based outputs. The proposed super link is the seamless integration of the two techniques. Despite the simplicity of our method, it is able to extend MLLMs to handle hundreds of tasks by largely extending the output formats, e.g., box, mask, keypoint and image. Meanwhile, it can address several challenges when scaling up various tasks: (i) precise decoder invocation, (ii) mitigating task conflicts and (iii) efficient message transmission in an end-to-end manner.Current MLLMs [89, 218, 44] face reduced conversational abilities when augmented with additional capacities. To create a generalist model capable of handling hundreds of tasks without compromising vision understanding, we propose a three-stage training strategy, where the first stage focuses on building an MLLM with strong image-level and region-level vision understanding. In the subsequent stages, we add task-specific decoders and continue training to equip the model with advanced capabilities.Stage-1: Mutimodal Training. In the first stage, we follow the training settings of LLaVA [107, 105] , comprising pre-training and instruction tuning phases. The pre-training phase aims to establish the image-level and region-level vision-language alignment, where only the region encoder and the projections for image embedding and region embedding are trained for efficiency. The instruction tuning phase unfreezes the LLM and trains the model on a wide range of high-quality instruction data. After the training in this stage, we can obtain a strong MLLM with excellent conversation ability, which we term as VisionLLM v2-Chat.Stage-2: Multi-capacity Fine-tuning. At this stage, we integrate task-specific decoders into the model and perform multi-task joint training. In addition to the instruction data utilized in stage-1, we incorporate extensive visual datasets such as COCO [104] , ADE20K [222] for their specific tasks. We construct a series of instruction templates for these visual datasets to perform instruction tuning, ensuring that the LLM can accurately invoke the downstream decoders. During this stage, the region encoder and all decoders undergo training, and we only finetune the input and output embeddings of the LLM to maximally preserve its original conversational ability.Stage-3: Decoder-only Fine-tuning. Since the decoders cannot converge within a single epoch, we further train the decoders for 12 epochs using visual datasets while freezing all other components. It is noted that the super-link queries continue to be trained during this stage. After finishing the three-stage training, our model has diverse capacities for visual tasks while maintaining effectiveness in global vision understanding, named VisionLLM v2. [57] , GQA test-balanced [71] , VizWiz test-dev [62] , Sci-enceQA test [154] and TextVQA val [160] . The instruction-following datasets include: POPE [101] , MME [49] , MMBench-EN/CN [114] , SEED-Bench (all/image/video) [55] . * The training annotations of the dataset are observed during training.4 Experiments 
implementation details: 
 Dataset Details. To support the joint training of our model, we meticulously collect and re-organize the datasets across a wide range of tasks from publicly available sources. For the first stage training, we utilize a substantial amount of high-quality instruction data for both image-level and region-level visual question answering, including ShareGPT4V [28] , All-Seeing [181] , VQAv2 [57] , etc. In the last two stages, we further incorporate extensive visual datasets, e.g., COCO [104] , RefCOCO/+/g [204, 126] , LAION-Aesthetics [3] , to enhance our model with numerous capacities. These datasets encompass multiple tasks such as object detection, pose estimation, image generation, and span various domains such as natural scenes, remote sensing images, medical images, etc. To facilitate the training of diverse datasets in our MLLM framework, we construct a series of instruction templates for different tasks, which are completely listed in the Appendix. Additionally, we also collect a multimodal dataset termed MMIC focusing on visual prompting and in-context learning. The data in our MMIC comes from various sources, including fine-grained visual recognition, object detection, instance segmentation, and pose detection. We elaborate on all datasets used in this work as well as the dataset construction of MMIC in the Appendix.Model Details. We adopt the CLIP-L/14 [144] as the image encoder and Vicuna-7B-v1.5 [219] as the language model. Grounding-DINO [112] and UniPose [198] are selected as object decoder and keypoint decoder, respectively. And for these two decoders, we experiment with Swin-T [115] backbone. Additionally, image decoders are kept as Stable Diffusion v1.5 [152] for image generation and InstructPix2Pix [20] for image editing. All these components load the pre-trained weights while the region encoder is randomly initialized. For visual perception and visual generation tasks, the number N of super-link queries is set to 4 and 64, respectively. During training, we adjust the dataloader so that each GPU processes samples from only one dataset. More training details are provided in the Appendix.In the following subsections, we present the experimental results to cover as many tasks, interactive modes, and domains. It is noted that all the results of our method are reported using a single generalist model with the same parameters. More results can be found in the Appendix.Multimodal Dialogue. We first evaluate our models on academic-oriented VQA datasets and recent instruction-following datasets for MLLMs, as presented in Table 1 . The results clearly demonstrate that our models outperform previous methods under the same parameter scale, particularly on the instruction-following datasets. For instance, VisionLLM v2-Chat surpasses LLaVA-NeXT-7B [106] by +9.7 and +7.0 points on MMBench-EN/CN [114] , respectively. Additionally, we find that VisionLLM v2 achieves comparable performance to VisionLLM v2-Chat on these multimodal benchmarks and even performs better on some benchmarks, such as POPE [101] , a popular benchmark for evaluating object hallucination. This phenomenon indicates that our framework effectively mitigates the issue of multi-task conflict and maintains proficiency in conversational ability. Region Recognition. The region recognition task needs the model to identify the object category given the ground-truth bounding box. We compare our method with both feature-based and textoutput approaches in Table 2a . Feature-based methods, such as RegionCLIP [221] and ASM [181] , compute similarity scores between region visual features and candidate category text features. In contrast, text-output methods [25, 60, 207] directly predict the category name using a single word or phrase, embracing the advantage of openness. As shown in the table, our models demonstrate the significant superior performance on COCO [104] , long-tail LVIS [61] and part-level PACO [147] .Visual Commonsense Reasoning. Visual commonsense reasoning (VCR) requires the model to possess strong region-level question-answering and reasoning abilities, as it needs to select not only the correct answer but also the correct rationale behind it. We present the comparison results on the VCR dataset [209] in Table 2b . Without task-specific fine-tuning, VisionLLM v2-Chat achieves an accuracy of 82.9% in the crucial Q→AR task, which precedes the previous best model, ASMv2 [180] , by +3.5 points. VisionLLM v2 also outperforms the previous methods for all the metrics, highlighting the promising common sense reasoning capability of our model.Object Detection and Instance Segmentation. In Table 3 , we compare the results of VisionLLM v2 with state-of-the-art methods on two fundamental vision tasks, i.e., object detection, and instance segmentation. As can be seen, using the lightweight backbone Swin-T, our generalist model achieves Table 5 : Ablation on the super-link queries number.We evaluate the results on the four crucial visual perception tasks: instance segmentation (COCO), visual grounding (RefCOCO), pose estimation (COCO), and interactive segmentation (COCO using scribble). Our default setting is marked in gray . the performance of 56.7 AP b and 47.8 AP m on COCO. The results significantly outperform the previous methods using ResNet50 [64] backbone and are comparable with the specialist model Grounding-DINO-T [112] . Moreover, we also validate our model on the crowded pedestrian detection dataset, i.e., CrowdHuman. VisionLLM v2 surpasses the previous best generalist model Hulk [185] by 0.9 points on AP 50 .Pose Estimation. We present the results on the multiple pose estimation dataset in Table 4 . While most previous methods [197, 194, 108] only focus on the person scenes, our VisionLLM v2 is effective in performing the keypoint detection for multiple objects. As shown in the table, our model achieves competitive performance with UniPose-T [198] using the same Swin-T backbone. Especially, our model demonstrates the superior performance on AP-10K [203] and Macaque [88] datasets and sets a new state-of-the-art result on CrowdPose [96] . These results prove the effectiveness of our model for pose estimation.We evaluate the generation capabilities of our model on two tasks, i.e., text-to-image generation and instruction-based image editing. In Figure 3 , we demonstrate that even if our model uses Stable Diffusion v1.5 as an image decoder, it achieves better visual quality than SD v1.5 with better conditional embedding produced by LLM. Moreover, the use of LLM for conditional encoding of user instructions makes it possible to benefit from the merits of LLM. For example, our model trained on English data is able to perform zero-shot bilingual image generation. Besides, we show the qualitative results of applying our model for instruction-based image editing, which also achieves appealing performance in a unified approach.In the ablation studies, we follow the training setup of stage-2 unless otherwise specified. We only train our model on the crucial visual perception tasks, i.e., instance segmentation, visual grounding, pose estimation, and interactive segmentation, for rapid validation.Super-Link Queries Number. We ablate the number N of super-link queries in Table 5 . We observe that the performance of these tasks consistently improves with an increasing number of queries. This is reasonable as more queries can lead to richer and stronger representations. Table 7 : Ablation on the one-stage and three-stage training. We evaluate the models on image VQA, instance segmentation and pose estimation Shared vs.Unshared Super-Link Queries for Different Decoders. To determine if one set of super-link queries is sufficient for all decoders, we conducted an ablation study by either using shared queries for all decoders or defining separate queries for each decoder. In this ablation, we only train the decoders and super-link queries while freezing all other components as the training setting of stage-3. In Figure 4 , we plot the performance of box AP (using the object decoder) and keypoint AP (using the keypoint decoder) on COCO. We observe that the keypoint AP would decrease over training when using shared queries, which may be attributed to the fact that most data are used for object decoder. Besides, the box AP with shared queries is also inferior to decoupled ones. Therefore, we define separate super-link queries for each decoder in our model.Multi-Task Influence. As indicated by previous works [225, 206] , different tasks with shared parameters may cause conflict with each other. This is mainly due to inconsistent optimization in multi-task learning. To investigate the mutual influence of multi-task joint training in our framework, we start from the same checkpoint and train the model on a single task (image VQA, instance segmentation, or image generation) for 1000 iterations. We record the loss change for all three tasks in Table 6 . In the table, a decrease in the loss value indicates beneficial training for the task, while an increase is detrimental. We can observe that training on image VQA is advantageous for all three tasks, which is reasonable as the conversation ability of MLLM is enhanced. Whereas training exclusively on instance segmentation or image generation leads to conflicts with other tasks. This aligns with the findings in Uniperceiver-MoE [225] .One-Stage vs.Three-Stage Training. Some previous generalist models [176, 15] train the model in one stage. Our model encompasses much more tasks and thus introduces a training conflict: the MLLM requires only 1 epoch of training on chat data to prevent overfitting, whereas the decoders need longer training epochs (e.g., Grounding-DINO need 12 epochs of training on visual data) to achieve convergence. One possible solution for one-stage training is to give a higher sample ratio for the visual data. In the following, we conduct the ablation to study the effect of one-stage v.s. three-stage training. We use image-level chat data, COCO, and COCO-Pose for image understanding, instance segmentation, and pose estimation, respectively. For one-stage training, we repeat the COCO and COCO-Pose datasets 12 times. As can be seen from Table 7 , the conversation ability of the model is significantly decreased due to extreme data imbalance. And the performance of instance segmentation and pose estimation is also slightly reduced. These results prove the effectiveness of our three-stage training. 
conclusion & limitation: 
 In this paper, we presented VisionLLM v2, a comprehensive MLLM that unifies visual perception, understanding, and generation within a single framework. The proposed super link mechanism facilitates flexible information transmission between the MLLM and task-specific decoders, addressing training conflicts and enhancing gradient feedback. Experiments show that VisionLLM v2 achieves performance comparable to specialized models while maintaining broad applicability.Regarding limitations, our model's training encompasses three stages, which are relatively complex. Moreover, the integration of downstream tools has only been preliminarily validated. Future work will further explore solutions to these issues, aiming to enhance the model's performance and efficiency.Broader Impact. We envision that this work will further promote the fusion of visual and language tasks. In addition, since our work is built on open-source pre-trained vision foundation models and large language models, requiring low training resources, thus reducing the carbon footprint. We do not foresee obvious undesirable ethical/social impacts at this moment. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The paper does not report error bars of the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See the conclusion section for details.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]  
 
 Appendix: A More ResultsRegion Captioning. To access the region understanding capabilities of VisionLLM v2, we evaluate our models on two prominent benchmarks: RefCOCOg [126] and Visual Genome [86] . The results are presented in Table A1b . Notably, VisionLLM v2-Chat significantly outperforms the state-of-theart methods, with the improvements of +8.6 and +8.4 points in CIDEr scores on RefCOCOg [126] and Visual Genome (validation subset) [86, 148] , respectively. The generalist VisionLLM v2 also shows promising performance on RefCOCOg. These results demonstrate the strong fine-grained region captioning capabilities of our model.Visual Grounding. Visual grounding is a crucial vision task that associates the language description with the specific object within an image. Using the box or mask as the output format, visual grounding can be further categorized into referring expression comprehension (REC) and referring expression segmentation (RES) tasks. We comprehensively list the comparison results for the two tasks in Table A2 and Table A3 , respectively. From Table A2 , it is found that VisionLLM v2 achieves the competitive performance on RefCOCO [204] among MLLMs. We also showcase that VisionLLM v2 exhibits remarkable pixel-level segmentation capacities in Table A3 . Without further fine-tuning, our model demonstrates the good gIoU result of 51.0 on the challenging ReasonSeg dataset [89] .Semantic Segmentation. In addition to the instance-level segmentation, our model also has the capacity to address the task of semantic segmentation. We present the results on ADE20K [222] in Table A4 . The previous works mainly follow the standard training setting for 160k iterations on 8 GPUs with a total batch size of 16. As ADE20K only constitutes a small proportion of our joint-training dataset, our generalist model has a slightly inferior performance on this dataset. By fine-tuning this dataset with fewer training iterations, i.e., 45k, VisionLLM v2 can achieve a mIoU of 52.3 points, surpassing the previous methods under the same backbone.Interactive Segmentation. Interactive segmentation [82] is an emerging task that uses visual prompts as conditions for instance segmentation. We compare our method with state-of-the-art approaches on the COCO-interactive dataset [218] in Table A5 . This dataset, proposed by [218] , utilizes points, scribbles, boxes, and masks as visual prompts and is annotated on the COCO dataset [104] . As shown in the table, our generalist model VisionLLM v2 demonstrates performance advantages over SEEM-B [230] across all metrics but falls behind the recently proposed MLLM method PSALM [218] . We hypothesize that this is due to our region encoder being frozen during stage-3 of training, which constrains the model's performance. Therefore, we further fine-tune our model on this task by unfreezing the region encoder. It is observed that the performance of our model is significantly improved, as illustrated in the last row of Table A5 . method Flickr30K NoCaps Flamingo-80B [8] 67.2 -Kosmos-2 [140] 66.7 -BLIP-2 [97] 71.6 103.9 InstructBLIP [42] 82.8 121.9 Shikra-13B [27] 73.9 -ASM [181] 87 [142] and NoCaps validation set [6] , using CIDEr as evaluation metric. For region captioning on Visual Genome [86] , full set refers to the use of all validation samples for evaluation, while subset denotes the 5000 samples specified by [148] . [109] . Visualization across various domains. Besides the quantitative results, we also display the visualization results of VisionLLM v2 across various domains. As illustrated in Figure A1 , our model also shows strong perception capacities for remote sensing, PCB, and medical images.↑ Em ↑ F ω β ↑ M ↓ Sm ↑ Em ↑ F ω β ↑ M ↓ Sm ↑ Em ↑ F ω β ↑ M ↓ Sm ↑ Em ↑ F ω β ↑ M ↓ Sm ↑ Em ↑ F ω β ↑ M ↓ PoolNetZero-shot Image Captioning. Benefiting from the joint training on large-scale vision-language datasets, VisionLLM v2 exhibits promising capacities for zero-shot image captioning. As shown in Table A1a , both VisionLLM v2-Chat and VisionLLM v2 achieve competitive performance on Flickr30K [142] and NoCaps [6] compared with previous methods.Zero-shot Object Detection on OdinW13. We conduct the zero-shot object detection evaluation on OdinW13 dataset [98] , as shown in Table A8 . The results demonstrate that our VisionLLM v2 with a Swin-Tiny backbone is even on par with GLEE-Plus [190] with a Swin-Large backbone in AP avg . This indicates that our model benefits from the extensive dataset joint training, thereby providing robust general object detection capabilities.In-Context Segmentation Method mIoU Painter [183] 44.26 SegGPT [184] 54 In-Context Segmentation & In-Context Image Captioning. To evaluate the in-context learning ability of VisionLLM v2, we compare the results of in-context segmentation and in-context image captioning in Table A9 . For in-context segmentation, we construct a benchmark based on the validation set of COCO2017, where the number of in-context examples used during inference ranges from 1 to 5. For in-context image-captioning, we follow the same evaluation protocol as OpenFlamingo [11] and use 4-shot to assess the performance between different methods. The validation set is built upon COCO2017. From the table, VisionLLM v2 exhibits clear performance advantages compared with state-of-the-art methods in both in-context learning settings, which demonstrates the superior in-context capacities of our method.Super-Link Queries v.s. Token Embeddings in LISA [89] . Current MLLMs [89, 148, 151] introduce a segmentation token [SEG] into the LLM vocabulary and directly use its corresponding token embedding as a condition for SAM [82] to achieve pixel-level segmentation, which we refer to as the token embedding method. We also ablate this method for linking the LLM with taskspecific decoders, as shown in Table A10 . The performance difference between the two methods is negligible for tasks using text prompts, such as instance segmentation. We hypothesize that this is because the category names are seen during training, allowing the token embeddings to Table A10 : Ablation on the comparison between super-link queries and token embeddings.We evaluate the results on the four crucial visual perception tasks: instance segmentation, visual grounding, pose estimation and interactive segmentation. Our default setting is marked in gray .effectively capture category semantics. However, our super-link queries method outperforms the token embedding method for more open-ended tasks, such as interactive segmentation with visual prompts, demonstrating the greater flexibility of our approach.We emphasize two fundamental differences between the two methods: (1) The token embedding method requires sequential prediction of the special tokens during inference, which is time-consuming when the number of tokens is large. In contrast, our super-link technique requires only a single forward pass and the super-link queries would be automatically appended. This is efficient for cases requiring many tokens, such as image generation.(2) The super-link queries are not constrained by the cross-entropy loss of the LLM, allowing for more flexible and stronger representations for open-ended tasks.Visual Perception. We evaluate VisionLLM v2 on various visual perception tasks and display the visualization results from Figure A2 to Figure A6 . The qualitative examples showcase that VisionLLM v2 exhibits strong visual perception capacities, from coarse to fine-grained perception (box, keypoint, pixel), from basic to novel classes, from commonly-seen domains to long-tailed domains (natural scenes, industry, agriculture, etc).Visual Generation. Figure A7 shows more text-to-image generation results of VisionLLM v2. It could be observed that our model could generate high-quality images that not only properly follow the concepts and relations but also different styles specified in the instructions. Figure A8 shows more instructed-based image editing results of VisionLLM v2. Our model could successfully perform image editing for over five types of editing instructions, such as style transfer, object replacement, object addition, and attribute change. Please assist me in identifying the gas cylinders within the image.Please assist me in identifying the workers within the image.Please assist me in identifying the vegetables within the image.Please assist me in identifying the red blood cells within the image. A room with chairs, a table, and a woman in it.A woman posing for the camera standing on skis.A blue Multimodal In-context Learning Ability. To qualitatively verify the in-context capabilities of our model after trained on MMIC, we provide comprehensive visualizations across different tasks. As demonstrated in Figure A9 , A10, A11 and A12, our method can handle both visual and textual prompts, enabling it to perform tasks that require understanding and integration of information from different modalities. In addition, our models can distinguish between different prompting strategies and can correctly use the corresponding detection or segmentation tools to obtain the expected output based on given in-context examples. including categories such as cats, dogs, fruits, vegetables, food, cars, birds, etc. Additionally, we also make efforts on in-context object detection, in-context object segmentation, in-context captioning, in-context OCR, and in-context VAQ. Inspired by [157] , visual marks can also serve as the input for multimodal LLMs. As a result, we design five types of visual marks: circle, hand-drawn circle, arrow, box, and mask. Each visual mark can be either solid or hollow. We primarily construct this dataset based on COCO [104] , AP10K [203] and some OCR datasets [165, 161] , where we randomly sample M (∈ [1, 5] ) instances per image. The same type of visual mark is used to highlight the selected instance within one image, ensuring consistency and clarity for the model's learning process.The examples of constructed instructions can refer to Figure A9 , A10, A11 and A12. The entire dataset has constructed a multimodal corpus with ∼862K question&answer pairs. We expect that this dataset can further advance the development of this field. Settings of Stage-1. Stage-1 consists of pretraining and instruction tuning phases as [107, 105] .As shown in Table A13 , in the pretraining phase, We freeze the LLM. And only the region encoder and projections for image embedding and region embedding are trained for efficiency. We adopt the AdamW optimizer [119] with the peak learning rate of 1e-3 and weight decay of 0. The training involves a total batch size of 2048 across 64 A100 GPUs. In the instruction tuning phase, LLM is unfrozen for full-parameter training. The peak learning rate is decreased to 2.5e-5 for training stabilization. The model is trained on 64 A100 GPUs with a total batch size of 1024. And we begin adopting the dynamic resolution approach [106, 33] in this phase. The maximal number of local patches, i.e., max tile, is set as 4.Table A16 : A list of instructions for multi-region caption. Table A20 : A list of instructions for semantic segmentation.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.We consider tasks such that those with differing input and output formats, or those involving data from different domains as distinct tasks. 


Decision: The work presents a Vision-Language model for producing diverse output types like text, bounding boxes, segmentations, pose, segmentation keypoints etc. The key novelty is a "super-link" mechanism based on learnable soft-prompts for connecting a multimodal LLM to various task-specific decoders. 

The work received positive scores (A, BA, BA) from reviewers. The reviewers appreciated extensive experiments, good performance, and presentation/writing. Concerns raised by reviewers included complexity of the 3 stage training pipeline and transfer between tasks. The authors rebuttal appears to have satisfied these reviewer concerns. 

Having read the reviews, rebuttal, and the discussion, the AC generally agrees with the reviewers and is happy to recommend "accept". The AC recommends the authors to incorporate the discussion in their revised draft and also to refer to the figures and tables in the appendix in the main paper (for example, would be useful to refer the reader to Fig A13 in the caption of Fig 2 in the main paper). The AC also found Fig.1 to be too abstract and not be very informative and would suggest the authors modify it to ground it more to the proposed approach (for examples, its unclear from the figure what the difference between a,b,c are and how exactly each of them is implemented).

Reviews: review: paper_summary: paper_summary: This paper proposes an end-to-end generalist Multimodal Large Language Model (MLLM) for a variety of vision-language tasks, including captioning, detection, segmentation, and image generation. It introduces the concept of "Super Link" for triggering different tasks and attaches corresponding task decoders to the MLLM through query embeddings for end-to-end gradient updates. The model demonstrates strong results on the problems of interest and opens a new avenue for future multimodal generalist studies.
strength_weakness: strength_weakness: The paper lacks a comprehensive review of related work. Relevant works such as AnyGPT [r1], Chameleon [r2], the CM3 series, and other earlier publications that unify image generation in MLLM with image quantization and decoding should be reviewed and acknowledged.

Clarification is needed on how super-link queries are generated. For [DET] and [SEG], these can be boilerplate templates, but what about text-to-image generation? Are the super-link queries generated on-the-fly based on the context? Examples for [GEN], similar to the one at the bottom of Page 5, would be appreciated.

Regarding the joint multi-task training stage elaborated in Sec. 3.3, ablation studies on how each task benefits the others should be conducted. Does performing fine-grained vision tasks enhance the model's image generation capability, or do multimodal understanding and generation remain conflicting as shown in the literature? Do detection/segmentation tasks lead to better controllability or spatial reasoning for image generation? These questions need to be studied.

[r1] Zhan et al., AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling.

[r2] Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models.
questions: questions: see previous sections
limitations: limitations: appear to be sufficient
paper_summary: paper_summary: This paper introduces an advanced multimodal large model (MLLM) that integrates visual perception, understanding, and generation in a unified framework. Unlike traditional models limited to text outputs, it expands its capabilities to tasks like object localization, pose estimation, and image generation and editing via a "super link" mechanism to connect the MLLM with task-specific decoders, facilitating flexible information and gradient feedback transmission while resolving multi-tasking training conflicts. The model is trained on data from hundreds of vision and vision-language tasks, allowing it to generalize across these tasks with shared parameters and achieve performance comparable to task-specific models. VisionLLM v2 aims to enhance the generalization of MLLMs.
strength_weakness: strength_weakness: 1. The training process is relatively complex, consisting of three stages, making it difficult to promote and use in the industry. The method described in the paper seems like it could also be done in a single end-to-end training stage, similar to LLMs. [1] is the one-stage training general vision model. Discussing the general models [1][2]  trained in a one-stage manner with a pure language interface would be better. :) 
2. In Table 3, the detection performance is basically on par with the single-task decoder performance later on. Does this imply that the performance mainly depends on the task-specific decoder? Is the method proposed in the paper primarily aimed at facilitating the integration and scheduling of various decoders? Will the computation of MLLM enhance the performance of the subsequent decoder?
3. The advantage of LLMs is not only their compatibility with various tasks but also their ability to mutually enhance different tasks, which is the so-called multi-task capability. In the method described in the paper, different tasks share the same MLLM and encoder. So, can joint training of different tasks, like GiT[1], lead to mutual enhancement?

Points 2-3 are just areas where I think there might be room for improvement, but they won't significantly affect my decision on whether to accept this paper. It's just for discussion. :)

[1] GiT: Towards Generalist Vision Transformer through Universal Language Interface. (ECCV2024)

[2] Fuyu-8B (blog)
questions: questions: This question is just to hear the authors' perspectives: Should a general model use a bridging approach (LLaVa or VisionLLM) or a simple multi-layer transformer (Fuyu8B, GiT, Gemini)? The former might be easier to implement but has complex training and scaling issues, while the latter has greater advantages in scaling and simpler training, similar to GPT. (Authors can answer freely without considering that the paper uses a bridging framework. This will not affect the final score.) :)
limitations: limitations: This is purely academic research and does not involve any potential negative societal impact.
paper_summary: paper_summary: The paper proposes a model that supports a wide range of multimodal tasks, beyond text generation. The approach leverages an LLM, modality specific encoders and task specific decoders to effectively handle different tasks. The LLM communicates with task decoders via  “super link”,  which consists of special tokens and soft prompts for each task.
strength_weakness: strength_weakness: 1. The paper title is a bit misleading. It gives the impression that the paper proposes a single model that can handle many tasks, while it is basically an agglomeration of many powerful pretrained models (CLIP, LLM, Stable Diffusion, UniPose…). 
2. The contribution  is limited. The super link is basically soft prompts (learnable tokens) and special tokens per task. Soft prompt is widely used in other approaches (shared learnable query as in InstructBLIP) or (learnable query per task as in eP-ALM/MAPL). For the special tokens, FROMAGe uses special tokens to handle multimodal outputs. The main contribution of the paper is in scaling this approach to many multimodal tasks. It is also important for the paper to discuss these similar approaches.
3. The paper lacks details about the model training. In particular which training objectives are used on top of the task-specific decoders? And how the different losses are weighted?
4. Did the authors experiment with one-stage training instead of the 3-stages training? I didn’t find any experiment to support this design choice
5. The paper claims “Multimodal In-Context Ability. our model …  exhibits superiority over the previous in-context models …” However there is no quantitative comparison to support the superiority of the proposed model in a a few-shot ICL setting.

InstructBLIP: Dai, Wenliang, et al. "Instructblip: Towards general-purpose vision-language models with instruction tuning." Advances in Neural Information Processing Systems 36 (2024).

eP-ALM: Shukor, Mustafa, Corentin Dancette, and Matthieu Cord. "ep-alm: Efficient perceptual augmentation of language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

MAPL: Mañas, Oscar, et al. "Mapl: Parameter-efficient adaptation of unimodal pre-trained models for vision-language few-shot prompting." arXiv preprint arXiv:2210.07179 (2022).

FROMAGe: Koh, Jing Yu, Ruslan Salakhutdinov, and Daniel Fried. "Grounding language models to images for multimodal inputs and outputs." International Conference on Machine Learning. PMLR, 2023.
questions: questions: Check weaknesses section (e.g. 3-4)
limitations: limitations: The authors discuss the limitations and the broader impact in the paper.

Comments: title: Official Comment by Authors
Thanks for your positive feedback
Questions have been addressed in the rebuttal
Concerns addressed?
Official Comment by Authors
comment: Thank you for your thoughtful feedback. We design the three-stage training strategy in this work to effectively train the model from an initial vision encoder and a large language model, with the primary goal of maximizing performance.

We agree that it is meaningful for training simplicity and efficiency. As mentioned in common questions Q2, our model has the potential to be trained in a single stage by adjusting the dataset sample ratios. And we believe the performance gap between the one-stage and three-stage strategies could be significantly reduced by starting with pretrained, strong vision-language models (e.g. LLaVA, InternVL). In our recent exploration, we directly load the pretrained InternVL and augment it with Grounding-DINO. We meticulously adjust the dataset ratios of all combined datasets to simulate 1 epoch training on chat data and 12 epoches training on detection data. Then the model is trained within one stage to simplify the training pipeline. Through this careful configuration, we observe the much smaller performance decrease: -0.5 / -2.0 points on MMB EN/CN compared with original InternVL, -1.3 points for object detection on COCO compared with original Grounding-DINO.

In the final version of our paper, we will include a detailed discussion on the limitations of our approach compared to single-stage works like GIT and Fuyu-8B, highlighting the trade-offs between performance and training simplicity.

Thank you once again for your valuable feedback. We will incorporate these considerations into our future research to explore simpler training strategy, with the hope that our work will ultimately bridge the gap between academic research and industrial application.
We are glad we could address your questions. We sincerely appreciate the time and effort you put into reviewing our paper and providing valuable comments. We will ensure the revisions are incorporated into the final version.
Thank you for the author's response. Regarding Q1, I'm sorry to say that I am not yet fully convinced. A single-stage training framework is essential for industrial applications; otherwise, the final paper might remain confined to academia. The frameworks that can be widely adopted in the industry are those that are simple and effective. Even if a single-stage training approach may reduce performance, the industry is likely to prefer it. We all hope to create work that is truly useful for the industry, not just a paper. Therefore, I hope the author will discuss this limitation in the camera-ready version, comparing it with single-stage works like GiT and Fuyu-8B. Looking forward to seeing the author's future work on refining and optimizing the training strategy.
The rebuttal has addressed the questions from the initial review and the authors indicated corresponding revisions in the final version. Therefore, I maintain my original positive review.
Dear reviewer, the authors have responded to your questions - are you satisfied by their response? If so, would you like to update your rating? Is there any more information you need from the authors to make that decision?
Thank you so much for taking the time to re-evaluate our paper, thoughtfully considering our responses, and agreeing to raise the score. We will carefully take your valuable suggestions and continuously improve our paper.
Thanks for detailed feedback. The reviewers addressed most of my concerns. I am still not convinced about the originality of the work. However, the work contains some important messages that could benefit the community. Thus, I will raise my score.