VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Authors: Wenhai Wang
Zhe Chen
Xiaokang Chen
Jiannan Wu
Xizhou Zhu
Gang Zeng
Ping Luo
Tong Lu
Jie Zhou
Yu Qiao
Jifeng Dai
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Vision-Language Model
Detection
Image Caption
TL;DR: VisionLLM is an LLM-based framework that allows open-ended vision and language tasks to be defined and performed using language instructions.

Abstract: Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over 60% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The code shall be released.

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Wenhai Wang*
Zhe Chen*1.3
Xiaokang Chen Jiannan Wu*
Xizhou Zhu 1.6
Gang Zeng*
Ping Luo
Tong Lu
Jie Zhou
Yu Qiao' Jifeng D
'OpenGVLab, Shanghai AI Laboratory
'The Chinese University of Hong Kong
3 "Nanjing University
*Peking University 'The University of HongKong TTinghua University
Code: http:: htgpttab..cm//peeee//aa/aaa.....
Abstract
Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an --seed framework for vision-centric tasks, termed VisionLLM. This framework provides unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be fiexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level coarse- sonsse-grained task-level customization, all with good results. It's noteworthy that, with generalist LLM- based framework, our model achieve over 60% mAp on COCO, on par with inrection-specifii models. We hope this model can set new baseline for generalist vision and language models. The code shall be released.
Introduction
The emergence of large language models (LLMs) like ChatGPT [35] has revolutionized the landscape of artificial general intelligence (AGI), showcasing their impressive zero-shot capabilities in ad- dressing various natural language processing (NLP) tasks through user-tailored prompts or language instructions. Despite these advancements, it's essential note that the triumph of LLMs does not effortlessly extend to pure vision and vision-language tasks, due to the inherent disparities between modalities and task formats.
The field of computer vision presents a unique set of challenges and paradigms that differ from those of NLP. The traditional paradigm of vision foundation models is pre-training followed by fine-tuning [51, 11, 43, 53, 17, 44], which is effective but comes with significant marginal costs when adapting diverse downstream scenarios. As shown in Figure la, while approaches such as multi-task unification [38. 50, 1, 49, 72] have been used to achieve generalist capability, they often struggle to overcome the limitations imposed by pre-defined tasks, resulting in a gap in open- ended task capabilities compared to LLMs. Recently, visual prompt tuning [24, 66, 70, 67, 54] has emerged as way flexibly outline some pure vision tasks (see Figure lb), such as object detection, instance
* equal contribution, corresponding author (daijifeng @tsinghua.eduu.c))
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Vision Generalist Model
Desired output: <cl> <p1> <p3> ...
11
Visual Prompt Tuning
Vision LLM
Pre-defined tasks:
detection, captioning,
VQA, grounding,
Task defined by instructions
(a) Vision generalist models [51, 53, 74] are constrained by the for- mat of pre-defined tasks.
(b) Visual prompt tuning [24, 56, 54] are inconsistent with the for- mat of LLMs.
(c) VisionLLM (ours) can flexiily manage ciston-centric tasks using language instructions like LLMs.
Figure 1: Comparison of our Li with popular paradigms. Unlike current vision generalist models that depend on pre-defined task formats and visual prompt tuning models that inconsistent with large language models (LLMss), L. leverages the power of LLMs for open-ended vision tasks by using language instructions.
segmentation, and pose estimation, using visual masking. However, the format of visual prompts considerably deviates from that of language instructions, making it challenging to directly apply the reasoning abilitics and world knowledge of LLMs to vision tasks. Therefore, there is an urgent need for unified generalist framework that can seamlessly integrate the strengths of LLMs with the specific requirements of vision-centric tasks.
In this work, we present VisionLLM, novel framework that aligns the definitions of vision-centric tasks with the methodologies of LLMs. Leveraging the reasoning and parsing capacities of LLMs, VisionLLM is designed to empower open-ended task capabilities for vision-centric tasks. Specifically, comprises three core components: (1) unified language instruction designed for vision and vision- saiion-angguag tasks, (2) a language-guided image tokenizer, and (3) an LLM-based open-ended task decoder that orchestrates various tasks using language instructions. With this framework, a wide range of vision-centric tasks can be seamlessly integrated, including object detection, instance segmentation, image captioning, and visual grounding. In addition, the framework also facilitates task customization at different levels of granularity, allowing for the customization of target objects, output formats, task descriptions, etc.
Compared to current popular API-based applications [60, 65, 42, 32, 28], our model takes unified, end-to-end approach to integrate VFMs and LLMs, streamlining and enhancing the overall efficiency of the overall process, and leveraging the strengths and data of both VFMs and LLMs within single, cohesive system. Furthermore, our model surpasses the limitations of generalist vision models pre-trained on pre-defined tasks. VisionI.I.M can effectively manage vision-centric tasks through language instructions, embodying a flexible and open-ended approach that is not constrained by pre-set tasks. This versatility makes VisionLLM a robust and powerful generalist model for vision and language tasks, opening up new possibilities for the development of unified generalist models that bridge the domains of vision and language.
In summary, our main contributions are as follows:
(1) We propose VisionLLM, the first framework that leverages the power of LLMs to address vision- centric tasks in open-ended and customizable manner. By aligning the definitions of vision-centric tasks with L.M methodologies, VisionIIMM breaks new ground in enabling the unified modeling of vision and language, opening up possibilities for advancing the field.
(2) We overcome many difficulties when porting LLMs to vision-centric tasks, by designing unified language instruction that matches the format of language models and covers various vision-centric tasks including visual perception. Correspondingly, we develop a language-guize and LLM-based task decoder that can handle open-ended tasks according to the given language instructions based on the LLMs' reasoning and parsing capabilities.
image tokenizer
(3) We construct a series of tasks with different granularities verify the effectiveness of our models, ranging from easy to hard, and from pre-defined to flexible. Through these validations, we demonstrate the remarkable generality of our models, showcasing their ability to handle diverse

---

Human: "For each object in <image> that is a member of l'motorbike': <c0>), output tuple with the class label and the coordinates of a polygon with 16 points that encloses the object. The coordinates should be within the range 1024, The output format should be (c, x1. y1. X2, y2, x16, y16)."
Human: "Identify the objects <image> belong to ('What the child eating? 3 <cO>, 'red gamepod': <c1>] and drow a bounding box around each one. The outout should a of tuples in
SLOP
the format (G, x1, yi, x2, y2), where is the class label and x1, yl, x2, y2 ore the coordinates of the top-left and bottom-right corners of the bounding box, The coordinates should within range 1024."
VisionLLM: "The detected polygons are [(<c0>, 135,3, 95 123.4, 53.4, 84.9, 57.6, 66.8, 60.5, 60.1, 72.3, 34.2, 71.4, 124.9, 119.3)J."
Vis "The bounding bowes are [[<c0>, 226.4, 229.8. 363.1, 347.4), (<cl>, 441.1, 183.5, 538.6, 269.9)1.
(a) Object Detection with Customized Class Set
(b) Object Location with Editable Outpur Format
Human: "Provide long
description of the <image>."
Human:
"With objects <image>, if you feel hot, what you do?"
VisionLLM: "The image features white dog loying down foldoble outdoor
VisionLLM:
"First, You can also try turning off the computer and other electricol appliances in the room to reduce the heat generated by their operation. As an alternative, you can use the desk fan provided in the image to create cooling effect in the room. You can plug in the fan and adjust its settings to ensure it blows air in a direction thot reaches the room's occupants effectively."
cooling pad,
which is placed on top of wooden tablé. The dog has its head in the middle of the cooling pad, and a fan is positioned behind it, blowing oir onto the pad. The table is outdoors, chairs placed around it, one on each side. The scene appears to be set in a sunny weather, with bright and sunny sky, which contributes to the overall outdoor atmosphere."
grossy area, and there
two
(c) Image Description with Controllable Text Length
(d) Visual Question Answer with Complex Reasoning
Figure 2: Results and visualizations of our VisionLLM. Guided by language instructions, our unified generalist framework showcases its effectiveness on diverse open-ended vision-centric tasks. The text marked with gray background indicates the customized instructions and the desired outputs.
scenarios, including random object categories, random output formats, and random task descriptions, as shown in Figure 2. The successful outcomes of these validations underscore the tremendous potential of our model in harnessing the capabilities of LLMs to control and guide vision-centie tasks. In addition, with generalist LL-M-based framework, our model also yields promising results on various vision-centric tasks. Notably, our generalist model achieves impressive mAP score of 60+% on the COCO dataset, surpassing many eetection-speciffe models [73, 6, 20] and approaching the state-of-the-aat record.
2 Related Work
2.1 Large Language Model
Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP) and artificial general intelligence (AGI), due to their impressive capabilities in language generation, in-context learning, world knowledge, and reasoning. The GPT family, including GPT-3 [5], ChatGPT [35], GPT-4 [34], and InstructGPT [36] are most representative works of LLMs. Other LLMs like OPT [69], LLaMA [46], MOSS [14], and GLM [68] have also made substantial contributions to the field. These models achieve high performance and are open-sourced, serving as valuable resources for training large models and as foundations for further fine-tuning for specific purposes. For instance, Alpaca [45] introduces a self-instruct framework that facilitates instruction tuning of the LLaMA model, reducing the reliance on human- written instruction data. Recently, the emergence of these LLMs has also opened up API-based applications for solving vision- centric tasks. These applications have integrated visual APIs with language models to enable decision-makinn
or planning based on visual information, such as Visual ChatGPT [60], MM-REACT [65], HuggingGPT [42), InternGPT [32], and VideoChat [28]. However, despite the convenience of using language-bar based instructions to define tasks and describe visual elements, these interactive systems [60, 65, 42, 32, 28] still face limitations capturing fine-grained visual details
and understanding complex visual contexts, which hinder their ability to effectively connecting vision
3

---

and language models. In summary, while LLMs have shown tremendous potential in various NLP applications, their applicability to vision-centric tasks has been limited by the challenges posed by modalities and task formats.
2.2 Vision Generalist Model
The pursuit of generalist models [74, 33, 62], which aim to handle wide range of tasks using shared architecture and parameters, has been a long-standing goal in the machine learning community. Inspired by the success of sequence-to-sequenee (seq2seq) models in the field of NLPP [38], recent advancements such as OFA [50], Flamingo [1], and GIT [49] propose modeling diverse tasks as sequence generation tasks. Unified-IO [33], Pix2Seq v2 [8] and UniTab [63] extend this idea by using discrete coordinate tokens to encode and decode spatial information for more tasks. Gato [39] also incorporates reinforcement learning tasks into the seq2seq framework, while GPV [19] develops general-purpon vision system by combining a seq2seq module with a DETR-based visual encoder [6]. However, these methods suffer from some limitations, such as slow inference speed and performance degradation due to the non- poraallll auto-regressive decoding process. Uni-Perceive [74, 72, 26] solve these issues by unifying different tasks using the maximum likelihood target for each input based on representation similarity, regardless of their modality, making it possible to support both generation and non-genennn non-generation tasks unified framework. Nevertheless, these generalist models are still restricted by pre-defined tasks and cannot support flexible open-ended task customization based on language instructions like LLMs.
2.3 Instruction Tuning
Language instructions are a powerful way to express various NLP tasks and examples for LLMs, as introduced by GPT-3 [5]. Following this idea, subsequent works, such InstructGPT [36], FLAN [13, 59], and OPT-IML [23], explore the instruction-tune tutiing method [58, 57] and demonstrate that this simple approach effectively enhances the zero-shot and few-shot capabilities of LLMs. The language instruction paradigm has also been adopted by the computer vision community to define image-to-text tasks. Flamingo [1] is milestone work that uses vision and language inputs as prompts and achieves remarkable few-shot results in various vision-language tasks, such as image captioning [9] and VQA [2]. BLIP-2 [27] further connects the visual encoder with LLMs through a querying transformer and linear projection layer to build strong multimodal models. MiniGPT-4 [71] and LLaVA [30] finetune the 2LIP-2-style models on synthetic multimodal cestructionfoollwwvvi data to unleash the potential of LLMs. However, these models mainly focus on image-to-ted tasks and fail to address visual perception, such as object detection, instance segmentation, pose estimation, etc. To tackle image inpainting tasks, Bar et al. [3] introduces the first visual prompting framework that utilizes inpainting with discrete tokens on images. Painter [55] and SegGPT [56] employ masked image modeling on raw pixels for in-context learning with paired images. While these visual prompt models demonstrate good results in segmentation tasks, their applicability to numerous real-world vision tasks is challenging. Moreover, defining the visual prompts as image inpainting is inconsistent with the language instructions in LLMs, hard to leverage the reasoning, parsing ability, and world knowledge of LLMs. In this work, aim to align vision-centric tasks with language tasks, use language instructions to unifiedly and flexibly define all tasks, and solve them with a shared LLM- based task decoder.
3
VisionLLM
3.1 Overall Architecture
This work targets to provide a unified generalist framework that can seamlessly integrate the strengths of large language models (LLMs) with the specific requirements of vision-centric tasks. As shown in Figure 3, the overall architecture of VisionLLM consists of three key designs: (I) a unified language instruction that provides a consistent interface for vision-centric task definition and customization; (2) language-guut guided image tokenizer, which encodes visual information in alignment with the given language prompt, enabling the model to comprehend and parse the visual content effectively; and (3) an -.---d open-task decoder, which utilizes the encoded visual information and language instructions to generate satisfactory predictions or outputs. The three designs work together to achieve a flexible and open-ended framework that can handle various vision-centric tasks at different levels of task customization through language instructions.

---

Language-Guides
Desired Output: <c1> <p1> <p3>...
Random Query
Image Token
Fg
Language- Guided
Open-Ender Task Decoder with LLM
Backbone
Image Tokenizer
Fc
<text>
<text>
Vision-language example: "Describe the image <image> in details."
Language Instructions <text>
Vision-only example: "For each object in image <image> that is a member of class set <class>, output a tuple with the class label and the coordinates of polygon with 16 points that encloses the object. The coordinates should be within range <range>. The output format should (c, x1, yI,
Figure 3: Overall architecture of the proposed VisionLLM. It consists of three parts: unified language instruction designed to accommodate both vision and vision-language tasks, an image tokenizer that encodes visual information guided by language instructions, and an LLM-based open- ended task decoder that exccutes diverse tasks defined by language instructions.
Different from previous interactive systems [60, 65, 42, 32, 28] that rely on APIs, our Vision LLM presents a more flexible and end-to-end pipeline. Given language instructions that describe the current tasks and input image, the model first uses a sanguage-guider image tokenizer to encode the image tokens based on the given prompt. Then, the image tokens and language instructions are fed to an LLM--bsed open-ended task decoder. Finally, it evaluates the generated outputs against the task definition given by the unified language instructions, enabling the model to produce task-specific results. This seamless, end-to-end pipeline enables VisionLLM to effectively combine vision and language, achieving remarkable performance in open-ended and customizable vision-centric tasks.
3.2 Unified Language Instruction
We first introduce unified language instructions to describe vision-centric tasks. This design enables the unification of various vision-only and vision-language task descriptions and allows for flexible task customization.
Vision-Language Tasks. The instructions for vision-language tasks such as image captioning and visual question answering (VQA) are straightforward and similar to NLP tasks. Following previous methods [27, 74, 30), we describe the image captioning task like "The image is <image>. Please generate a caption for the image: ", and the VQA task like "The image is <image>. Please generate an answer for the image according the question: <question>", Here, <image> and <question> are the placeholders of the image tokens and the question, respectively. The image tokens are directly placed at the placeholder <image>.
Vision-Only Tasks. Designing effective language instructions for vision tasks is a challenging endeavor due to the differences in modality and task format between vision and language. Here, describe vision tasks by providing a task description and specifying the desired output format via language instructions.
(1) The task description conveys the intended task to the language model. Following self-instrus [57], we design set of seed instructions with placeholders and employ LLMs to generate a large number of related task descriptions and randomly select one them during training.
(2) For conventional visual perception tasks like object detection and instance segmentation, we propose a unified output format represented as tuple (C, P), where C' denotes the class index in the category set <class>, and P = {I,, 34 J represents N points that locate the object. To align with the format of word tokens, both the class index C and the coordinates of points 2;, y; are transformed into discretized tokens. Specifically, the class index is an integer starting from 0, and the continuous coordinates of the points are uniformly discretized into an integer within the range [-<range>, <range>]. For object detection and visual grounding tasks, the point number N is equal to 2, representing the the top-left and bottom-right points of object's bounding box. In the case of instance segmentation, we employ multiple (N > 8) points along the object boundary to represent an instance mask [61]. Other perception tasks such as pose estimation (keypoint detection) can also be formulated as language instructions in this way.
5

---

An example of language instruction for the instance segmentation task is as follows: "Segment all the objects of category set <class> within the <range> of the image and generate a list of the format (c, xl, yl, x2, y2, ... x8, y8). Here, represents the index of the class label starting from O, and (x1, yl, x2, y2, x8, y8) correspond the offsets of boundary points of the object relative to the center point. The image is: <image>".
3.3 Language-Guided Image Tokenizer
VisionI..M considers images as a kind of foreign language and converts them into token represen- tations. Unlike previous works [16, 52, 31] that utilize fixed-size patch embeddings to represent images, we introduce the language-guu guided image tokenizer to flexibly encode visual information that aligns with task-specific language prompts or instructions.
Specifically, give an image X E IR HRW x3 with height H/ and width W, we first feed the image backbones (e.g., ResNet (21J) and extract visual features F2 of four different scales. Addition- ally, we leverage text encoder (e.g., BERT [15]) to extract the language features Fi from given prompts. The language features are then injected into each scale of visual features through cross- attention [47]. yielding multi-scale language-aware visual features, enabling the alignment of features across modalities.
Afterward, we propose adopt transformer- based network (e-g., Deformable DETR [73]) with M random-initue initiallized queries Q = {g: ) i=1 to capture the high-level information of images. We build the transformer-based network on top the multi-scale language-aware visual features extract M image tokens T ((e; 4) Jim=1, each of which is represented by an embedding e, and a location 4, denoting the semantic and positional information of the token. This design not only represents the images independent of input resolution but also extracts the visual representation that is informative with respect to the language prompts.
3.4 LLM-based Open-Ended Task Decoder
We build our decoder on Alpaca [45], an LLM that is adapted from LLaMA [46], to handle various vision-related tasks with language guidance. However, Alpaca has some inherent drawbacks for vision-centric tasks, such as (1) It only has a few digit tokens (e.g., 0~~9) in its vocabulary, which restricts its ability locate objects by numbers: (2) It uses multiple tokens to represent the category name, resulting in an inefficient scheme in object classification; and (3) It is a causal model that is inefficient for visual perception tasks.
To tackle these issues, we expand the vocabulary of LLM with additional tokens specially designed for vision-centric tasks. First, we add set of location tokens, denoted as (<p-512>, .... <p0>, <p512>}, where <p i> represents the discretized offset of i E [-512, 512] to the location 4 of the image token, and the relative value to image height or width is equal to i/512. These tokens successfully transform the object localization task from continuous variable prediction to more unified discrete bin classification. Second, we introduce semantics-agnostic classification tokens <<o>, <c1>, .... <c511>) to replace category name tokens, which overcomes the inefficiency of using multiple tokens to represent categories. The mapping between category names and the classification tokens is flexibly provided in the category set <class> of language instructions, such as "perssn"" " <co>, "car": : <<1>, "black cat" : <co>>...), This design allows our model to select the appropriate category name from the provided category set, facilitating efficient and accurate object classification.
Moreover, to address the inefficiency caused by the causal framework, introduce output- format-as-query decoding. We first use L.LMs to parse the structural output format from the task instructions (e.g.. "<cls> <x1> <y1> <x2> <y2>" for object detection, "<bos>" for image captioning), and then feed the tokens of struc- tural output format as queries to the decoder to generate the desired output according to the queries. This simple method enables our model to not only avoid inefficient token-by-toker de- coding in visual perception tasks, but also keep unified framework for rision-languae tasks.
LLM-based Open-Ended Task Decoder
Task defined by instructions
parsing
foomat 1: "< exi> <yl>...
format 2: "<hoss"
format n:
Figure 4: Illustration of the "output-format-as- query" decoding process. "<cls> <x1> <y1> denote the queries of the object's class index and boundary points, and "<bos>" denotes the begin- ning of string.

---

Note that, during both the training and inference phases in the object detection task, we input 100 sets of "<els> <xl> <yl> <x2> <y2>" "2>""m> the decoder, generating 100 object predictions. Those predictions with higher confidence scores will be retained, adhering to common practice of the object detection task.
In this way, the output of object location and classification is formulated as a foreign language, thus unifying these vision-centric tasks into the format of token classification. Therefore, both vision-language and vision-only tasks can be supervised with the cross-entropy loss like language tasks. In addition, for efficient training, we adopt the Low-Rank Adaptation (LoRA) approach [22], which allows us to train and fine-tune the models without excessive computational costs. We set the LoRA rank to 64 and use LoRA on the QKVO (Query, Key. Value, and Output) in the attention layers. It also acts as a bridge between the language and visual tokens, facilitating effective alignment between the two modalities, ensuring better task customization, and improving the convergence of the overall system.
Experiment
4.1 Implementation Details.
We implement two variants of VisionLLM with two image backbones, i.e., ResNet [21] and Internlmage-H [51]. For the language- guigedd image tokenizer, we adopt BERT-Large [4] as the text encoder and Deformable DETR (D-DETR) [73] to capture high-level information. For the LLM, we employ Alpaca-7B [45], a LLaMA [46] model fine-tuned with instructions, and equip it with LoRA [22] for parfieter-efffcientt fine-tuning.
The model is trained in two stages. In the first stage, we initialize the model with the pre-trained weights of D-DETR and BERT, and train the visual backbone and language-guiee image tokenizer to produce language-awan aware visual features. In the second stage, we connect the image tokenizer with Alpaca-7B and introduce the unified supervision of multiple tasks. We freeze the visual backbone while freezing most parameters of the LLM except few LoRA parameters. More details on the experimental setup can be found in Sec. B of the supplementary material.
4.2 Task-Level Customization
We first evaluate the task-level customization capability of VisionLLM. VisionLLM supports coarse- grained task customization, including visual perception tasks and iisual-languaa tasks. Table 1 presents the evaluation results on four standard vision-centric tasks, including object detection, instance segmentation, visual grounding, and image captioning. We compare our model with task- specific methods as well as recently-proposed mentioned, the results of our model come from shared-parameter generalist model and switch different lasks by changing the language instructions only. Detailed instructions could be found in the supplementary material.
proposed vision generalist models. Note that, unless specifically
Object detection a fundamental computer vision task that involves identifying and localizing objects of interest within image. Our method achieves comparable or higher results to others, 44.6 mAP, with a ResNet-50 [21] backbone. With the same backbone i.e. ResNet-550, method outperforms Pix2Seq [7] by 1.4 mAP, which also discretizes the output coordinates to integers. Furthermore, benefiting from the output-format-as saatt--o-mttt---uuueeeeee qusry framework (see Sec. 3.4), we can decode multiple predictions in parallel during inference, making our approach more efficient. Using InternImage-Ie [51] as the visual backbone, we obtained 60.2% mAP, which is close to the current state-of-the-ar detection-specific model [51], demonstrating the scalability of our generalist model.
Object Detection.
Visual Grounding. Visual grounding associates textual descriptions with corresponding regions or objects within image. Training visual grounding and object detection can potentially conflict with each other, as object detection aims to detect all the objects, while visual grounding should only localize the referred object and suppress other objects. Benefiting from our unified task instructions and the strong instruction comprehension capabilities of LLMs, our model performs both tasks effectively and achieves result of 80.6 P@0.5 for visual grounding. With InternlmaeeHH as the backbone, we achieve 86.7 P.0.5 on the validation set of RefCOCO.
Instance Segmentation. Instance segmentation involves identifying and segmenting individual objects within an image. We employ a flexible number of points (i.e., 8~24) along the object boundary to represent instance mask. Compared to mainstream models specific to instance
7

---

Table 1: Results standard vision-centrie tasks. "sep" indicates that the model is separately
trained on each task.
Open- Ended AP APso AP7s
Detection
Instance Seg.
Grounding
Captioning
Method
Backbone
AP APso APTs
P@0.5 BIDE-
Specialist Models
Faster R- C-NN-NPPP [40]
ResNet-50
x
40.3 61.0 44.0
DETR-DCS [6]
ResNet-50
X
43.3
63.1 45.9
Deformable-DETR [73]
Re
X
45.7
65.0 49.1
Mask R-CNN [20]
ResNet-50
X
41.0
61.7
44.9
37.1
58.4 40.1
Polar Mask [6]]
ResNet-50
30.5 52.0
31.1
Pix2Scq [7]
ResNet-50
43.2 61.0 46.1
UNITER [10]
ResNet-101
81.4
VIILA [18]
ResNet-101
82.4
MDETR [25]
ResNet-101 101
86.8
BEiT-3 [53]
ViT-g
147.6
VL-T5 [12]
T5-B
116.5
Generalist Models
UniTab [64]
ResNet- 101
88.6
115.8
Uni-Perceiver [74]
ViT-B
X
32.0
Uni-Perceeier-Moo [72]
ViT-B
X
33.2
Uni-Perceiver-W2 [26]
Swin-B
X
58.6
50.6
35.4
116.9
Pix2Scq v2 [8]
ViT-B
X
46.5
38.2
34.9
LMsionLLL-RR50..0
ResNet-50
44.8
64.1
48.5
25.2
50.6
22.4
84.4
30.8
112.4
VisionLLM-RR R50
ResNet-50
44.6
64.0
48.1
25.1
50.0
22.4
80.6
31.0
112.5
VisionLLM-H
Interlmaae-Ht
60.2 79.3 65.8 30.6
61.2 27.6
86.7
32.1
114.2
Table 2: Experiments of object-level and output format customization. We conduct these experiments based on VisionLLM-R50, and report the performance of box AP and mask AP on COCO minival for (a) and (b), respectively. "#Classes" and "#Points" indicate the number classes and boundary points, respectively. "*" indicates that we report the mean AP of the given classes, e.g.,
10 classes.
(a) Object-level customization.
(b) Output format customization.
#C Classes
AP
AP50
AP75
APs
APM
APr.
#Points
AP
AP50
AP75
AP3
APM
APr.
10"
48.9
72.6
51.2
31.7
47.5
67.3
8
18.5
45.7
11.6
9.9
19.7
28.7
20"
52.7
73.6
56.8
31.8
53.2
70.5
14
22.9
48.3
19.4
11.0
25.1
36,0
40"
49.3
70.7
53.2
33.1
53.6
63.8
16
24.2
49.9
20.9
11.5
26.3
36.8
80"
44.6
64.0
48.1
26.7
47.9
60.5
24
25.1
50.0
22.4
12.5
27.4
38,2
segmentation, our model has a comparable mask APso (61.2% with InternImage-Hi [51]) but relatively low mask AP73- This gap could potentially arise from factors as follows: (1) We discretize the output coordinates to integers for unifying tasks, which introduces information loss; (2) Due to the memory and computational constraint, the number of points in our model limited, which also results in performance drop: and (3) Point-based methods typically yield lower results compared to direct mask prediction methods, such as Mask R-CNN [20].
Image Captioning. We also evaluate our model representative vision-language task, i.e. image captioning task, and report the BLEU-4 [37] and CIDEr [48] metrics. Note that we do not adopt the CIDEr optimization [4 We can observe that VisionLLM achieves competitive performance to obtain BLEU-4 score of 31.0 and CIDEr score of 112.5. When using InternImage-B as the backbone, our model achieves comparable BLEU-4 score of 32.1 and CIDEr score of 114.2. These results demonstrate the effectiveness of VisionLLM in generating descriptive and contextually relevant captions for images.
previous methods. With ResNet-50,
4.3 Object-Level & Output Format Customization
Our VisionLLM not only allows for customizing the task description, but also for adjusting the target object and the output format using language instructions. Here, we evaluate our model's fine-grained customization ability on COCO. In particular, to customize the target object, we modify <class> in language instructions to change the model's recognition target from classes to 80 classes. Likewise, to customize the output format, modify the number of points in language instructions to change the task output format. Table shows that our method can perform well for both object-level and output format changes.

---

Table 3: Ablation studies on language-guided image tokenizer and hyper-parameters.
(a) Effect of text encoder in the language-guided image tokenizer.
(c) Effect the num- ber of bins (#Bins).
(b) Effect of image tokenization method.
#Bins
AP
BERT Freeze COCO
RefCOCO
Tokenization
AP
257
34.9
44.7
48.1
Average Pooling 23.1
513
40.8
44.8
84.1
Ours
44.8
1025
44.8
1.3
34.3
2049
44.8
4.4
Ablation Study
In this section, we analyze the effect of key components and hyper-paramene parameters on VisionLLM. Unless otherwise specified, we use ResNet-50 [21] backbone and perform the ablation experiments for object detection tasks with random classes and task descriptions on COC02017 [29].
Single Task vs. Multiple Tasks, We perform ablation study to assess the impact of multi-task learning with language instructions on Vis. As shown in Table 1, the single-task trained model VisionL is slightly better than the jointly trained model VisionLLM-RS except image captioning. This is due to the multitasking conflicts that also affect previous generalist models [74, 72], and it reflects trade-off between accuracy and generalization.
Text Encoder in Language-Guided Guided Image Tokenizer. We examine the role of text encoder (i.e., BERT) in our language-guider image tokenizer in Table 3a, where report the results for object detection and visual grounding. The first two rows show that BERT is not essential for object detection but it is crucial for visual grounding. We also investigate the effect of freezing the text encoder during training. The last row indicates that freezing BERT hinders the alignment of vision and language modalities and thus degrades the performance for both tasks.
Image Tokenization Method. As a comparison our query-based tokenization, we employ average pooling the feature maps from the D-DETR encoder to obtain M patch embeddings, which serve as token representations for the image. Results in Table 3b indicate a clear advantage our method. This is due its ability to capture information from objects of various sizes in more flexible way.
Number of Localization Tokens. We vary the number of localization tokens from 257 (i.e.- 128-~188 to 2049 (i.e,- -1024~102) to investigate its impact on visual perception performance. As presented in Table 3c, the model consistently exhibits improvement the number of localization tokens increases until it reaches a saturation point. Remarkably, a substantial performance boost observed when the number is raised from 257 1025 (+9.9 AP). These results indicate that higher number of localization tokens enables the models to achieve finer localization abilities, thereby improving localization accuracy.
Conclusion
In this paper, we have presented VisionLLM, a novel framework that leverages the power of large language models (LLMs) to address vision-centric tasks in an open-ended and customizable manner. We have designed unified language instruction that matches the format of language models and covers various vision-centric tasks including visual perception. We have also developed language-guid guided image tokenizer and an LLM-based lask decoder that can handle open-ended tasks according the given language instructions. We have verified the effectiveness of our models on a series of tasks with different granularities, demonstrating their remarkable generality and flexibility.
Broader Impact. We envision that this work will promote the fusion of visual and language tasks. In addition, since work is built on open-source pre-trained vision foundation models and large language models, requiring low training resources, thus reducing the carbon footprint. We do not foresee obvious undesirable ethical/social impacts at this moment.
Acknowledgement
The work supported by the National Key R&D Program of China (NO. 20222D01613000 the Na- tional Natural Science Foundation of China (Grant No. 62376134. 61672273, 61832008, 62372223). the Shanghai Committee of Science and Technology (Grant No. 21DZI 100100), and the Fundamental Research Funds for the Central Universities (No. XI2023000701).

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint 220: 14/98, 2022. 1,4
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick. and Devi Parikh, Vqa: Visual question answering. In Proceedings of the IEEE International Conference Computer Vision, 2015. 4
[3] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexci Efros, Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 2022. 4
[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding? In International Conference on Machine Learning, 2021."
[5] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerrs. Advances in Neural Information Processing Systems, 2020. 3,4
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020. 3, 4,8 8
[7] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection, arXiv preprint arXiv:2109. 10852, 2021. 7,8
[8] Ting Chen, Saurabh Saxena, Lala Li, Tsung Lin, David Fleet, and Geoffrey Hinton, A unified sequence interface for vision tasks. arXiv preprint arXiv:2206. 07669, 2022. 4, 8
[9] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zimick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: arXiv:1504.000 1504.0035,, 2015, 4
[10] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, 7he Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation leamiig. In Computer ViCion-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XXX. Springer, 2020. 8
[11] Zhe Chen, Yuchen Duan, Wenhai Wang. Junjun He, Tong Lu, Jifeng Dai, and Qiao. Vision transformer adapter for dense predictions. In International Conference on Learning Representations, 2023.
[12] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and lanna-aa--l tasks via text generation. In International Conference on Machine Learning, 2021. 8
[13] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling fintetution-finetueed language models. arXiv preprint a2 1416, 2022. 4
[14] MOSS contributors. Moss. https: // giiggtttt...oo hot../.ccmm/ppeiiaaa///S.s.
2023. 3
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 2018
[16] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 6
[17] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. arXiv preprint 2xXiv:2211006::. 2022.
[18] Zhe Gan, Ven-Chun Chen, Linjie Li. Chen Zhu. Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems, 2020. 8
[19] Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems, arXiv preprint arXiv:2104.00743 2021. 2021, 4
10

---

[20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2017. 3,8
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016. 6, 7,9
[22] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 2021. 7
[23] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212. 2017, 2022. 4
[24] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, 2022. 1,2
[25] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- Mdetr-modulated modulated detection for end-to-end multi modal understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021. 8
[26] Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks, arXiv preprint arXiv:2211.0900 1.09808, 2022. 4, 4
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language- sanguag--mmag pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301 201.12230..1297, 2023, 4,5
[28] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0633 06355, 2023. 2,3,
[29] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision. Springer, 2014. 9
[30] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee, Visual instruction tuning. arXiv preprint vrXiv:2304.08455 2023. 4,
[31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, 2021. 6
[32] Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Interngpt: Solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint 2005:2200500562, 2023. 2, 3,5
[33] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206. 08916, 2022. 4
[34] OpenAL. Gpt-4 technical report. arXiv, 2023.
[35] TB OpenAl. Chatgpt: Optimizing language models for dialogue. OpenAl, 2022. 1,3
[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with
human feedback. Advances in Neural Information Processing Systems, 2022. 3,4
[37] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002. 8
[38] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding
by generative pre-training. 2018. 1,4
[39] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikow, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, al. A generalist agent. arXiv preprint arXiv:2005.06175. 20222067,72222.222...22
2022.
[40] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real time object detection with region proposal networks. In Advances in Neural Information Processing Systems, 2015.
11

---

[41] Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross. and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognittin,, 2017.
[42] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv prepriat 1...::2:33317888 2023. 2, 3,
[43] Weijie Su, Xizhou Zhu, Chenxin Tao, Lewei Lu, Bin Li, Gao Huang. Yu Qiao, Xiaogang Wang. Jic Zhou, and Jifeng Dai. Towards all oll-in-onc pre-training via maximizing multi-modal mutual information. In P Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
[44] Chenxin Tao, Xizhou Zhu, Gao Huang. Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning, arXiv preprint ar Arx:v:2206.0011 220.00204, 2022.
[45] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: strong, replicable cnttrruttion-oolowinn model. Stanford Center for Research on Foundation Models. hups://crfm. stanford. 011/2022//33///aappaa huml, 2023. 3. 6,7
[46] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Amm Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 1 13971, 2023. 3, 6, 7
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polasukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.6 6
[48] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 8
Consensus-based image description
[49] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Cc Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 2022. 1.4
[50] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. arXiv preprint arXiv:2202.030022 2022. 1,4 4
[51] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 1, 2,
7,8
[52] Wenhai Wang, Enze Xie, Xiang Li. Deng- Peng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media. 833):415-244, 2022.
[53] Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as foreign language: Beit pretraining for all vision and vision-language tasks, arXiv preprint arXiv:2208. 10442, 2022. 1, 1,2, 8
[54] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.024499 2022. 1,2
[55] Xinlong Wang, Wen Wang. Yue Cao. Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 2022. 4
[56] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. arXiv preprint arXiv:2304.0322 032844 2023. 2,4
[57] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint or Xiv:22 10560, 2022. 4,5
[58] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-contex instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705 2022. 4
12

---

[59] Jason Wei, Maarten Bosma, Vincent Y Zhao. Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M Dai, and Quoe V Le. Finetuned language models zero-shot leamers. arXiv preprint 4rXiv:2109.0652. 2021. 4
[60] Chenfei Wu. Shengming Yin, Weizhen Qi. Xiaodong Wang. Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint XXV:22303.06611, 2023. 2, 3.
[61] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang. Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, 2020. 5, 8
[62] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. arXiv preprint 1:Xiv:2303306677 2023. 4
[63] Zhengyuan Yang. Zhe Gan, Jianfeng Wang. Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-languae modeling. In European Conference on Computer Vision, pages 521-539. Springer, 2022. 4
[64] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unitying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision, 2022.
[65] Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action, arXiv preprint or Xix::23033.13333 2023. 2, 3,5
[66] Yuan Yao, Zhang. Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision- lision-language models. arXiv preprint arXiv:2109. 11797, 2021.
[67] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022.
[68] Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm- 130b; An open bilingual pre- pre-trained model. arXiv preprint arXiv:2210.02414, 2022. 3
[69] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diah, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint 2205::2005.01668 2022. 3
[70] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt scarch. arXiv preprint a:X6v:2206.04677,
2022. 1
[71] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhaseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint 2rXiv:2304.1033 2023.
[72] Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang. Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-mmo:: Learning sparse generalist models with conditional moes. arXiv preprint arXiv:2206.04672 2022. 1, 4, 8,
[73] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. International Conference on Learning Representations, 2021. 3, 6, 7,8
[74] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni- perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 2, 4, 5, 8,
13


Reviews:
[{"review_id": "rxYcxJnypb", "reviewer": "Reviewer_kyDb", "summary": "This paper construction a vision/language model by passing visual features into a LLM. They trained the model on several standard tasks as well object detection and referring expression by adding special location tokens to the LLM's vocabulary. The model features a text-guided image tokenizer and an efficient decoding approach when generating segmentation or bounding boxes.", "questions": "Did the authors try pre-training the model? Pre-training to initially learn the alignment for the vision/language component is common practice for these kinds of models.\n\nCan the model generalize to outputting bounding boxes for tasks other than refexp or object detection? For example for pointing VQA questions where you output a bounding to answer a question.\n\nIn section 3.4, what does \"...feed the tokens of structural output format as queries to the decoder\" mean? That they are used as the initial starting token? What happens if there are multiple objects so they are then multiple class and x1 coordinates to produce? Or if the model needs to interleave text with structured output like in Figure 2a?\n\nTable 1 is missing many models that achieve better scores, BEiT-3 is better at detection and captioning for example. I think they should be included for reference", "limitations": "I think the authors should at least note some of the potential issues with these kinds of models (bias, potential for abuse my generating misinformation, hallucination, ect.)", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The suggestion in section 3.4 seems like a nice way to avoid auto-regressive decoding when it is not needed, although the method was a bit hard to parse.\n- Showing that the approach of adapting visual features for LLM can work objet detection and segmentation is interesting, these tasks have been less well explored in this area.\n-  The quantitive examples in the appendix at least suggests the model has instruction following capabilities similar to other models tuned on the LLaVA dataset.", "weaknesses": "- The scores are decent but not amazing. Only slightly better than pix2seq if using the same backbone, and the CiDER scores are lower then relatively simple models like ClipClap or VL-T5. \n- The central idea is essentially following the pix2seq method combined with the now pretty well studied method of adapting visual features for a LLM approach, which makes sense but does not feel hugely novel to me. \n- The authors should consider evaluating follow the methodology of LLAVa given they are using that data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Details of the `output-format-as-query` decoding.**\n\n**A1:** We thank the reviewer's appreciation of our output-format-as-query design. In Common Questions Q1, we provide more details about output-format-as-query regarding the data construction, training, and inference process. Here, we answer your detailed questions about it:\n\n**Q1-1: What does \"...feed the tokens of structural output format as queries to the decoder\" mean? That they as the initial starting token?**\n\n**A1-1:** Yes, the parsed outputs are appended to the original user instructions, and the resulting instructions are then fed into the LLM-based decoder.\n\n**Q1-2: What happens if there are multiple objects so they are then multiple classes and x1 coordinates to produce? Or if the model needs to interleave text with structured output like in Figure 2a?**\n\n**A1-2:** For the perception tasks (e.g., detection), the output format consists of a string with 100 segments of \"<cls><x1><y1><x2><y2>\", which can accommodate multiple objects in the scene. For interleave text with structured output, our model naturally supports this, as constructed user instructions default to this format (see Common Questions Q1).\n\n\n**Q2: The scores are decent but not amazing.**\n\n**A2:** Besides performance scores, the primary objective of this work is to design a generalist vision model available for user-tailored open-ended tasks. However, there are a few issues that are beyond the scope of this paper: \n\n(1) Due to the shared weights of our generalist model, conflicts may arise among different tasks, potentially leading to lower performance compared to specialized or foundation models that follow the \"pre-train then fine-tune\" paradigm (e.g., Pix2Seq v1, BEiT-3);\n\n(2) We use LLaVA-Instruct-150K to preserve the language capability of LLMs during training, which tends to generate long and detailed captions. But these long captions are suboptimal for traditional BLEU and CIDEr metrics.\n\n\n**Q3: The central idea is essentially following the pix2seq method.**\n\n**A3:** We argue that our model significantly differs from Pix2Seq v1/v2 in terms of its ability to handle open-ended tasks, model architecture, and decoding process (See Common Questions Q2). We think the reason why the two models may seem similar is that both our model and the Pix2Seq series use coordinate discretization to model the perception task, but this is NOT the main contribution of this work.\n\n**Q4: The authors should consider evaluating following the methodology of LLaVA given they are using that data.**\n\n**A4:** The evaluating method in LLaVA is unstable as it is affected by the online version of GPT-4, which is a closed-source system and not available in all countries or regions. Furthermore, there is no widely recognized average standard for such evaluation at the moment. Therefore, we adopt a more stable and controllable evaluation approach in this work. We test the performance of VisionLLM by generalizing it to various representative visual perception and understanding tasks, and evaluate it with standard benchmarks. In addition, we also design variant evaluations based on standard benchmarks to examine the open-ended task ability of our models (see Table 2(a)(b), and Figure B and C in supplementary material).\n\n**Q5: Did the authors try pre-training the model? Pre-training to initially learn the alignment for the vision/language component is common practice for these kinds of models.**\n\n**A5:** The first training stage of VisionLLM involves open-ended detection tasks with random user instructions. This is a specific form of vision-language alignment. \n\n**Q6: Can the model be generalized to output bounding boxes for tasks other than refexp or object detection? For example, for pointing VQA questions where you output a bounding to answer a question.**\n\n**A6:** Yes, our model can be generalized to use bounding boxes to answer questions. For example, as shown in Figure 2(a), when you ask a question like \"What is the child eating?\" in the class set `<class>`, VisionLLM will predict the bounding box of the doughnut as the answer to this question. We present more examples in Figure I in the attached PDF file for rebuttal to show this feature of VisionLLM. \n\n**Q7: Table 1 missing many models that achieve better scores, BEiT-3 is better at detection and captioning for example. I think they should be included for reference.**\n\n**A7:** In Table 1, we have listed recently popular generalist models capable of handling various tasks using shared weights. Differently, BEiT-3 is a foundational model that utilizes additional decoders for fine-tuning, incorporating a range of specialized designs, which do not support open-ended tasks. We will add and discuss these works in our revised version. Thanks for your suggestion.\n\n**Q8: I think the authors should at least note some of the potential issues with these kinds of models (bias, potential for abuse by generating misinformation, hallucination, etc.**\n\n**A8:** Thank you for the suggestions! We will attempt to address this through some de-biasing methods before the model is released."}, {"review_id": "K9GJjm4nqp", "reviewer": "Reviewer_wdrB", "summary": "This work proposes VisionLLM, a unified framework for vision tasks and vision-language tasks, using natural language task prompts. It demonstrates capabilities in a good variety of tasks.", "questions": "How are image tokens $T$ and `<text>` passed into the LLM? On line 166 it says `<image>` and `<question>` are placeholder tokens for image tokens and question tokens. I imagine `<question>` will be replaced by the actual question inside the language instruction text (`<text>`), but will the `<image>` token be replaced by the image tokens $T$? Figure 3 seems to imply that $T$ is passed into the LLM separately from `<text>`. I think it's more flexible if all inputs are formulated into one piece of text to pass into the LLM, as one can directly extend it to use more than one images, etc.", "limitations": "NA.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is the first attempt to use natural language prompts for vision-centric tasks such as object detection and instance segmentation. I think this work is tackling an important problem.\n- The language-guided image tokenizer is a novel component to convert image into tokens guided by text.\n- VisionLLM treats image tokens the same way as text tokens, such that the entire task (including image and class list) can be encoded as one piece of text. This converts vision tasks into sequence generation problem handled by a LLM. (I'm actually not sure if this understanding is correct, so I ask this in the Questions section as well.)", "weaknesses": "The framework is not a simplistic one, consisting of various components and types of losses, e.g. using bipartite matching for one type of outputs. Representing image as a set of ${e_i, l_i}$ is also kind of specific.\n\nThe model can do VQA but I didn't see any VQA results in the paper.\n\nOverall my main complaint is that the framework and pipeline seem a bit complicated, but it is designed to incorporate a wide variety of tasks and seems effective at it. It is a good first attempt at using natural language and LLM to tackle vision tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Pipeline seems a bit complicated.**\n\n**A1:** We would like to recap the components of VisionLLM. It has two key components: the language-guided image tokenizer and the LLM-based decoder. They work together in the following way:\n\n(1) Firstly, the visual encoder extracts features from the image at different scales, and the text encoder obtains features from the language input. \n\n(2) Then, in the language-guided image tokenizer, the multi-scale image features interact with the language features to generate language-guided image tokens. Each token is represented by embedding and location information.\n\n(3) Finally, these image tokens replace the placeholder `<image>` in the language prompts.  The resulting language prompts are fed into the LLM-based decoder for open-ended tasks. \n\nIn addition, some specific designs, e.g., bipartite matching, representing images as a set of $(e_i, l_i)$, are useful to accelerate the convergence of VisionLLM. Generally speaking, all the components are closely interconnected and indispensable for VisionLLM. Building a generalist model for open-ended tasks is a complicated system engineering, and we will explore more simplified implementations in the future.\n\n**Q2: The model can do VQA but I didn't see any VQA results in the paper.**\n\n**A2:** For VQA tasks, we qualitatively showcase the performance of VisionLLM on complicated VQA scenarios, as shown in Figure 2(d) and Figure F in the supplementary material. There are two-fold reasons: \n\n(1) Our model was trained on the LLaVA-Instruct-150K dataset, which encourages the model to generate long and detailed answers for visual questions, as explained in L256-257. However, most existing VQA benchmarks expect short answers, which makes our model score low (VisionLLM-R50: 33.86 vqa-score on VQAv2 test-dev). It is unfair to compare our model with other models on these benchmarks. \n\n(2) As we discussed in the Q4 of Reviewer kyDb, the existing GPT-based evaluation methods (e.g., metrics in LLaVA) are unstable and are affected by the online closed-sourced model released by OpenAI. So there is currently no widely recognized standard for such VQA evaluation.\n\n**Q3: How are image tokens `<image>` and `<text>` passed into the LLM?**\n\n**A3:** The image tokens are directly placed at the placeholder `<image>` at the language prompt. We will make it clearer in our revised version.\n"}, {"review_id": "eTrq9jzBnx", "reviewer": "Reviewer_TifL", "summary": "This paper introduces VisionLLM, an instruction-following agent that can perform various vision-only (classification/detection/segmentation) and vision-language (captioning/VQA) tasks. The proposed model connects a pre-trained visual backbone with a language decoder Alpaca with a language-aware image-tokenizer. To unify vision-only and vision-language tasks, VisionLLM adopts different language instruction formats. Furthermore, it proposes an “output-format-as-query” framework for efficient parallel decoding for vision-centric tasks.", "questions": "I have some questions about technical details:\n- For detection/segmentation tasks, I am confused how the proposed “output-format-as-query” approach (L227-239) works with variable numbers of objects per image. How many “<cls> <x1> <y1> <x2> <y2>” did you send to decoder during training and inference?\n- How does the “output-format-as-query” approach avoid parallel decoding for image captioning (L232)? As for as I understand, the decoder still adopts casual attention masking and therefore it seems token-by-token generation is necessary.\n- The proposed system supports customization of number of points for segmentation tasks — does it require specific training paradigms, i.e., balancing the number of masks with different number of points?\n- Does your system generalize to unseen instructions during test time?\n- What is the sampling procedure for open-ended tasks? For example, do you use top-k/nucleus sampling?", "limitations": "No. Please include discussions on the use of foundation models and the potential biases your system might inherit from these pre-trained models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an impressive effort in developing a large decoder for vision-only and vision-language tasks, using state-of-the-art multimodal foundational models while treating images as foreign language. The technical details are comprehensive and sound. Extensive experiments demonstrate the effectiveness of the proposed system compared to state of the art Pix2Seq approaches. Ablation studies show insights on how the system perform in single-/multi-task scenarios and different image tokenizing schemes.", "weaknesses": "Even though the proposed system is technically sound, it is still quite complicated. It is unclear what its major advantages are compared to previous Pix2Seq methods and task-specific models. \n\nI have some doubts about the system design: \n- Is the pre-trained instruction-following LLM (Alpaca) crucial in your system design? Can the system perform as well with a (non-instruction-following) LLaMA? Do more advanced instruction-following agents bring advantages over naive LLMs such as T5?\n- One of the major disadvantages of using a large-scale pre-trained LLM (Alpaca) is the increased training and inference costs. Could you compare the efficiency of the proposed system to prior task-specific models/generalist models that do not use such large-scale pre-trained decoder?\n- Why not adopt a ViT-B visual encoder to facilitate comparison to other generalist models such as Uni-Perceiver and Pix2Seq (as in Table 1)?\n\nThe paper is overall well-written. I found one typo:\nL166: placeholdersok?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Is pre-trained instruction-following LLM (Alpaca) crucial in your system design?**\n\n**A1:** The instruction-following LLM is important for the convergence of VisionLLM. We have observed that Alpaca converges more easily than LLaMa. But the LLM is not limited to Alpaca, Flan-T5 (an instruction-following version model of T5) is also a good alternative.\n\n\n**Q2: Time cost analysis.**\n\n**A2:** Thanks for your suggestion. As shown in the following table, we compare the inference speed of Pix2Seq and VisionLLM. Specifically, we executed tests on a single A100 GPU, utilizing code and model weights from Pix2Seq's official repository.  For both methods, we set the batch size as 1 and the image size as 1024x1024.\n\nAs can be seen from the table, although VisionLLM is equipped with a large LLM-based decoder, its inference speed is faster than Pix2Seq. This shows that VisiomLLM has an acceptable inference speed thanks to the proposed output-format-as-query decoding. We will add time cost analysis in our revised version.\n\n| Method          | FPS       | Times per Image |\n| --------------- | --------- | --------------- |\n| VisionLLM-R50   | 5.1 img/s | 197.4 ms        |\n| Pix2Seq-R50     | 4.4 img/s | 227.3 ms        |\n| VisionLLM-ViT-B | 4.0 img/s | 251.7 ms        |\n| Pix2SeqV2-ViT-B | 3.4 img/s | 294.1 ms        |\n\n**Q3: Why not adopt a ViT-B visual encoder to facilitate comparison to other generalist models such as Uni-Perceiver and Pix2Seq (as in Table 1)?**\n\n**A3:** We evaluate VisionLLM on two representative backbones of different scales. ResNet-50 is the most representative common-scale backbone, while InternImage-H is a large-scale backbone with state-of-the-art performance. We provide additional experiments using ViT-B as a vision encoder, through which we reach similar conclusions as ResNet50 and InternImage. See Common Questions Q3 for more details.\n\n**Q4: How many “<cls> <x1> <y1> <x2> <y2>” did you send to the decoder during training and inference?**\n\n**A4:** During both the training and inference phases, we input 100 sets of \"<cls><x1><y1><x2><y2>\" to the decoder, generating 100 object predictions. Those predictions with higher confidence scores will be retained, adhering to a common practice of the object detection task. We will make it clearer in our revised version.\n\n**Q5: How does the “output-format-as-query” approach avoid parallel decoding for image captioning (L232)?**\n\n**A5:** Sorry for the misunderstanding. We uniform the output format for various types of tasks in the form of natural language tokens. However, different types of tasks use different output formats, as illustrated in Figure 4 of the main paper. \n\nFor perception tasks, we use \"<cls><x1><y1> ...\" as the output format, employing the “output-format-as-query” approach for parallel decoding. \n\nFor understanding tasks, such as image captioning and VQA, we use \"<bos>\" as the output format, following a token-by-token generation process. We will provide a more explicit explanation of this aspect in the revised version.\n\n**Q6: Does it require specific training paradigms?**\n\n**A6:** No, VisionLLM does not require specific training paradigms. Like those of LLMs, the task instructions are randomly changed in terms of task type, task target, and output format (including the number of points).\n\n**Q7: Does your system generalize to unseen instructions during test time?**\n\n**A7:** Of course, our system could generalize to unseen instructions during test time. We incorporate randomized task descriptions, diverse task output formats, and randomized object categories during training. As a result, the system exhibits robustness in the face of changes in instructions.\n\n**Q8: What is the sampling procedure for open-ended tasks?**\n\n**A8:** We employed top-1 sampling in our approach. Utilizing more intricate sampling techniques, such as top-k sampling, could potentially lead to improved performance."}, {"review_id": "eBQfvzRQxX", "reviewer": "Reviewer_87vj", "summary": "This work presents a LLM-based framework VisionLLM for vision-centric tasks. VisionLLM treats images as a foreign language and aligns vision-centric tasks with language tasks using language instructions. Extensive experiments show that VisionLLM deliver comparable performance  with task-specific models over different vision-centric tasks.", "questions": "Some questions have been listed in the weakness part. Other than that, more questions about the details are as follows：\n1. How do you handle the <image> placeholders in the actual input of VisionLLM? Do you replace them with actual image tokens in the latent space?\n2. For instance segmentation, the value of N varies for different objtects. How do you detetermine this value?\n3. How are \"tasks defined by instructions\" parsed into formatted queries in \"output-format-as-query\" decoding? By pre-defined templates for different task categories?", "limitations": "This paper tries to catch the takeoff of LLMs by building a LLM-based generalist model for vision tasks. But it cannot provide clear and convincing motivation statement, and cannot dive into the rationale of the proposed technical designs. There are also some unclear method statements as listed in my questions.\n\nOverall, the idea behind is straightforward. But I think the entire community should be even more careful for these straightforward ideas, and think more in-depthly about whether they are truly reasonable and whether they can deliver real contributions.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors demonstrate the feasibility of using large language models as visual decoders in which vision information is treated as a foreign language. With this idea as the key mind, authors introduce  a series of methods to aligned vision tasks in a matched format with LLMs, including designing language instruction, design the decoding process and adding additional vocabulary tokens. Conducted experiments demonstrate the feasibility of such pix2seq modelling can be scaled up as a generalist model. Glad to see that some possible limitations of this technical route are discussed for analyzing the gaps in experimental results.", "weaknesses": "This work follows the idea of pix2seq to build a generalist model and the authors believe that it is natural to do so, without providing a sufficient explanation of the motivation behind it. Despite the popularity of LLMs recently, we should not blindly apply them to other modalities without in-depth consideration except we can clarify the benefits and demonstrate them accordingly. Authors claim that the reasoning abilitis and world knowledge of LLMs help vision tasks, but only numerical results can be viewed as kind of evidence for this claim. It's not clear and convincing for me from two aspects:\n- Whether the reasoning abilities and world knowledge of LLMs truly help? and how?\n- How about scaling up task-specific vision models when compared to VisionLLM? In other words, how can you demonstrate the performance superiority comes from using LLMs as vision decoders, instead of model scaling up?\n\nThere are a lot of teachnical designs introduced to model vision-centric tasks in a language-matched format. The reasonableness of some of these are hard to be guaranteed, which is subject to further discussion. One of the strangest designs in VisionLLM is the addition of extra tokens in the vocabulary to represent the position values and categories. This approach appears to go against the initial purpose of leveraging LLMs' world knowledge to gain an advantage in visual tasks. Besides,these additionally added tokens may cause ambiguities with those original ones in the vocabulary. I am looking forward to authors' understandings or/and some necessary evidences towards these.\n\nThe effevetiveness of VisionLLM is sorely evaluated on ResNet-50 and Intern-H backbones. The general applicability is subject to further evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: This work follows the idea of pix2seq to build a generlist (generalist) model and the authors believe that it is natural to do so, without providing a sufficient explanation of the motivation behind it.**\n\n**A1:** We argue that _**VisionLLM is not the scaling up of Pix2Seq**_. Although both models incorporate coordinate discretization for object detection, they differ significantly in task generality, model design, and decoding process, as explained in Common Questions Q2. Moreover, directly scaling up Pix2Seq cannot achieve an open-ended task model. Such a model would not converge, as we discussed in L130-135 of the supplementary material and Table A(d).\n\nFor the motivation behind this work, please note that this work focuses on open-ended vision tasks that can be customized by users according to their needs. We have summarized the limitations of the existing paradigms for open-ended tasks in Figure 1 and the first two paragraphs of the Introduction, which inspire us and shape the main objective of our work. To achieve open-ended tasks, we tackle the challenge of aligning vision tasks and LLMs from various aspects including: instruction design, model design, and training. We have explained the motivation for each design choice in the first paragraph of each subsection of the method section.\n\n**Q2: Whether the reasonding (reasoning) abilities and world knowledge of LLMs truely (truly) help? and how?**\n\n**A2:** LLM plays a crucial role in our open-ended task framework, for the following reasons:\n\n(1) LLM can parse instructions, which is a key feature of our system. As we explained in Reviewer TifL's Q1, our system cannot converge without instruct-tuned LLM. Furthermore, instruction parsing enables models to comprehend the target object mapping and the output format of the perception tasks. LLM is the only model that possesses this feature, as it is pre-trained on a large corpus of user instruction and code data.\n\n(2) LLM also facilitates image description with controllable text length and visual question answering with complex reasoning, as illustrated in Figure 2(c)(d), and Figure F in supplementary material. These tasks require the capabilities of instruct-following and relation reasoning among objects. These capabilities are not learned from visual data and models, but from LLMs pre-trained on web-scale NLP data, as demonstrated by [1].\n\n**Q3: How about scaling up task-specific vision models when compared to VisionLLM? In other words, how can you demonstrate the performance superiority (that) comes from using LLMs as vision decoders, instead of model scalling (scaling) up?**\n\n**A3:** This is a common misconception. Model scaling up could not lead to open-ended task capability. Large-scale vision models like ViT-22B, Swin-G, and InternImage-G are still limited by tasks in specific formats. They cannot handle vision tasks with different or unknown formats. _**So even if we scale up task-specific vision models, they are still not comparable to our model in terms of open-ended tasks.**_ On the contrary, LLM pre-trained on web-scale corpus is proved to effectively understand the user instructions and provide reasonable answers, which is important for VisionLLM. The benefits of LLM for our model can refer to Reviewer 87vj's Q2.\n\n**Q4: One of the strangest designs in VisionLLM is the addition of extra tokens in the vocabulary to represent the position values and categories. Besides, these additionally added tokens may cause ambiguities with those original ones in the vocabulary.**\n\n**A4:** _**Extra tokens are a common way to extend the capabilities of an LLM**_, especially when supporting a new language and task [2]. To support vision tasks, the original vocabulary of LLM is not enough (see L210-213), it is necessary to increase the token size and ensure alignment during training. These tokens will be aligned with the LLM by instruction tuning. _**They will not conflict with the original tokens**_, because in our constructed training data, the newly added tokens are only used for vision tasks without any overlapped semantics with the original tokens.\n\n**Q5: The effectiveness of VisionLLM is sorely evaluated on ResNet-50 and Intern-H backbones. The general applicability is subject to further evaluation.**\n\n**A5:** We evaluate VisionLLM on two representative backbones of different scales. ResNet-50 is the most representative common-scale backbone, while InternImage-H is a large-scale backbone with state-of-the-art performance. We provide additional experiments using ViT-B as a vision encoder, through which we reach similar conclusions as ResNet-50 and InternImage. See Common Questions Q3 for more details.\n\n**Q6: How do you handle the <image> placeholders in the actual input of VisionLLM? Do you replace them with actual image tokens in the latent space?**\n\nA6: Yes, for `<image>`, we replace it with image tokens in the latent space.\n\n**Q7: For instance segmentation, the value of N varies for different objects. How do you determine this value?**\n\n**A7:** During training, the number of points varies randomly with the language instruction. During inference, the number of points N is specified by the user in his/her instructions.\n\n**Q8: How are \"tasks defined by instructions\" parsed into formated (formatted) queries in \"output-format-as-query\" decoding? By pre-defined templates for different task categories?**\n\n**A8:** See details of the output-format-as-query decoding in Common Questions Q1. We introduce the data construction, training, and inference details of the decoding process.\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in Neural Information Processing Systems 35 (2022): 27730-27744.\n\n[2] Schick, Timo, et al. \"Toolformer: Language models can teach themselves to use tools.\" arXiv preprint arXiv:2302.04761 (2023).\n"}, {"review_id": "hk8PdHE5oZ", "reviewer": "Reviewer_eUTR", "summary": "The paper presents a new vision-language model where tasks are specified via a language interface instead of being “hard-coded” into the architecture. The model relies on a pretrained decoder model to simply output the answer as text. E.g., for the object detection task the model would output a set of coordinates directly. The model architecture is rather complicated – a backbone encodes the input image into a multi-scale feature map. The language information is encoded by a Bert model and injected into the multi-scale feature map via cross-attent. Then a DETR model produces a set of visual “tokens” from the feature map. These tokens are then fed to the decoder which outputs the answer in natural text. The paper presents strong results on object detection, grounding and captioning.", "questions": "1. Could you provide pseudo-code for your implementations? E.g. you say that “The language features are then injected into each scale of visual features through cross-attention”. Knowing exactly how the cross-attention is implemented would be good.\n\n2. What is output-format-as-decoding? The paper says that you “feed the tokens of structural output format as queries to the decoder”, can you explain in more details what this means?\n\n3. The paper says “except a few LoRA parameters” – can you specify how many?\n\n4. Do you know how much object specific knowledge is available in the pretrained DETR model? Is it possible to do an ablation here? I assume the DETR model has been pretrained on e.g. COCO detection, so it’s maybe not really a zero-shot task for the model?\n\n5. Why is two-stage training needed? Are there ablations showing the effects of this?\n\n6. Are the resnet parameters pretrained? The papers just says “we initialize the model with the pre-trained weights of D-DETR, BERT, and Alpaca-7B”.\n", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper provides very strong empirical results. Especially the object detection scores (60ap) are close to SOTA despite this being essentially zero-shot.\n* The paper is relatively easy to train -- it relies on LORA and pre-trained models. This and the open-sourced code should make the method available to other researchers.\n* Decoding the output in natural text is a good research direction, it is much more flexible than visual prompt tuning and predefined formats.\n", "weaknesses": "\n* The method is rather complicated. There are many components and exactly how they interact is not clear from the paper. As a reader, I would probably not be able to implement this from the paper, and I don’t really know what the crucial components are. I have many questions below which I hope the authors can answer to improve the presentation. E.g. the output-format-as-decoding method is not clearly described.\n\n* There are also VL tasks (e.g. VQA) which I think should be added to the main paper. Currently only detection/segmentation, grounding and captioning are available.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: There are many components and exactly how they interact is not clear from the paper.**\n\n**A1:** There are only two main components in the VisionLLM: language-guided image tokenizer and LLM-based decoder, each of which has specific designs for open-ended tasks. We kindly invite the reviewer to see Reviewer wdrB's Q1 for more clarification and detailed interaction among components. We will make it clearer in our revised version.\n\n**Q2: Details of `output-format-as-query` decoding.**\n\n**A2:** Thanks for your good question. Since it was also raised by other reviewers, we provide details about output-format-as-query in terms of data construction, training, and inference process in Common Questions Q1. We will make it clearer in our revised version.\n\n**Q3: There are also VL tasks (e.g. VQA) which I think should be added to the main paper.**\n\n**A3:** For VQA tasks, we qualitatively showcase the performance of VisionLLM on complicated VQA scenarios, as shown in Figure 2(d) and Figure F in the supplementary material. The reason is that conventional VQA metrics are unsuitable for measuring long and detailed answers (see L259-260), and the current GPT-based evaluation is unstable enough, see Reviewer wdrB's Q2 for more details.\n\n**Q4: The language features are then injected into each scale of visual features through cross-attention.**\n\n**A4:** Thanks for your careful review. Here we provide the pseudo-code of the cross-attention layer. In this code, we inject text features into each scale of visual features through cross-attention.\n\n```python\ndef vision_language_fusion(img_features, text_features, text_masks):\n    outs = []\n    for img_feature in img_features:\n        img_feature = rearrange(img_feature, 'b c h w -> b (h w) c')\n        img_feature = cross_attention(src=img_feature, ref=text_features, key_padding_mask=text_masks)\n        img_feature = rearrange(img_feature, 'b (h w) c -> b c h w')\n    outs.append(out)\n    return outs\n```\n\n**Q5: the paper says \"except a few LoRA parameters\" – can you specify how many?**\n\n**A5:** We set the LoRA rank to 64 and use LoRA on the QKVO (Query, Key, Value, and Output) in the attention layers, resulting in approximately 0.9% of trainable parameters. We will clarify this in the revised version.\n\n**Q6: Do you know how much object specific knowledge is available in the pretrained DETR model? Is it possible to do an ablation here? I assume the DETR model has been pretrained on e.g. COCO detection, so it’s maybe not really a zero-shot task for the model?**\n\n**A6:** The determination of whether it is a zero-shot scenario depends on the training and test samples. As Deformable DETR model trained on the COCO dataset acquires knowledge specific to objects, it cannot be truly considered as achieving zero-shot performance for COCO evaluation. Nonetheless, it is important to note that VisionLLM is specifically designed for open-ended tasks. During testing, _**we employ instruction descriptions with arbitrary object categories, arbitrary task descriptions, and output formats, which are unseen during training**_. By leveraging the vast world knowledge embedded in a pre-trained LLM, we have observed exceptional performance of our model in comprehending and effectively handling previously unseen categories, such as \"red gamepad\" (see Figure 1(a)), which are not included in the COCO categories.\n\n**Q7: Why is two-stage training needed? Are there ablations showing the effects of this?**\n\n**A7:** Thanks for your careful review. As depicted in Figure A in the supplementary material, we adopted a two-stage training approach to expedite the convergence of VisionLLM. Our experiments showed that the two-stage training, starting from easy to hard tasks, resulted in faster convergence than a single-stage training approach.\n\n**Q8: Are the ResNet parameters pretrained? The paper just says “we initialize the model with the pre-trained weights of D-DETR, BERT, and Alpaca-7B”.** \n\n**A8:** Yes, the ResNet parameters are loaded from the pre-trained weights of Deformable DETR."}]

Author rebuttal:
Dear all reviewers:

We sincerely appreciate the reviewers for their time and effort in the review. This submission received 5 review comments, and 4 reviewers gave positive scores. We first address some common questions, followed by detailed responses to each reviewer separately. We hope our responses could clarify existing doubts.

&emsp;

###  Common Questions
**Q1: Details of the `output-format-as-query` decoding.**

**A1:** The `output-format-as-query` decoding technique is designed to parse the standard output format, which is compatible with the LLM-based decoder, from user instructions. The details are as follows:

**Data Construction:** Following self-instruct [1], we create various user instructions for each task to simulate human interaction. Here are some examples:
```
System message: 
"You are an AI assistant for translating the user instructions to the standard prompt. Please help me parse the following input.
Input: {input}
Output:"

# Object detection
Input: 
"The image is: <image>. Please thoroughly examine the image and detect all objects belonging to the category set {'person': <c0>, 'bicycle': <c1>, 'car': <c2>, 'motorcycle': <c3>}."
Output:
"The bounding boxes are <cls><x1><y1><x2><y2><cls><x1><y1><x2><y2>...<cls><x1><y1><x2><y2>."

# Image caption
Input:
"The image is: <image>. Please write a short caption for this image."
Output:
"The image shows that <bos>"
```

**Training:** After obtaining the data of user instructions, we finetune Alpaca using the next token prediction task for supervision, making it able to accomplish the output format parsing process.

**Inference:** As described in L229-236 and Figure 4, the inference process involves the following steps:

(1) We first use the fine-tuned Alpaca to parse the user instructions into standard output formats for different tasks. For instance, in the case of object detection, the output format may be "The bounding boxes are <cls><x1><y1><x2><y2><cls><x1><y1><x2><y2>...<cls><x1><y1><x2><y2>". For image captioning, the output format could be "The image shows that <bos>".

(2) The parsed outputs are then appended to the original user instructions as suffix texts. The extended instructions are fed into the LLM-based decoder as queries.

(3) Since the output format contains special tokens, such as <cls>, <x1>, <y1>, <x2>, <y2>, and <bos>, by treating these tokens as queries, the LLM-based decoder can predict the corresponding results. This approach enables the detection task to run in parallel like the cloze task, while the captioning task remains the next token prediction.

We will make it clearer in our revised version. 

&emsp;

**Q2: The difference between VisionLLM and Pix2Seq.**

**A2:** Although both VisionLLM and Pix2Seq v1/v2 employ coordinate discretization for object detection tasks, _**they differ significantly in terms of task generality, model design, and decoding process**_.

**Task Generality:**  VisionLLM allows users to customize vision tasks using language instructions, supporting user-tailored output formats, task targets, task descriptions, etc.  In contrast, Pix2Seq v1 is a special model for object detection, and  Pix2Seq v2 only supports pre-defined task switching with learnable prompt tokens, lacking the flexibility of task customization.

**Model Design:** VisionLLM consists of a series of careful designs for open-ended tasks, including (1) language instructions that align vision tasks with NLP tasks; (2) a flexible tokenizer guided by natural language instructions (Pix2Seq v2 uses _**unreadable embedding**_ for task switching); and (3) an open-ended task decoder based on LLMs along with an improved decoding process.

**Decoding Process:** Pix2Seq struggles to converge in open-ended task scenarios with random user instructions (see Table A(d) in supplementary material). VisionLLM solves this problem effectively by using its output-format-as-query approach, which enables the model to work with the Hungarian matching loss and handle highly random open-ended task instructions efficiently.

To sum up, VisionLLM and Pix2Seq are distinct models. Pix2Seq is a pioneering generalist model but has limitations (see Figure 1(a)). VisionLLM explores new possibilities for end-to-end models that unify vision and language tasks in the LLM era. We will clarify this in the revised version.

&emsp;

**Q3: Evaluation VisionLLM on more backbones.**

**A3:** We chose to evaluate our model on ResNet-50 and InternImage-H backbones because ResNet-50 is widely recognized as a representative backbone at a common scale, and InternImage-H is known for its large-scale backbone with top-notch performance. Results on ResNet-50 and InternImage-H demonstrate the generality of VisionLLM on backbones at different parameter scales. In the table below, we have included the results of ViT-B, which still meet our expectations.

| Method | Backbone | Open-Ended |  | Detection |  |  | Instance Seg. |  | Grounding | Captioning |   |
| - | - | - | - | - | - | - | - | - | - | - | - |
|  |  |  | AP | AP50 | AP75 | AP | AP50 | AP75 | P\@0.5 | BLEU-4 | CIDEr |
| Uni-Perceiver | ViT-B | - | - | - | - | - | - | - | - | 32.0 | - |
| Uni-Perceiver-MoE | ViT-B | - | - | - | - | - | - | - | - | 33.2 | - |
| Uni-Perceiver-V2  | Swin-B | -  | 58.6 | - | - | 50.6 | - | -  | - | 35.4 | 116.9 |
| Pix2Seq v2 | ViT-B | - | 46.5 | -  | - | 38.2 | - | - | - | 34.9 | - |
| VisionLLM-R50 | ResNet-50 | ✓  | 44.6 | 64.0  | 48.1 | 25.1 | 50.0 | 22.4 | 80.6  | 31.0 | 112.5 |
| VisionLLM-ViT-B | ViT-B | ✓ | 47.3 | 68.6 | 51.4 | 26.8 | 57.7 | 22.6 | 81.3  | 31.5  | 113.1 |
| VIsionLLM-H | Intern-H | ✓ | 60.2 | 79.3 | 65.8 | 30.6 | 61.2 | 27.6 | 86.7  | 32.1 | 114.2 |


[1] Wang, Yizhong, et al. "Self-instruct: Aligning language model with self-generated instructions." arXiv preprint arXiv:2212.10560 (2022).


Meta-review: While initially there were mixed ratings, there was an effective rebuttal and five knowledgeable reviewers recommended accepting the paper: Weak Accept, Accept, Weak Accept, Borderline Accept, and Borderline Accept.  The paper presents an effective framework that provides a unified perspective for vision and language tasks treating images as a visual language and aligning vision-centric tasks. The authors build on top of existing models but extend them to facilitate predicting boxes which improves the explainability perspective and augments foundation models with grounding capabilities.  Based on the clear positive consensus, it is our recommendation to accept the paper. No basis to overturn the reviews.  Authors should attend to the main points in the reviews. when preparing a final version. No basis to overturn the reviews.
Decision: Accept (poster)