Title: VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large Vision-Language Model

Detection

Image Caption
Abstract: Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over 60% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The code shall be released.

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Abstract

Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLMbased framework, our model can achieve over 60% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The code shall be released.

1 Introduction

The emergence of large language models (LLMs) like ChatGPT [35] has revolutionized the landscape of artificial general intelligence (AGI), showcasing their impressive zero-shot capabilities in addressing various natural language processing (NLP) tasks through user-tailored prompts or language instructions. Despite these advancements, it's essential to note that the triumph of LLMs does not effortlessly extend to pure vision and vision-language tasks, due to the inherent disparities between modalities and task formats.

The field of computer vision presents a unique set of challenges and paradigms that differ from those of NLP. The traditional paradigm of vision foundation models is pre-training followed by fine-tuning [51, 11, 43, 53, 17, 44], which is effective but comes with significant marginal costs when adapting to diverse downstream scenarios. As shown in Figure 1a, while approaches such as multi-task unification [38, 50, 1, 49, 72] have been used to achieve generalist capability, they often struggle to overcome the limitations imposed by pre-defined tasks, resulting in a gap in open-ended task capabilities compared to LLMs. Recently, visual prompt tuning [24, 66, 70, 67, 54] has emerged as a way to flexibly outline some pure vision tasks (see Figure 1b), such as object detection, instance

Vision

Pre-defined tasks:

detection, captioning,

VQA, grounding, ...

(a) Vision generalist models [51, 53, 74] are constrained by the for- mat of pre-defined tasks.

Visual

(b) Visual prompt tuning [24, 56,

54] are inconsistent with the for-

mat of LLMs.

Vision + LLM

Task defined

by  instruc6ons

Desired output:

<c1> <p1> <p3> ...

segmentation, and pose estimation, using visual masking. However, the format of visual prompts considerably deviates from that of language instructions, making it challenging to directly apply the reasoning abilities and world knowledge of LLMs to vision tasks. Therefore, there is an urgent need for a unified generalist framework that can seamlessly integrate the strengths of LLMs with the specific requirements of vision-centric tasks.

In this work, we present VisionLLM, a novel framework that aligns the definitions of vision-centric tasks with the methodologies of LLMs. Leveraging the reasoning and parsing capacities of LLMs, VisionLLM is designed to empower open-ended task capabilities for vision-centric tasks. Specifically, it comprises three core components: (1) a unified language instruction designed for vision and vision-language tasks, (2) a language-guided image tokenizer, and (3) an LLM-based open-ended task decoder that orchestrates various tasks using language instructions. With this framework, a wide range of vision-centric tasks can be seamlessly integrated, including object detection, instance segmentation, image captioning, and visual grounding. In addition, the framework also facilitates task customization at different levels of granularity, allowing for the customization of target objects, output formats, task descriptions, etc.

Compared to current popular API-based applications [60, 65, 42, 32, 28], our model takes a unified, end-to-end approach to integrate VFMs and LLMs, streamlining and enhancing the overall efficiency of the overall process, and leveraging the strengths and data of both VFMs and LLMs within a single, cohesive system. Furthermore, our model surpasses the limitations of generalist vision models pre-trained on pre-defined tasks. VisionLLM can effectively manage vision-centric tasks through language instructions, embodying a flexible and open-ended approach that is not constrained by pre-set tasks. This versatility makes VisionLLM a robust and powerful generalist model for vision and vision-language tasks, opening up new possibilities for the development of unified generalist models that bridge the domains of vision and language.

In summary, our main contributions are as follows:

We propose VisionLLM, the first framework that leverages the power of LLMs to address visioncentric tasks in an open-ended and customizable manner. By aligning the definitions of vision-centric tasks with LLM methodologies, VisionLLM breaks new ground in enabling the unified modeling of vision and language, opening up possibilities for advancing the field.

We overcome many difficulties when porting LLMs to vision-centric tasks, by designing unified language instruction that matches the format of language models and covers various vision-centric tasks including visual perception. Correspondingly, we develop a language-guided image tokenizer and an LLM-based task decoder that can handle open-ended tasks according to the given language instructions based on the LLMs' reasoning and parsing capabilities.

We construct a series of tasks with different granularities to verify the effectiveness of our models, ranging from easy to hard, and from pre-defined to flexible. Through these validations, we demonstrate the remarkable generality of our models, showcasing their ability to handle diverse

Human: ' Identify the objects in <image> that belong to {'What is the child eating?': <c0> , 'red gamepad': <c1> } and draw a bounding box around each one. The output should be a list of tuples in

<c1>

<c0>

the format (c, x1, y1, x2, y2), where c is the class label and x1, y1, x2, y2 are the coordinates of the top-left and bottom-right corners of the bounding box. The coordinates should be within the range 1024 ."

VisionLLM: " The bounding boxes are [( <c0> , 226.4, 229.8, 363.1, 347.4), ( <c1> , 441.1, 183.5, 538.6, 269.9)] ."

(a) Object Detection with Customized Class Set

Human: " Provide a long description of the <image> . "

VisionLLM:  " The image features a white dog laying down on a foldable outdoor cooling pad ,

which is placed on top of a wooden table. The dog has its head in the middle of the cooling pad, and a fan is positioned behind it, blowing air onto the pad. The table is outdoors, in a grassy area, and there are two chairs placed around it, one on each side. The scene appears to be set in a sunny weather, with a bright and sunny sky, which contributes to the overall outdoor atmosphere ."

(c) Image Description with Controllable Text Length

Human: " For each object in <image> that is a member of {'motorbike': <c0> }, output a tuple with the class label and the coordinates of a polygon with 16 points that

<c0>

encloses the object.  The coordinates should be within the range 1024. The output format should be (c, x1, y1, x2, y2, ..., x16, y16) ."

VisionLLM: " The detected polygons are [( <c0> , 135.3, 95.7, 123.4, 53.4, 84.9, 57.6, 66.8, 60.5, 60.1, 72.3, 34.2, 71.4, ..., 124.9, 119.3)] ."

(b) Object Location with Editable Output Format

Human: " With the objects in the <image> , if you feel hot, what will you do? "

VisionLLM: " First, You can also try turning off the computer and other electrical appliances in the

room to reduce the heat generated by their operation. As an alternative, you can use the desk fan provided in the image to create a cooling effect in the room. You can plug in the fan and adjust its settings to ensure it blows air in a direction that reaches the room's occupants effectively. "

Figure 2: Results and visualizations of our VisionLLM . Guided by language instructions, our unified generalist framework showcases its effectiveness on diverse open-ended vision-centric tasks. The text marked with a gray background indicates the customized instructions and the desired outputs.

scenarios, including random object categories, random output formats, and random task descriptions, as shown in Figure 2. The successful outcomes of these validations underscore the tremendous potential of our model in harnessing the capabilities of LLMs to control and guide vision-centric tasks. In addition, with a generalist LLM-based framework, our model also yields promising results on various vision-centric tasks. Notably, our generalist model achieves an impressive mAP score of 60+% on the COCO dataset, surpassing many detection-specific models [73, 6, 20] and approaching the state-of-the-art record.

2 Related Work

2.1 Large Language Model

Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP) and artificial general intelligence (AGI), due to their impressive capabilities in language generation, in-context learning, world knowledge, and reasoning. The GPT family, including GPT-3 [5], ChatGPT [35], GPT-4 [34], and InstructGPT [36] are most representative works of LLMs. Other LLMs like OPT [69], LLaMA [46], MOSS [14], and GLM [68] have also made substantial contributions to the field. These models achieve high performance and are open-sourced, serving as valuable resources for training large models and as foundations for further fine-tuning for specific purposes. For instance, Alpaca [45] introduces a self-instruct framework that facilitates instruction tuning of the LLaMA model, reducing the reliance on human-written instruction data. Recently, the emergence of these LLMs has also opened up API-based applications for solving vision-centric tasks. These applications have integrated visual APIs with language models to enable decision-making or planning based on visual information, such as Visual ChatGPT [60], MM-REACT [65], HuggingGPT [42], InternGPT [32], and VideoChat [28]. However, despite the convenience of using language-based instructions to define tasks and describe visual elements, these interactive systems [60, 65, 42, 32, 28] still face limitations in capturing fine-grained visual details and understanding complex visual contexts, which hinder their ability to effectively connecting vision

and language models. In summary, while LLMs have shown tremendous potential in various NLP applications, their applicability to vision-centric tasks has been limited by the challenges posed by modalities and task formats.

2.2 Vision Generalist Model

The pursuit of generalist models [74, 33, 62], which aim to handle a wide range of tasks using a shared architecture and parameters, has been a long-standing goal in the machine learning community. Inspired by the success of sequence-to-sequence (seq2seq) models in the field of NLP [38], recent advancements such as OFA [50], Flamingo [1], and GIT [49] propose modeling diverse tasks as sequence generation tasks. Unified-IO [33], Pix2Seq v2 [8], and UniTab [63] extend this idea by using discrete coordinate tokens to encode and decode spatial information for more tasks. Gato [39] also incorporates reinforcement learning tasks into the seq2seq framework, while GPV [19] develops a general-purpose vision system by combining a seq2seq module with a DETR-based visual encoder [6]. However, these methods suffer from some limitations, such as slow inference speed and performance degradation due to the non-parallel auto-regressive decoding process. Uni-Perceivers [74, 72, 26] solve these issues by unifying different tasks using the maximum likelihood target for each input based on representation similarity, regardless of their modality, making it possible to support both generation and non-generation tasks in a unified framework. Nevertheless, these generalist models are still restricted by pre-defined tasks and cannot support flexible open-ended task customization based on language instructions like LLMs.

2.3 Instruction Tuning

Language instructions are a powerful way to express various NLP tasks and examples for LLMs, as introduced by GPT-3 [5]. Following this idea, subsequent works, such as InstructGPT [36], FLAN [13, 59], and OPT-IML [23], explore the instruction-tuning method [58, 57] and demonstrate that this simple approach effectively enhances the zero-shot and few-shot capabilities of LLMs. The language instruction paradigm has also been adopted by the computer vision community to define image-to-text tasks. Flamingo [1] is a milestone work that uses vision and language inputs as prompts and achieves remarkable few-shot results in various vision-language tasks, such as image captioning [9] and VQA [2]. BLIP-2 [27] further connects the visual encoder with LLMs through a querying transformer and a linear projection layer to build strong multimodal models. MiniGPT-4 [71] and LLaVA [30] finetune the BLIP-2-style models on synthetic multimodal instruction-following data to unleash the potential of LLMs. However, these models mainly focus on image-to-text tasks and fail to address visual perception, such as object detection, instance segmentation, pose estimation, etc. To tackle image inpainting tasks, Bar et al. [3] introduces the first visual prompting framework that utilizes inpainting with discrete tokens on images. Painter [55] and SegGPT [56] employ masked image modeling on raw pixels for in-context learning with paired images. While these visual prompt models demonstrate good results in segmentation tasks, their applicability to numerous real-world vision tasks is challenging. Moreover, defining the visual prompts as image inpainting is inconsistent with the language instructions in LLMs, hard to leverage the reasoning, parsing ability, and world knowledge of LLMs. In this work, we aim to align vision-centric tasks with language tasks, use language instructions to unifiedly and flexibly define all tasks, and solve them with a shared LLM-based task decoder.

3 VisionLLM

3.1 Overall Architecture

This work targets to provide a unified generalist framework that can seamlessly integrate the strengths of large language models (LLMs) with the specific requirements of vision-centric tasks. As shown in Figure 3, the overall architecture of VisionLLM consists of three key designs: (1) a unified language instruction that provides a consistent interface for vision-centric task definition and customization; (2) a language-guided image tokenizer, which encodes visual information in alignment with the given language prompt, enabling the model to comprehend and parse the visual content effectively; and (3) an LLM-based open-task decoder, which utilizes the encoded visual information and language instructions to generate satisfactory predictions or outputs. The three designs work together to achieve a flexible and open-ended framework that can handle various vision-centric tasks at different levels of task customization through language instructions.

Language-Guided

Image Token

...

<text>

<text>

Desired Output:

<c1> <p1> <p3> ...

Open-Ended Task

Decoder with LLM

Language-Guided

Image Tokenizer

Backbone

...

Vision-language example: " Describe the image <image> in details. "

Vision-only example: " For each object in image <image> that is a member of class set <class> , output a tuple with the class label and the coordinates of a polygon with 16 points that encloses the object. The coordinates should be within range <range> . The output format should be (c, x1, y1, ...). "

Language Instructions

<text>

𝐹

!

𝐹

"

𝑇

Different from previous interactive systems [60, 65, 42, 32, 28] that rely on APIs, our VisionLLM presents a more flexible and end-to-end pipeline. Given language instructions that describe the current tasks and an input image, the model first uses a language-guided image tokenizer to encode the image tokens based on the given prompt. Then, the image tokens and language instructions are fed to an LLM-based open-ended task decoder. Finally, it evaluates the generated outputs against the task definition given by the unified language instructions, enabling the model to produce task-specific results. This seamless, end-to-end pipeline enables VisionLLM to effectively combine vision and language, achieving remarkable performance in open-ended and customizable vision-centric tasks.

3.2 Unified Language Instruction

We first introduce unified language instructions to describe vision-centric tasks. This design enables the unification of various vision-only and vision-language task descriptions and allows for flexible task customization.

Vision-Language Tasks. The instructions for vision-language tasks such as image captioning and visual question answering (VQA) are straightforward and similar to NLP tasks. Following previous methods [27, 74, 30], we describe the image captioning task like ' The image is <image> . Please generate a caption for the image: ', and the VQA task like ' The image is <image> . Please generate an answer for the image according to the question: <question> '. Here, <image> and <question> are the placeholders of the image tokens and the question, respectively. The image tokens are directly placed at the placeholder <image> .

Vision-Only Tasks. Designing effective language instructions for vision tasks is a challenging endeavor due to the differences in modality and task format between vision and language. Here, we describe vision tasks by providing a task description and specifying the desired output format via language instructions.

The task description conveys the intended task to the language model. Following self-instruct [57], we design a set of seed instructions with placeholders and employ LLMs to generate a large number of related task descriptions and randomly select one of them during training.

For conventional visual perception tasks like object detection and instance segmentation, we propose a unified output format represented as a tuple ( C, P ) , where C denotes the class index in the category set <class> , and P = { x i , y i } N i =1 represents N points that locate the object. To align with the format of word tokens, both the class index C and the coordinates of points x i , y i are transformed into discretized tokens. Specifically, the class index is an integer starting from 0, and the continuous coordinates of the points are uniformly discretized into an integer within the range [-<range> , <range> ]. For object detection and visual grounding tasks, the point number N is equal to 2, representing the the top-left and bottom-right points of object's bounding box. In the case of instance segmentation, we employ multiple ( N> 8 ) points along the object boundary to represent an instance mask [61]. Other perception tasks such as pose estimation (keypoint detection) can also be formulated as language instructions in this way.

An example of language instruction for the instance segmentation task is as follows: ' Segment all the objects of category set <class> within the <range> of the image and generate a list of the format (c, x1, y1, x2, y2, ..., x8, y8). Here, c represents the index of the class label starting from 0, and (x1, y1, x2, y2, ..., x8, y8) correspond to the offsets of boundary points of the object relative to the center point. The image is: <image> '.

3.3 Language-Guided Image Tokenizer

VisionLLM considers images as a kind of foreign language and converts them into token representations. Unlike previous works [16, 52, 31] that utilize fixed-size patch embeddings to represent images, we introduce the language-guided image tokenizer to flexibly encode visual information that aligns with task-specific language prompts or instructions.

Specifically, give an image X ∈ R H × W × 3 with height H and width W , we first feed it to the image backbones ( e.g. , ResNet [21]) and extract visual features F v of four different scales. Additionally, we leverage a text encoder ( e.g. , BERT [15]) to extract the language features F l from given prompts. The language features are then injected into each scale of visual features through crossattention [47], yielding multi-scale language-aware visual features, enabling the alignment of features across modalities.

Afterward, we propose to adopt a transformer-based network ( e.g. , Deformable DETR [73]) with M random-initialized queries Q = { q i } i M =1 to capture the high-level information of images. We build the transformer-based network on top of the multi-scale language-aware visual features to extract M image tokens T = { ( e i , l i ) } i M =1 , each of which is represented by an embedding e i and a location l i , denoting the semantic and positional information of the token. This design not only represents the images independent of input resolution but also extracts the visual representation that is informative with respect to the language prompts.

3.4 LLM-based Open-Ended Task Decoder

We build our decoder on Alpaca [45], an LLM that is adapted from LLaMA [46], to handle various vision-related tasks with language guidance. However, Alpaca has some inherent drawbacks for vision-centric tasks, such as (1) It only has a few digit tokens ( e.g. , 0 ∼ 9) in its vocabulary, which restricts its ability to locate objects by numbers; (2) It uses multiple tokens to represent the category name, resulting in an inefficient scheme in object classification; and (3) It is a causal model that is inefficient for visual perception tasks.

To tackle these issues, we expand the vocabulary of LLM with additional tokens specially designed for vision-centric tasks. First, we add a set of location tokens, denoted as { <p-512> , ..., <p0> , ..., <p512> }, where <p i> represents the discretized offset of i ∈ [ -512 , 512] to the location l i of the image token, and the relative value to image height or width is equal to i/ 512 . These tokens successfully transform the object localization task from continuous variable prediction to more unified discrete bin classification. Second, we introduce semantics-agnostic classification tokens { <c0> , <c1> , ..., <c511> } to replace category name tokens, which overcomes the inefficiency of using multiple tokens to represent categories. The mapping between category names and the classification tokens is flexibly provided in the category set <class> of language instructions, such as { "person":<c0> , "car":<c1> , "black cat":<c2>, ... } . This design allows our model to select the appropriate category name from the provided category set, facilitating efficient and accurate object classification.

Moreover, to address the inefficiency caused by the causal framework, we introduce outputformat-as-query decoding. We first use LLMs to parse the structural output format from the task instructions ( e.g. , ' <cls> <x1> <y1> <x2> <y2> ' for object detection, ' <bos> ' for image captioning), and then feed the tokens of structural output format as queries to the decoder to generate the desired output according to the queries. This simple method enables our model to not only avoid inefficient token-by-token decoding in visual perception tasks, but also keep a unified framework for vision-language tasks.

LLM-based Open-Ended Task Decoder

Task defined

by instructions

parsing

format 1: "< cls > <x1> <y1> ..."

format 2: "<bos>"

...

format n: ...

Note that, during both the training and inference phases in the object detection task, we input 100 sets of '<cls> <x1> <y1> <x2> <y2>" to the decoder, generating 100 object predictions. Those predictions with higher confidence scores will be retained, adhering to a common practice of the object detection task.

In this way, the output of object location and classification is formulated as a foreign language, thus unifying these vision-centric tasks into the format of token classification. Therefore, both vision-language and vision-only tasks can be supervised with the cross-entropy loss like language tasks. In addition, for efficient training, we adopt the Low-Rank Adaptation (LoRA) approach [22], which allows us to train and fine-tune the models without excessive computational costs. We set the LoRA rank to 64 and use LoRA on the QKVO (Query, Key, Value, and Output) in the attention layers. It also acts as a bridge between the language and visual tokens, facilitating effective alignment between the two modalities, ensuring better task customization, and improving the convergence of the overall system.

4 Experiment

4.1 Implementation Details.

We implement two variants of VisionLLM with two image backbones, i.e. , ResNet [21] and InternImage-H [51]. For the language-guided image tokenizer, we adopt BERT-Large [4] as the text encoder and Deformable DETR (D-DETR) [73] to capture high-level information. For the LLM, we employ Alpaca-7B [45], a LLaMA [46] model fine-tuned with instructions, and equip it with LoRA [22] for parameter-efficient fine-tuning.

The model is trained in two stages. In the first stage, we initialize the model with the pre-trained weights of D-DETR and BERT, and train the visual backbone and language-guided image tokenizer to produce language-aware visual features. In the second stage, we connect the image tokenizer with Alpaca-7B and introduce the unified supervision of multiple tasks. We freeze the visual backbone while freezing most parameters of the LLM except a few LoRA parameters. More details on the experimental setup can be found in Sec. B of the supplementary material.

4.2 Task-Level Customization

We first evaluate the task-level customization capability of VisionLLM. VisionLLM supports coarsegrained task customization, including visual perception tasks and visual-language tasks. Table 1 presents the evaluation results on four standard vision-centric tasks, including object detection, instance segmentation, visual grounding, and image captioning. We compare our model with taskspecific methods as well as recently-proposed vision generalist models. Note that, unless specifically mentioned, the results of our model come from a shared-parameter generalist model and switch different tasks by changing the language instructions only. Detailed instructions could be found in the supplementary material.

Object Detection. Object detection is a fundamental computer vision task that involves identifying and localizing objects of interest within an image. Our method achieves comparable or higher results to others, 44 . 6 mAP, with a ResNet-50 [21] backbone. With the same backbone i.e. ResNet-50, our method outperforms Pix2Seq [7] by 1 . 4 mAP, which also discretizes the output coordinates to integers. Furthermore, benefiting from the output-format-as-query framework (see Sec. 3.4), we can decode multiple predictions in parallel during inference, making our approach more efficient. Using InternImage-H [51] as the visual backbone, we obtained 60.2% mAP, which is close to the current state-of-the-art detection-specific model [51], demonstrating the scalability of our generalist model.

Visual Grounding. Visual grounding associates textual descriptions with corresponding regions or objects within an image. Training visual grounding and object detection can potentially conflict with each other, as object detection aims to detect all the objects, while visual grounding should only localize the referred object and suppress other objects. Benefiting from our unified task instructions and the strong instruction comprehension capabilities of LLMs, our model performs both tasks effectively and achieves a result of 80 . 6 P@0.5 for visual grounding. With InternImage-H as the backbone, we achieve 86 . 7 P@0.5 on the validation set of RefCOCO.

Instance Segmentation. Instance segmentation involves identifying and segmenting individual objects within an image. We employ a flexible number of points ( i.e. , 8 ∼ 24) along the object boundary to represent an instance mask. Compared to mainstream models specific to instance

(a) Object-level customization.

(b) Output format customization.

segmentation, our model has a comparable mask AP 50 (61.2% with InternImage-H [51]) but relatively low mask AP 75 . This gap could potentially arise from factors as follows: (1) We discretize the output coordinates to integers for unifying tasks, which introduces information loss; (2) Due to the memory and computational constraint, the number of points in our model is limited, which also results in a performance drop; and (3) Point-based methods typically yield lower results compared to direct mask prediction methods, such as Mask R-CNN [20].

Image Captioning. We also evaluate our model in a representative vision-language task, i.e. image captioning task, and report the BLEU-4 [37] and CIDEr [48] metrics. Note that we do not adopt the CIDEr optimization [41]. We can observe that VisionLLM achieves competitive performance to previous methods. With ResNet-50, we obtain a BLEU-4 score of 31 . 0 and a CIDEr score of 112 . 5 . When using InternImage-H as the backbone, our model achieves a comparable BLEU-4 score of 32 . 1 and a CIDEr score of 114 . 2 . These results demonstrate the effectiveness of VisionLLM in generating descriptive and contextually relevant captions for images.

4.3 Object-Level & Output Format Customization

Our VisionLLM not only allows for customizing the task description, but also for adjusting the target object and the output format using language instructions. Here, we evaluate our model's fine-grained customization ability on COCO. In particular, to customize the target object, we modify the <class> in language instructions to change the model's recognition target from 10 classes to 80 classes. Likewise, to customize the output format, we modify the number of points in language instructions to change the task output format. Table 2 shows that our method can perform well for both object-level and output format changes.

(c) Effect of the num-

4.4 Ablation Study

In this section, we analyze the effect of key components and hyper-parameters on VisionLLM. Unless otherwise specified, we use ResNet-50 [21] backbone and perform the ablation experiments for object detection tasks with random classes and task descriptions on COCO2017 [29].

Single Task vs. Multiple Tasks. We perform an ablation study to assess the impact of multi-task learning with language instructions on VisionLLM. As shown in Table 1, the single-task trained model VisionLLM-R50 sep is slightly better than the jointly trained model VisionLLM-R50 except image captioning. This is due to the multitasking conflicts that also affect previous generalist models [74, 72], and it reflects a trade-off between accuracy and generalization.

Text Encoder in Language-Guided Image Tokenizer. We examine the role of text encoder ( i.e. , BERT) in our language-guided image tokenizer in Table 3a, where we report the results for object detection and visual grounding. The first two rows show that BERT is not essential for object detection but it is crucial for visual grounding. We also investigate the effect of freezing the text encoder during training. The last row indicates that freezing BERT hinders the alignment of vision and language modalities and thus degrades the performance for both tasks.

Image Tokenization Method. As a comparison to our query-based tokenization, we employ average pooling on the feature maps from the D-DETR encoder to obtain M patch embeddings, which serve as token representations for the image. Results in Table 3b indicate a clear advantage of our method. This is due to its ability to capture information from objects of various sizes in a more flexible way.

Number of Localization Tokens. We vary the number of localization tokens from 257 ( i.e. , -128 ∼ 128) to 2049 ( i.e. , -1024 ∼ 1024), to investigate its impact on visual perception performance. As presented in Table 3c, the model consistently exhibits improvement as the number of localization tokens increases until it reaches a saturation point. Remarkably, a substantial performance boost is observed when the number is raised from 257 to 1025 ( +9 . 9 AP). These results indicate that a higher number of localization tokens enables the models to achieve finer localization abilities, thereby improving localization accuracy.

5 Conclusion

In this paper, we have presented VisionLLM, a novel framework that leverages the power of large language models (LLMs) to address vision-centric tasks in an open-ended and customizable manner. We have designed unified language instruction that matches the format of language models and covers various vision-centric tasks including visual perception. We have also developed a language-guided image tokenizer and an LLM-based task decoder that can handle open-ended tasks according to the given language instructions. We have verified the effectiveness of our models on a series of tasks with different granularities, demonstrating their remarkable generality and flexibility.

Broader Impact. We envision that this work will promote the fusion of visual and language tasks. In addition, since our work is built on open-source pre-trained vision foundation models and large language models, requiring low training resources, thus reducing the carbon footprint. We do not foresee obvious undesirable ethical/social impacts at this moment.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 , 2022. 1, 4

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision , 2015. 4

Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems , 2022. 4

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In International Conference on Machine Learning , 2021. 7

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 2020. 3, 4

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision , 2020. 3, 4, 8

Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. arXiv preprint arXiv:2109.10852 , 2021. 7, 8

Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. arXiv preprint arXiv:2206.07669 , 2022. 4, 8

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015. 4

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX . Springer, 2020. 8

Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In International Conference on Learning Representations , 2023. 1

Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning , 2021. 8

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022. 4

MOSS contributors. Moss. https://github.com/OpenLMLab/MOSS , 2023. 3

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. 6

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. 6

Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. arXiv preprint arXiv:2211.07636 , 2022. 1

Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-language representation learning. Advances in Neural Information Processing Systems , 2020. 8

Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. Towards general purpose vision systems. arXiv preprint arXiv:2104.00743 , 2021. 4

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2017. 3, 8

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016. 6, 7, 9

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021. 7

Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017 , 2022. 4

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision , 2022. 1, 2

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021. 8

Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. arXiv preprint arXiv:2211.09808 , 2022. 4, 8

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023. 4, 5

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 , 2023. 2, 3, 5

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision . Springer, 2014. 9

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485 , 2023. 4, 5

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021. 6

Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Yang Yang, Qingyun Li, Jiashuo Yu, et al. Interngpt: Solving vision-centric tasks by interacting with chatbots beyond language. arXiv preprint arXiv:2305.05662 , 2023. 2, 3, 5

Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916 , 2022. 4

OpenAI. Gpt-4 technical report. arXiv , 2023. 3

TB OpenAI. Chatgpt: Optimizing language models for dialogue. OpenAI , 2022. 1, 3

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 2022. 3, 4

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002. 8

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. 1, 4

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 , 2022. 4

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems , 2015. 8

Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017. 8

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 , 2023. 2, 3, 5

Weijie Su, Xizhou Zhu, Chenxin Tao, Lewei Lu, Bin Li, Gao Huang, Yu Qiao, Xiaogang Wang, Jie Zhou, and Jifeng Dai. Towards all-in-one pre-training via maximizing multi-modal mutual information. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023. 1

Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint arXiv:2206.01204 , 2022. 1

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html , 2023. 3, 6, 7

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. 3, 6, 7

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017. 6

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015. 8

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100 , 2022. 1, 4

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-tosequence learning framework. arXiv preprint arXiv:2202.03052 , 2022. 1, 4

Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023. 1, 2, 7, 8

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media , 8(3):415-424, 2022. 6

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022. 1, 2, 8

Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 , 2022. 1, 2

Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. arXiv preprint arXiv:2212.02499 , 2022. 4

Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 , 2023. 2, 4

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560 , 2022. 4, 5

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705 , 2022. 4

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021. 4

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 , 2023. 2, 3, 5

Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020. 5, 8

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retrieval. arXiv preprint arXiv:2303.06674 , 2023. 4

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision , pages 521-539. Springer, 2022. 4

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision , 2022. 8

Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381 , 2023. 2, 3, 5

Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797 , 2021. 1

Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225 , 2022. 1

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 , 2022. 3

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022. 3

Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673 , 2022. 1

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing visionlanguage understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023. 4

Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li, Xiaogang Wang, and Jifeng Dai. Uni-perceiver-moe: Learning sparse generalist models with conditional moes. arXiv preprint arXiv:2206.04674 , 2022. 1, 4, 8, 9

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations , 2021. 3, 6, 7, 8

Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uniperceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022. 2, 4, 5, 8, 9

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new vision-language model where tasks are specified via a language interface instead of being “hard-coded” into the architecture. The model relies on a pretrained decoder model to simply output the answer as text. E.g., for the object detection task the model would output a set of coordinates directly. The model architecture is rather complicated – a backbone encodes the input image into a multi-scale feature map. The language information is encoded by a Bert model and injected into the multi-scale feature map via cross-attent. Then a DETR model produces a set of visual “tokens” from the feature map. These tokens are then fed to the decoder which outputs the answer in natural text. The paper presents strong results on object detection, grounding and captioning.\n\nSTRENGTHS:\n* The paper provides very strong empirical results. Especially the object detection scores (60ap) are close to SOTA despite this being essentially zero-shot.\n* The paper is relatively easy to train -- it relies on LORA and pre-trained models. This and the open-sourced code should make the method available to other researchers.\n* Decoding the output in natural text is a good research direction, it is much more flexible than visual prompt tuning and predefined formats.\n\nWEAKNESSES:\n* The method is rather complicated. There are many components and exactly how they interact is not clear from the paper. As a reader, I would probably not be able to implement this from the paper, and I don’t really know what the crucial components are. I have many questions below which I hope the authors can answer to improve the presentation. E.g. the output-format-as-decoding method is not clearly described.\n\n* There are also VL tasks (e.g. VQA) which I think should be added to the main paper. Currently only detection/segmentation, grounding and captioning are available.\n\nQUESTIONS:\n1. Could you provide pseudo-code for your implementations? E.g. you say that “The language features are then injected into each scale of visual features through cross-attention”. Knowing exactly how the cross-attention is implemented would be good.\n\n2. What is output-format-as-decoding? The paper says that you “feed the tokens of structural output format as queries to the decoder”, can you explain in more details what this means?\n\n3. The paper says “except a few LoRA parameters” – can you specify how many?\n\n4. Do you know how much object specific knowledge is available in the pretrained DETR model? Is it possible to do an ablation here? I assume the DETR model has been pretrained on e.g. COCO detection, so it’s maybe not really a zero-shot task for the model?\n\n5. Why is two-stage training needed? Are there ablations showing the effects of this?\n\n6. Are the resnet parameters pretrained? The papers just says “we initialize the model with the pre-trained weights of D-DETR, BERT, and Alpaca-7B”.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a LLM-based framework VisionLLM for vision-centric tasks. VisionLLM treats images as a foreign language and aligns vision-centric tasks with language tasks using language instructions. Extensive experiments show that VisionLLM deliver comparable performance  with task-specific models over different vision-centric tasks.\n\nSTRENGTHS:\nThe authors demonstrate the feasibility of using large language models as visual decoders in which vision information is treated as a foreign language. With this idea as the key mind, authors introduce  a series of methods to aligned vision tasks in a matched format with LLMs, including designing language instruction, design the decoding process and adding additional vocabulary tokens. Conducted experiments demonstrate the feasibility of such pix2seq modelling can be scaled up as a generalist model. Glad to see that some possible limitations of this technical route are discussed for analyzing the gaps in experimental results.\n\nWEAKNESSES:\nThis work follows the idea of pix2seq to build a generalist model and the authors believe that it is natural to do so, without providing a sufficient explanation of the motivation behind it. Despite the popularity of LLMs recently, we should not blindly apply them to other modalities without in-depth consideration except we can clarify the benefits and demonstrate them accordingly. Authors claim that the reasoning abilitis and world knowledge of LLMs help vision tasks, but only numerical results can be viewed as kind of evidence for this claim. It's not clear and convincing for me from two aspects:\n- Whether the reasoning abilities and world knowledge of LLMs truly help? and how?\n- How about scaling up task-specific vision models when compared to VisionLLM? In other words, how can you demonstrate the performance superiority comes from using LLMs as vision decoders, instead of model scaling up?\n\nThere are a lot of teachnical designs introduced to model vision-centric tasks in a language-matched format. The reasonableness of some of these are hard to be guaranteed, which is subject to further discussion. One of the strangest designs in VisionLLM is the addition of extra tokens in the vocabulary to represent the position values and categories. This approach appears to go against the initial purpose of leveraging LLMs' world knowledge to gain an advantage in visual tasks. Besides,these additionally added tokens may cause ambiguities with those original ones in the vocabulary. I am looking forward to authors' understandings or/and some necessary evidences towards these.\n\nThe effevetiveness of VisionLLM is sorely evaluated on ResNet-50 and Intern-H backbones. The general applicability is subject to further evaluation.\n\nQUESTIONS:\nSome questions have been listed in the weakness part. Other than that, more questions about the details are as follows：\n1. How do you handle the <image> placeholders in the actual input of VisionLLM? Do you replace them with actual image tokens in the latent space?\n2. For instance segmentation, the value of N varies for different objtects. How do you detetermine this value?\n3. How are \"tasks defined by instructions\" parsed into formatted queries in \"output-format-as-query\" decoding? By pre-defined templates for different task categories?\n\nLIMITATIONS:\nThis paper tries to catch the takeoff of LLMs by building a LLM-based generalist model for vision tasks. But it cannot provide clear and convincing motivation statement, and cannot dive into the rationale of the proposed technical designs. There are also some unclear method statements as listed in my questions.\n\nOverall, the idea behind is straightforward. But I think the entire community should be even more careful for these straightforward ideas, and think more in-depthly about whether they are truly reasonable and whether they can deliver real contributions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VisionLLM, an instruction-following agent that can perform various vision-only (classification/detection/segmentation) and vision-language (captioning/VQA) tasks. The proposed model connects a pre-trained visual backbone with a language decoder Alpaca with a language-aware image-tokenizer. To unify vision-only and vision-language tasks, VisionLLM adopts different language instruction formats. Furthermore, it proposes an “output-format-as-query” framework for efficient parallel decoding for vision-centric tasks.\n\nSTRENGTHS:\nThe paper presents an impressive effort in developing a large decoder for vision-only and vision-language tasks, using state-of-the-art multimodal foundational models while treating images as foreign language. The technical details are comprehensive and sound. Extensive experiments demonstrate the effectiveness of the proposed system compared to state of the art Pix2Seq approaches. Ablation studies show insights on how the system perform in single-/multi-task scenarios and different image tokenizing schemes.\n\nWEAKNESSES:\nEven though the proposed system is technically sound, it is still quite complicated. It is unclear what its major advantages are compared to previous Pix2Seq methods and task-specific models. \n\nI have some doubts about the system design: \n- Is the pre-trained instruction-following LLM (Alpaca) crucial in your system design? Can the system perform as well with a (non-instruction-following) LLaMA? Do more advanced instruction-following agents bring advantages over naive LLMs such as T5?\n- One of the major disadvantages of using a large-scale pre-trained LLM (Alpaca) is the increased training and inference costs. Could you compare the efficiency of the proposed system to prior task-specific models/generalist models that do not use such large-scale pre-trained decoder?\n- Why not adopt a ViT-B visual encoder to facilitate comparison to other generalist models such as Uni-Perceiver and Pix2Seq (as in Table 1)?\n\nThe paper is overall well-written. I found one typo:\nL166: placeholdersok?\n\nQUESTIONS:\nI have some questions about technical details:\n- For detection/segmentation tasks, I am confused how the proposed “output-format-as-query” approach (L227-239) works with variable numbers of objects per image. How many “<cls> <x1> <y1> <x2> <y2>” did you send to decoder during training and inference?\n- How does the “output-format-as-query” approach avoid parallel decoding for image captioning (L232)? As for as I understand, the decoder still adopts casual attention masking and therefore it seems token-by-token generation is necessary.\n- The proposed system supports customization of number of points for segmentation tasks — does it require specific training paradigms, i.e., balancing the number of masks with different number of points?\n- Does your system generalize to unseen instructions during test time?\n- What is the sampling procedure for open-ended tasks? For example, do you use top-k/nucleus sampling?\n\nLIMITATIONS:\nNo. Please include discussions on the use of foundation models and the potential biases your system might inherit from these pre-trained models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes VisionLLM, a unified framework for vision tasks and vision-language tasks, using natural language task prompts. It demonstrates capabilities in a good variety of tasks.\n\nSTRENGTHS:\n- This is the first attempt to use natural language prompts for vision-centric tasks such as object detection and instance segmentation. I think this work is tackling an important problem.\n- The language-guided image tokenizer is a novel component to convert image into tokens guided by text.\n- VisionLLM treats image tokens the same way as text tokens, such that the entire task (including image and class list) can be encoded as one piece of text. This converts vision tasks into sequence generation problem handled by a LLM. (I'm actually not sure if this understanding is correct, so I ask this in the Questions section as well.)\n\nWEAKNESSES:\nThe framework is not a simplistic one, consisting of various components and types of losses, e.g. using bipartite matching for one type of outputs. Representing image as a set of ${e_i, l_i}$ is also kind of specific.\n\nThe model can do VQA but I didn't see any VQA results in the paper.\n\nOverall my main complaint is that the framework and pipeline seem a bit complicated, but it is designed to incorporate a wide variety of tasks and seems effective at it. It is a good first attempt at using natural language and LLM to tackle vision tasks.\n\nQUESTIONS:\nHow are image tokens $T$ and `<text>` passed into the LLM? On line 166 it says `<image>` and `<question>` are placeholder tokens for image tokens and question tokens. I imagine `<question>` will be replaced by the actual question inside the language instruction text (`<text>`), but will the `<image>` token be replaced by the image tokens $T$? Figure 3 seems to imply that $T$ is passed into the LLM separately from `<text>`. I think it's more flexible if all inputs are formulated into one piece of text to pass into the LLM, as one can directly extend it to use more than one images, etc.\n\nLIMITATIONS:\nNA.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper construction a vision/language model by passing visual features into a LLM. They trained the model on several standard tasks as well object detection and referring expression by adding special location tokens to the LLM's vocabulary. The model features a text-guided image tokenizer and an efficient decoding approach when generating segmentation or bounding boxes.\n\nSTRENGTHS:\n- The suggestion in section 3.4 seems like a nice way to avoid auto-regressive decoding when it is not needed, although the method was a bit hard to parse.\n- Showing that the approach of adapting visual features for LLM can work objet detection and segmentation is interesting, these tasks have been less well explored in this area.\n-  The quantitive examples in the appendix at least suggests the model has instruction following capabilities similar to other models tuned on the LLaVA dataset.\n\nWEAKNESSES:\n- The scores are decent but not amazing. Only slightly better than pix2seq if using the same backbone, and the CiDER scores are lower then relatively simple models like ClipClap or VL-T5. \n- The central idea is essentially following the pix2seq method combined with the now pretty well studied method of adapting visual features for a LLM approach, which makes sense but does not feel hugely novel to me. \n- The authors should consider evaluating follow the methodology of LLAVa given they are using that data.\n\nQUESTIONS:\nDid the authors try pre-training the model? Pre-training to initially learn the alignment for the vision/language component is common practice for these kinds of models.\n\nCan the model generalize to outputting bounding boxes for tasks other than refexp or object detection? For example for pointing VQA questions where you output a bounding to answer a question.\n\nIn section 3.4, what does \"...feed the tokens of structural output format as queries to the decoder\" mean? That they are used as the initial starting token? What happens if there are multiple objects so they are then multiple class and x1 coordinates to produce? Or if the model needs to interleave text with structured output like in Figure 2a?\n\nTable 1 is missing many models that achieve better scores, BEiT-3 is better at detection and captioning for example. I think they should be included for reference\n\nLIMITATIONS:\nI think the authors should at least note some of the potential issues with these kinds of models (bias, potential for abuse my generating misinformation, hallucination, ect.)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}