SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenhai Wang*2 Zhe Chen*1,3 Xiaokang Chen*1,4 Jiannan Wu*1,5 Xizhou Zhu1,6
Gang Zeng $^{4}$  Ping Luo $^{5}$  Tong Lu $^{3}$  Jie Zhou $^{6}$  Yu Qiao $^{1}$  Jifeng Dai $^{1,6}$
<sup>1</sup>OpenGVLab, Shanghai AI Laboratory <sup>2</sup>The Chinese University of Hong Kong
$^{3}$ Nanjing University  $^{4}$ Peking University  $^{5}$ The University of HongKong  $^{6}$ Tsinghua University
Code: https://github.com/OpenGVLab/VisionLLM
Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over  $60\%$  mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The code shall be released.
The emergence of large language models (LLMs) like ChatGPT [35] has revolutionized the landscape of artificial general intelligence (AGI), showcasing their impressive zero-shot capabilities in addressing various natural language processing (NLP) tasks through user-tailored prompts or language instructions. Despite these advancements, it's essential to note that the triumph of LLMs does not effortlessly extend to pure vision and vision-language tasks, due to the inherent disparities between modalities and task formats.
The field of computer vision presents a unique set of challenges and paradigms that differ from those of NLP. The traditional paradigm of vision foundation models is pre-training followed by fine-tuning [51, 11, 43, 53, 17, 44], which is effective but comes with significant marginal costs when adapting to diverse downstream scenarios. As shown in Figure 1a, while approaches such as multi-task unification [38, 50, 1, 49, 72] have been used to achieve generalist capability, they often struggle to overcome the limitations imposed by pre-defined tasks, resulting in a gap in open-ended task capabilities compared to LLMs. Recently, visual prompt tuning [24, 66, 70, 67, 54] has emerged as a way to flexibly outline some pure vision tasks (see Figure 1b), such as object detection, instance
* equal contribution, † corresponding author (daijifeng@tsinghua.edu.cn)
Figure 1: Comparison of our VisionLLM with popular paradigms. Unlike current vision generalist models that depend on pre-defined task formats and visual prompt tuning models that are inconsistent with large language models (LLMs), VisionLLM leverages the power of LLMs for open-ended vision tasks by using language instructions.
segmentation, and pose estimation, using visual masking. However, the format of visual prompts considerably deviates from that of language instructions, making it challenging to directly apply the reasoning abilities and world knowledge of LLMs to vision tasks. Therefore, there is an urgent need for a unified generalist framework that can seamlessly integrate the strengths of LLMs with the specific requirements of vision-centric tasks.
In this work, we present VisionLLM, a novel framework that aligns the definitions of vision-centric tasks with the methodologies of LLMs. Leveraging the reasoning and parsing capacities of LLMs, VisionLLM is designed to empower open-ended task capabilities for vision-centric tasks. Specifically, it comprises three core components: (1) a unified language instruction designed for vision and vision-language tasks, (2) a language-guided image tokenizer, and (3) an LLM-based open-ended task decoder that orchestrates various tasks using language instructions. With this framework, a wide range of vision-centric tasks can be seamlessly integrated, including object detection, instance segmentation, image captioning, and visual grounding. In addition, the framework also facilitates task customization at different levels of granularity, allowing for the customization of target objects, output formats, task descriptions, etc.
Compared to current popular API-based applications [60, 65, 42, 32, 28], our model takes a unified, end-to-end approach to integrate VFMs and LLMs, streamlining and enhancing the overall efficiency of the overall process, and leveraging the strengths and data of both VFMs and LLMs within a single, cohesive system. Furthermore, our model surpasses the limitations of generalist vision models pre-trained on pre-defined tasks. VisionLLM can effectively manage vision-centric tasks through language instructions, embodying a flexible and open-ended approach that is not constrained by pre-set tasks. This versatility makes VisionLLM a robust and powerful generalist model for vision and vision-language tasks, opening up new possibilities for the development of unified generalist models that bridge the domains of vision and language.
(1) We propose VisionLLM, the first framework that leverages the power of LLMs to address vision-centric tasks in an open-ended and customizable manner. By aligning the definitions of vision-centric tasks with LLM methodologies, VisionLLM breaks new ground in enabling the unified modeling of vision and language, opening up possibilities for advancing the field.
(2) We overcome many difficulties when porting LLMs to vision-centric tasks, by designing unified language instruction that matches the format of language models and covers various vision-centric tasks including visual perception. Correspondingly, we develop a language-guided image tokenizer and an LLM-based task decoder that can handle open-ended tasks according to the given language instructions based on the LLMs' reasoning and parsing capabilities.
(3) We construct a series of tasks with different granularities to verify the effectiveness of our models, ranging from easy to hard, and from pre-defined to flexible. Through these validations, we demonstrate the remarkable generality of our models, showcasing their ability to handle diverse
Human:"Identify the objects inimage>that belong to {What is the child eating?':  $<  \mathrm{c0}>$  ,'red gamepad':  $<  \mathrm{c1}>$  } and draw a bounding box around each one. The output should be a list of tuples in
Figure 2: Results and visualizations of our VisionLLM. Guided by language instructions, our unified generalist framework showcases its effectiveness on diverse open-ended vision-centric tasks. The text marked with a gray background indicates the customized instructions and the desired outputs.
the format  $(c, x1, y1, x2, y2)$ , where  $c$  is the class label and  $x1, y1, x2, y2$  are the coordinates of the top-left and bottom-right corners of the bounding box. The coordinates should be within the range 1024."
VisionLLM:"The bounding boxes are [(c0>, 226.4, 229.8, 363.1, 347.4), (c1>, 441.1, 183.5, 538.6, 269.9)].
(a) Object Detection with Customized Class Set
Human:"Provide a long description of the <image>.
VisionLLM: "The image features a white dog laying down on a foldable outdoor cooling pad,
which is placed on top of a wooden table. The dog has its head in the middle of the cooling pad, and a fan is positioned behind it, blowing air onto the pad. The table is outdoors, in a grassy area, and there are two chairs placed around it, one on each side. The scene appears to be set in a sunny weather, with a bright and sunny sky, which contributes to the overall outdoor atmosphere."
Human:"For each object in <image> that is a member of{'motorbike':<c0>} output a tuple with the class label and the coordinates of a polygon with 16 points that encloses the object. The code the range 1024. The output x2,y2,...,x16,y16).
VisionLLM:"The detected polygons are [(c0>, 135.3, 95.7, 123.4, 53.4, 84.9, 57.6, 66.8, 60.5, 60.1, 72.3, 34.2, 71.4,..., 124.9, 119.3])."
(b) Object Location with Editable Output Format
Human:"With the objects in the image, if you feel hot, what will you do?"
VisionLLM: "First, You can also try turning off the computer and other electrical appliances in the
room to reduce the heat generated by their operation. As an alternative, you can use the desk fan provided in the image to create a cooling effect in the room. You can plug in the fan and adjust its settings to ensure it blows air in a direction that reaches the room's occupants effectively."
(c) Image Description with Controllable Text Length
(d) Visual Question Answer with Complex Reasoning
scenarios, including random object categories, random output formats, and random task descriptions, as shown in Figure 2. The successful outcomes of these validations underscore the tremendous potential of our model in harnessing the capabilities of LLMs to control and guide vision-centric tasks. In addition, with a generalist LLM-based framework, our model also yields promising results on various vision-centric tasks. Notably, our generalist model achieves an impressive mAP score of  $60 + \%$  on the COCO dataset, surpassing many detection-specific models [73, 6, 20] and approaching the state-of-the-art record.
2.1 Large Language Model
Large language models (LLMs) have gained significant attention in the field of natural language processing (NLP) and artificial general intelligence (AGI), due to their impressive capabilities in language generation, in-context learning, world knowledge, and reasoning. The GPT family, including GPT-3 [5], ChatGPT [35], GPT-4 [34], and InstructGPT [36] are most representative works of LLMs. Other LLMs like OPT [69], LLaMA [46], MOSS [14], and GLM [68] have also made substantial contributions to the field. These models achieve high performance and are open-sourced, serving as valuable resources for training large models and as foundations for further fine-tuning for specific purposes. For instance, Alpaca [45] introduces a self-instruct framework that facilitates instruction tuning of the LLaMA model, reducing the reliance on human-written instruction data. Recently, the emergence of these LLMs has also opened up API-based applications for solving vision-centric tasks. These applications have integrated visual APIs with language models to enable decision-making or planning based on visual information, such as Visual ChatGPT [60], MM-REACT [65], HuggingGPT [42], InternGPT [32], and VideoChat [28]. However, despite the convenience of using language-based instructions to define tasks and describe visual elements, these interactive systems [60, 65, 42, 32, 28] still face limitations in capturing fine-grained visual details and understanding complex visual contexts, which hinder their ability to effectively connecting vision
and language models. In summary, while LLMs have shown tremendous potential in various NLP applications, their applicability to vision-centric tasks has been limited by the challenges posed by modalities and task formats.
2.2 Vision Generalist Model
The pursuit of generalist models [74, 33, 62], which aim to handle a wide range of tasks using a shared architecture and parameters, has been a long-standing goal in the machine learning community. Inspired by the success of sequence-to-sequence (seq2seq) models in the field of NLP [38], recent advancements such as OFA [50], Flamingo [1], and GIT [49] propose modeling diverse tasks as sequence generation tasks. Unified-IO [33], Pix2Seq v2 [8], and UniTab [63] extend this idea by using discrete coordinate tokens to encode and decode spatial information for more tasks. Gato [39] also incorporates reinforcement learning tasks into the seq2seq framework, while GPV [19] develops a general-purpose vision system by combining a seq2seq module with a DETR-based visual encoder [6]. However, these methods suffer from some limitations, such as slow inference speed and performance degradation due to the non-parallel auto-regressive decoding process. Uni-Perceivers [74, 72, 26] solve these issues by unifying different tasks using the maximum likelihood target for each input based on representation similarity, regardless of their modality, making it possible to support both generation and non-generation tasks in a unified framework. Nevertheless, these generalist models are still restricted by pre-defined tasks and cannot support flexible open-ended task customization based on language instructions like LLMs.
2.3 Instruction Tuning
Language instructions are a powerful way to express various NLP tasks and examples for LLMs, as introduced by GPT-3 [5]. Following this idea, subsequent works, such as InstructGPT [36], FLAN [13, 59], and OPT-IML [23], explore the instruction-tuning method [58, 57] and demonstrate that this simple approach effectively enhances the zero-shot and few-shot capabilities of LLMs. The language instruction paradigm has also been adopted by the computer vision community to define image-to-text tasks. Flamingo [1] is a milestone work that uses vision and language inputs as prompts and achieves remarkable few-shot results in various vision-language tasks, such as image captioning [9] and VQA [2]. BLIP-2 [27] further connects the visual encoder with LLMs through a querying transformer and a linear projection layer to build strong multimodal models. MiniGPT-4 [71] and LLaVA [30] finetune the BLIP-2-style models on synthetic multimodal instruction-following data to unleash the potential of LLMs. However, these models mainly focus on image-to-text tasks and fail to address visual perception, such as object detection, instance segmentation, pose estimation, etc. To tackle image inpainting tasks, Bar et al. [3] introduces the first visual prompting framework that utilizes inpainting with discrete tokens on images. Painter [55] and SegGPT [56] employ masked image modeling on raw pixels for in-context learning with paired images. While these visual prompt models demonstrate good results in segmentation tasks, their applicability to numerous real-world vision tasks is challenging. Moreover, defining the visual prompts as image inpainting is inconsistent with the language instructions in LLMs, hard to leverage the reasoning, parsing ability, and world knowledge of LLMs. In this work, we aim to align vision-centric tasks with language tasks, use language instructions to unifiedly and flexibly define all tasks, and solve them with a shared LLM-based task decoder.
3 VisionLLM
This work targets to provide a unified generalist framework that can seamlessly integrate the strengths of large language models (LLMs) with the specific requirements of vision-centric tasks. As shown in Figure 3, the overall architecture of VisionLLM consists of three key designs: (1) a unified language instruction that provides a consistent interface for vision-centric task definition and customization; (2) a language-guided image tokenizer, which encodes visual information in alignment with the given language prompt, enabling the model to comprehend and parse the visual content effectively; and (3) an LLM-based open-task decoder, which utilizes the encoded visual information and language instructions to generate satisfactory predictions or outputs. The three designs work together to achieve a flexible and open-ended framework that can handle various vision-centric tasks at different levels of task customization through language instructions.
Figure 3: Overall architecture of the proposed VisionLLM. It consists of three parts: a unified language instruction designed to accommodate both vision and vision-language tasks, an image tokenizer that encodes visual information guided by language instructions, and an LLM-based open-ended task decoder that executes diverse tasks defined by language instructions.
Different from previous interactive systems [60, 65, 42, 32, 28] that rely on APIs, our VisionLLM presents a more flexible and end-to-end pipeline. Given language instructions that describe the current tasks and an input image, the model first uses a language-guided image tokenizer to encode the image tokens based on the given prompt. Then, the image tokens and language instructions are fed to an LLM-based open-ended task decoder. Finally, it evaluates the generated outputs against the task definition given by the unified language instructions, enabling the model to produce task-specific results. This seamless, end-to-end pipeline enables VisionLLM to effectively combine vision and language, achieving remarkable performance in open-ended and customizable vision-centric tasks.
3.2 Unified Language Instruction
We first introduce unified language instructions to describe vision-centric tasks. This design enables the unification of various vision-only and vision-language task descriptions and allows for flexible task customization.
Vision-Language Tasks. The instructions for vision-language tasks such as image captioning and visual question answering (VQA) are straightforward and similar to NLP tasks. Following previous methods [27, 74, 30], we describe the image captioning task like "The image is <image>. Please generate a caption for the image: ", and the VQA task like "The image is <image>. Please generate an answer for the image according to the question: <question>. Here, <image> and <question> are the placeholders of the image tokens and the question, respectively. The image tokens are directly placed at the placeholder <image>.
Vision-Only Tasks. Designing effective language instructions for vision tasks is a challenging endeavor due to the differences in modality and task format between vision and language. Here, we describe vision tasks by providing a task description and specifying the desired output format via language instructions.
(1) The task description conveys the intended task to the language model. Following self-instruct [57], we design a set of seed instructions with placeholders and employ LLMs to generate a large number of related task descriptions and randomly select one of them during training.
(2) For conventional visual perception tasks like object detection and instance segmentation, we propose a unified output format represented as a tuple  $(C,P)$ , where  $C$  denotes the class index in the category set  $\langle\text{class}\rangle$ , and  $P = \{x_i,y_i\}_{i=1}^N$  represents  $N$  points that locate the object. To align with the format of word tokens, both the class index  $C$  and the coordinates of points  $x_i,y_i$  are transformed into discretized tokens. Specifically, the class index is an integer starting from 0, and the continuous coordinates of the points are uniformly discretized into an integer within the range  $[- \langle\text{range}\rangle, \langle\text{range}\rangle]$ . For object detection and visual grounding tasks, the point number  $N$  is equal to 2, representing the top-left and bottom-right points of object's bounding box. In the case of instance segmentation, we employ multiple  $(N > 8)$  points along the object boundary to represent an instance mask [61]. Other perception tasks such as pose estimation (keypoint detection) can also be formulated as language instructions in this way.
An example of language instruction for the instance segmentation task is as follows: "Segment all the objects of category set <class> within the <range> of the image and generate a list of the format (c, x1, y1, x2, y2, ..., x8, y8). Here, c represents the index of the class label starting from 0, and (x1, y1, x2, y2, ..., x8, y8) correspond to the offsets of boundary points of the object relative to the center point. The image is: <image)".
3.3 Language-Guided Image Tokenizer
VisionLLM considers images as a kind of foreign language and converts them into token representations. Unlike previous works [16, 52, 31] that utilize fixed-size patch embeddings to represent images, we introduce the language-guided image tokenizer to flexibly encode visual information that aligns with task-specific language prompts or instructions.
Specifically, give an image  $\mathbf{X} \in \mathbb{R}^{H \times W \times 3}$  with height  $H$  and width  $W$ , we first feed it to the image backbones (e.g., ResNet [21]) and extract visual features  $F_{v}$  of four different scales. Additionally, we leverage a text encoder (e.g., BERT [15]) to extract the language features  $F_{l}$  from given prompts. The language features are then injected into each scale of visual features through cross-attention [47], yielding multi-scale language-aware visual features, enabling the alignment of features across modalities.
Afterward, we propose to adopt a transformer-based network (e.g., Deformable DETR [73]) with  $M$  random-initialized queries  $Q = \{q_{i}\}_{i = 1}^{M}$  to capture the high-level information of images. We build the transformer-based network on top of the multi-scale language-aware visual features to extract  $M$  image tokens  $T = \{(e_i,l_i)\}_{i = 1}^M$ , each of which is represented by an embedding  $e_i$  and a location  $l_{i}$ , denoting the semantic and positional information of the token. This design not only represents the images independent of input resolution but also extracts the visual representation that is informative with respect to the language prompts.
3.4 LLM-based Open-Ended Task Decoder
We build our decoder on Alpaca [45], an LLM that is adapted from LLaMA [46], to handle various vision-related tasks with language guidance. However, Alpaca has some inherent drawbacks for vision-centric tasks, such as (1) It only has a few digit tokens ( $e.g., 0 \sim 9$ ) in its vocabulary, which restricts its ability to locate objects by numbers; (2) It uses multiple tokens to represent the category name, resulting in an inefficient scheme in object classification; and (3) It is a causal model that is inefficient for visual perception tasks.
To tackle these issues, we expand the vocabulary of LLM with additional tokens specially designed for vision-centric tasks. First, we add a set of location tokens, denoted as  $\{<\mathsf{p}-512>, \ldots, <\mathsf{p}0>, \ldots, <\mathsf{p}512>\}$ , where  $<\mathsf{p}i>$  represents the discretized offset of  $i \in [-512, 512]$  to the location  $l_i$  of the image token, and the relative value to image height or width is equal to  $i/512$ . These tokens successfully transform the object localization task from continuous variable prediction to more unified discrete bin classification. Second, we introduce semantics-agnostic classification tokens  $\{<\mathsf{c}0>, <\mathsf{c}1>, \ldots, <\mathsf{c}511>\}$  to replace category name tokens, which overcomes the inefficiency of using multiple tokens to represent categories. The mapping between category names and the classification tokens is flexibly provided in the category set  $\langle \text{class} \rangle$  of language instructions, such as {"person": <c0>, "car": <c1>, "black cat": <c2>, ...}. This design allows our model to select the appropriate category name from the provided category set, facilitating efficient and accurate object classification.
Moreover, to address the inefficiency caused by the causal framework, we introduce output-format-as-query decoding. We first use LLMs to parse the structural output format from the task instructions (e.g., "<cls> <x1> <y1> <x2> <y2>" for object detection, "<bos>" for image captioning), and then feed the tokens of structural output format as queries to the decoder to generate the desired output according to the queries. This simple method enables our model to not only avoid inefficient token-by-token decoding in visual perception tasks, but also keep a unified framework for vision-language tasks.