SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Illustration of the "output-format-as-query" decoding process. "<c1s><x1><y1> ..." denote the queries of the object's class index and boundary points, and "<bos>" denotes the beginning of string.
Note that, during both the training and inference phases in the object detection task, we input 100 sets of "<cls><x1><y1><x2><y2>" to the decoder, generating 100 object predictions. Those predictions with higher confidence scores will be retained, adhering to a common practice of the object detection task.
In this way, the output of object location and classification is formulated as a foreign language, thus unifying these vision-centric tasks into the format of token classification. Therefore, both vision-language and vision-only tasks can be supervised with the cross-entropy loss like language tasks. In addition, for efficient training, we adopt the Low-Rank Adaptation (LoRA) approach [22], which allows us to train and fine-tune the models without excessive computational costs. We set the LoRA rank to 64 and use LoRA on the QKVO (Query, Key, Value, and Output) in the attention layers. It also acts as a bridge between the language and visual tokens, facilitating effective alignment between the two modalities, ensuring better task customization, and improving the convergence of the overall system.
4.1 Implementation Details.
We implement two variants of VisionLLM with two image backbones, i.e., ResNet [21] and InternImage-H [51]. For the language-guided image tokenizer, we adopt BERT-Large [4] as the text encoder and Deformable DETR (D-DETR) [73] to capture high-level information. For the LLM, we employ Alpaca-7B [45], a LLaMA [46] model fine-tuned with instructions, and equip it with LoRA [22] for parameter-efficient fine-tuning.
The model is trained in two stages. In the first stage, we initialize the model with the pre-trained weights of D-DETR and BERT, and train the visual backbone and language-guided image tokenizer to produce language-aware visual features. In the second stage, we connect the image tokenizer with Alpaca-7B and introduce the unified supervision of multiple tasks. We freeze the visual backbone while freezing most parameters of the LLM except a few LoRA parameters. More details on the experimental setup can be found in Sec. B of the supplementary material.
4.2 Task-Level Customization
We first evaluate the task-level customization capability of VisionLLM. VisionLLM supports coarse-grained task customization, including visual perception tasks and visual-language tasks. Table 1 presents the evaluation results on four standard vision-centric tasks, including object detection, instance segmentation, visual grounding, and image captioning. We compare our model with task-specific methods as well as recently-proposed vision generalist models. Note that, unless specifically mentioned, the results of our model come from a shared-parameter generalist model and switch different tasks by changing the language instructions only. Detailed instructions could be found in the supplementary material.
Object Detection. Object detection is a fundamental computer vision task that involves identifying and localizing objects of interest within an image. Our method achieves comparable or higher results to others, 44.6 mAP, with a ResNet-50 [21] backbone. With the same backbone i.e. ResNet-50, our method outperforms Pix2Seq [7] by 1.4 mAP, which also discretizes the output coordinates to integers. Furthermore, benefiting from the output-format-as-query framework (see Sec. 3.4), we can decode multiple predictions in parallel during inference, making our approach more efficient. Using InternImage-H [51] as the visual backbone, we obtained  $60.2\%$  mAP, which is close to the current state-of-the-art detection-specific model [51], demonstrating the scalability of our generalist model.
Visual Grounding. Visual grounding associates textual descriptions with corresponding regions or objects within an image. Training visual grounding and object detection can potentially conflict with each other, as object detection aims to detect all the objects, while visual grounding should only localize the referred object and suppress other objects. Benefiting from our unified task instructions and the strong instruction comprehension capabilities of LLMs, our model performs both tasks effectively and achieves a result of  $80.6\mathrm{P@0.5}$  for visual grounding. With InternImage-H as the backbone, we achieve  $86.7\mathrm{P@0.5}$  on the validation set of RefCOCO.
Instance Segmentation. Instance segmentation involves identifying and segmenting individual objects within an image. We employ a flexible number of points (i.e.,  $8\sim 24$ ) along the object boundary to represent an instance mask. Compared to mainstream models specific to instance
Table 1: Results on standard vision-centric tasks. "sep" indicates that the model is separately trained on each task.
Method	Backbone	Open- Ended	Detection	Instance Seg.	Grounding	Captioning
AP	\( AP_{50} \)	\( AP_{75} \)	AP	\( AP_{50} \)	\( AP_{75} \)	P@0.5	BLEU-4	CIDEr
Specialist Models
Faster R-CNN-FPN [40]	ResNet-50	X	40.3	61.0	44.0	-	-	-	-	-	-
DETR-DC5 [6]	ResNet-50	X	43.3	63.1	45.9	-	-	-	-	-	-
Deformable-DETR [73]	ResNet-50	X	45.7	65.0	49.1	-	-	-	-	-	-
Mask R-CNN [20]	ResNet-50	X	41.0	61.7	44.9	37.1	58.4	40.1	-	-	-
Polar Mask [61]	ResNet-50	X	-	-	-	30.5	52.0	31.1	-	-	-
Pix2Seq [7]	ResNet-50	X	43.2	61.0	46.1	-	-	-	-	-	-
UNITER [10]	ResNet-101	X	-	-	-	-	-	-	81.4	-	-
VILLA [18]	ResNet-101	X	-	-	-	-	-	-	82.4	-	-
MDETR [25]	ResNet-101	X	-	-	-	-	-	-	86.8	-	-
BEiT-3 [53]	ViT-g	X	-	-	-	-	-	-	-	-	147.6
VL-T5 [12]	T5-B	X	-	-	-	-	-	-	-	-	116.5
Generalist Models
UniTab [64]	ResNet-101	X	-	-	-	-	-	-	88.6	-	115.8
Uni-Perceiver [74]	ViT-B	X	-	-	-	-	-	-	-	32.0	-
Uni-Perceiver-MoE [72]	ViT-B	X	-	-	-	-	-	-	-	33.2	-
Uni-Perceiver-V2 [26]	Swin-B	X	58.6	-	-	50.6	-	-	-	35.4	116.9
Pix2Seq v2 [8]	ViT-B	X	46.5	-	-	38.2	-	-	-	34.9	-
VisionLLM-R50sep	ResNet-50	X	44.8	64.1	48.5	25.2	50.6	22.4	84.4	30.8	112.4
VisionLLM-R50	ResNet-50	✓	44.6	64.0	48.1	25.1	50.0	22.4	80.6	31.0	112.5
VisionLLM-H	InternImage-H	✓	60.2	79.3	65.8	30.6	61.2	27.6	86.7	32.1	114.2
Table 2: Experiments of object-level and output format customization. We conduct these experiments based on VisionLLM-R50, and report the performance of box AP and mask AP on COCO minival for (a) and (b), respectively. "#Classes" and "#Points" indicate the number of classes and boundary points, respectively. "* indicates that we report the mean AP of the given classes, e.g., 10 classes.
(a) Object-level customization.
#Classes	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_S \)	\( AP_M \)	\( AP_L \)
10*	48.9	72.6	51.2	31.7	47.5	67.3
20*	52.7	73.6	56.8	31.8	53.2	70.5
40*	49.3	70.7	53.2	33.1	53.6	63.8
80*	44.6	64.0	48.1	26.7	47.9	60.5
(b) Output format customization.
#Points	AP	\( AP_{50} \)	\( AP_{75} \)	\( AP_S \)	\( AP_M \)	\( AP_L \)
8	18.5	45.7	11.6	9.9	19.7	28.7
14	22.9	48.3	19.4	11.0	25.1	36.0
16	24.2	49.9	20.9	11.5	26.3	36.8
24	25.1	50.0	22.4	12.5	27.4	38.2
segmentation, our model has a comparable mask  $\mathrm{AP}_{50}$  (61.2% with InternImage-H [51]) but relatively low mask  $\mathrm{AP}_{75}$ . This gap could potentially arise from factors as follows: (1) We discretize the output coordinates to integers for unifying tasks, which introduces information loss; (2) Due to the memory and computational constraint, the number of points in our model is limited, which also results in a performance drop; and (3) Point-based methods typically yield lower results compared to direct mask prediction methods, such as Mask R-CNN [20].
Image Captioning. We also evaluate our model in a representative vision-language task, i.e. image captioning task, and report the BLEU-4 [37] and CIDEr [48] metrics. Note that we do not adopt the CIDEr optimization [41]. We can observe that VisionLLM achieves competitive performance to previous methods. With ResNet-50, we obtain a BLEU-4 score of 31.0 and a CIDEr score of 112.5. When using InternImage-H as the backbone, our model achieves a comparable BLEU-4 score of 32.1 and a CIDEr score of 114.2. These results demonstrate the effectiveness of VisionLLM in generating descriptive and contextually relevant captions for images.
4.3 Object-Level & Output Format Customization
Our VisionLLM not only allows for customizing the task description, but also for adjusting the target object and the output format using language instructions. Here, we evaluate our model's fine-grained customization ability on COCO. In particular, to customize the target object, we modify the <class> in language instructions to change the model's recognition target from 10 classes to 80 classes. Likewise, to customize the output format, we modify the number of points in language instructions to change the task output format. Table 2 shows that our method can perform well for both object-level and output format changes.
(a) Effect of text encoder in the language-guided image tokenizer.
(b) Effect of image tokenization method.
(c) Effect of the number of bins (#Bins).
w/ BERT	Freeze	COCO	RefCOCO
-	-	44.7	48.1
✓	-	44.8	84.1
✓	✓	1.3	34.3
Tokenization	AP
Average Pooling	23.1
Ours	44.8
Table 3: Ablation studies on language-guided image tokenizer and hyper-parameters.
#Bins	AP
257	34.9
513	40.8
1025	44.8
2049	44.8
In this section, we analyze the effect of key components and hyper-parameters on VisionLLM. Unless otherwise specified, we use ResNet-50 [21] backbone and perform the ablation experiments for object detection tasks with random classes and task descriptions on COCO2017 [29].
Single Task vs. Multiple Tasks. We perform an ablation study to assess the impact of multi-task learning with language instructions on VisionLLM. As shown in Table 1, the single-task trained model VisionLLM-R50<sub>sep</sub> is slightly better than the jointly trained model VisionLLM-R50 except image captioning. This is due to the multitasking conflicts that also affect previous generalist models [74, 72], and it reflects a trade-off between accuracy and generalization.
Text Encoder in Language-Guided Image Tokenizer. We examine the role of text encoder (i.e., BERT) in our language-guided image tokenizer in Table 3a, where we report the results for object detection and visual grounding. The first two rows show that BERT is not essential for object detection but it is crucial for visual grounding. We also investigate the effect of freezing the text encoder during training. The last row indicates that freezing BERT hinders the alignment of vision and language modalities and thus degrades the performance for both tasks.
Image Tokenization Method. As a comparison to our query-based tokenization, we employ average pooling on the feature maps from the D-DETR encoder to obtain  $M$  patch embeddings, which serve as token representations for the image. Results in Table 3b indicate a clear advantage of our method. This is due to its ability to capture information from objects of various sizes in a more flexible way.
Number of Localization Tokens. We vary the number of localization tokens from 257 (i.e.,  $-128\sim 128$ ) to 2049 (i.e.,  $-1024\sim 1024$ ), to investigate its impact on visual perception performance. As presented in Table 3c, the model consistently exhibits improvement as the number of localization tokens increases until it reaches a saturation point. Remarkably, a substantial performance boost is observed when the number is raised from 257 to 1025 (+9.9 AP). These results indicate that a higher number of localization tokens enables the models to achieve finer localization abilities, thereby improving localization accuracy.
In this paper, we have presented VisionLLM, a novel framework that leverages the power of large language models (LLMs) to address vision-centric tasks in an open-ended and customizable manner. We have designed unified language instruction that matches the format of language models and covers various vision-centric tasks including visual perception. We have also developed a language-guided image tokenizer and an LLM-based task decoder that can handle open-ended tasks according to the given language instructions. We have verified the effectiveness of our models on a series of tasks with different granularities, demonstrating their remarkable generality and flexibility.
Broader Impact. We envision that this work will promote the fusion of visual and language tasks. In addition, since our work is built on open-source pre-trained vision foundation models and large language models, requiring low training resources, thus reducing the carbon footprint. We do not foresee obvious undesirable ethical/social impacts at this moment.
The work is supported by the National Key R&D Program of China (NO. 2022ZD0161300), the National Natural Science Foundation of China (Grant No. 62376134, 61672273, 61832008, 62372223), the Shanghai Committee of Science and Technology (Grant No. 21DZ1100100), and the Fundamental Research Funds for the Central Universities (No. XJ2023000701).