SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenfei Yin $^{1,3}$ ; Jiong Wang $^{1,4}$ ; Jianjian Cao $^{1,4}$ ; Zhelun Shi $^{1,2}$ ; Dingning Liu $^{1,5}$ ; Mukai Li $^{1}$ ; Xiaoshui Huang $^{1}$ , Zhiyong Wang $^{3}$ , Lu Sheng $^{2}$ , Lei Bai $^{1\dagger}$ , Jing Shao $^{1\dagger}$ , Wanli Ouyang $^{1}$
$^{1}$ Shanghai Artificial Intelligence Laboratory
<sup>2</sup>Beihang University <sup>3</sup>The University of Sydney <sup>4</sup>Fudan University <sup>5</sup>Dalian University of Technology
{yinzhenfei,bailei,shaojing}@pjlab.org.cn
Large language models have emerged as a promising approach towards achieving general-purpose AI agents. The thriving open-source LLM community has greatly accelerated the development of agents that support human-machine dialogue interaction through natural language processing. However, human interaction with the world extends beyond only text as a modality, and other modalities such as vision are also crucial. Recent works on multi-modal large language models, such as GPT-4V and Bard, have demonstrated their effectiveness in handling visual modalities. However, the transparency of these works is limited and insufficient to support academic research. To the best of our knowledge, we present one of the very first open-source endeavors in the field, LAMM, encompassing a Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs, with a specific focus on facilitating AI agents capable of bridging the gap between ideas and execution, thereby enabling seamless human-AI interaction. Our main contribution is three-fold: 1) We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision. Extensive experiments validate the effectiveness of our dataset and benchmark. 2) We outline the detailed methodology of constructing multi-modal instruction tuning datasets and benchmarks for MLLMs, enabling rapid scaling and extension of MLLM research to diverse domains, tasks, and modalities. 3) We provide a primary but potential MLLM training framework optimized for modality extension. We also provide baseline models, comprehensive experimental observations, and analysis to accelerate future research. Our baseline model is trained within 24 A100 GPU hours, framework supports training with V100 and RTX3090 is available thanks to the open-source society. Codes and data are now available at https://openlamm.github.io/.
Humans interact with the real world through multi-modal information, such as vision and language, since each modality possesses unique capabilities to describe the world, thereby providing us with richer information to construct our world model. Developing AI agents capable of processing such multi-modal information, learning and memorizing world knowledge from it, and comprehending
†Corresponding Authors: Jing Shao (shaojing@pjlab.org.cn) and Lei Bai (bailei@pjlab.org.cn)
open-world instructions from humans to take actions and complete complex tasks has long been a core aspiration in artificial intelligence.
Large Language Models (LLM) have made remarkable progress toward achieving that aspiration. ChatGPT and GPT-4 [1] model can directly comprehend user intents and generalize to unknown real-world tasks [2]. LLM has become a universal task interface for general purposes. Almost all natural language understanding and generation tasks can be transformed into instruction inputs, enabling a single LLM to perform zero-shot generalization on various downstream applications [3, 4, 5]. Within the realm of open-source models, the LLaMA series [6, 7] stands out for its performance and transparency. Building upon the LLaMA ecosystem, models like Alpaca [8] and Vicuna [9] employ different strategies, such as utilizing various machine-generated high-quality instruction-following samples, to enhance the performance of LLMs, showcasing impressive results. Notably, these efforts are all text-only. While Multi-model Large Language Models (MLLM) like GPT-4V [10] and Bard [10] demonstrate remarkable capabilities in processing visual inputs, unfortunately, they are not currently available for use within the open-source academic community.
Hence, we present LAMM, encompassing the Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. As one of the very first open-source endeavors in MLLMs, our aim is to establish LAMM as a thriving ecosystem for training and evaluating MLLMs, and further empower us to cultivate multi-modal AI agents capable of bridging the gap between ideas and execution, facilitating seamless interaction between humans and AI machines. In this work, LLMs serve as the universal task interface, with inputs from vision tokens provided by pre-trained multimodal encoders and language instructions. The powerful modeling capability of LLMs, combined with a unified optimization objective, can help align the model to various modalities. This design sets LAMM apart from visual foundation models [11, 12], where each model is finely tuned for a specific task, or from multi-modal visual language foundation models that can only be used as pre-trained models for visual tasks or possess limited zero-shot capabilities [13], or from multi-task foundation models struggle in tag-of-war problems [14].
Thoroughly, we present a novel instruction tuning dataset, which extends the research of MLLMs to both image and point cloud. Our dataset emphasizes fine-grained information and factual knowledge. Additionally, we introduce the very first attempt of a benchmark for MLLMs that offers a comprehensive evaluation of existing open-source models on various computer vision tasks, with two new evaluation strategies designed explicitly for multi-modal language models. We conduct over 200 experiments to provide extensive results and valuable observations on the capabilities and limitations of MLLMs. Also, we establish an extensible framework to facilitate the extension of multi-modal language models to additional modalities. Our baseline model surpasses existing multi-modal language models in downstream tasks related to images, demonstrating the effectiveness of our framework and dataset. Above all, we have open-sourced our complete codebase for training and evaluating MLLMs, instruction tuning dataset covering both image and point cloud. various baseline models trained with our dataset and framework utilizing different settings to promote the development of an open research community for MLLMs.
Dataset We include an image instruction tuning dataset containing 186,098 image-language instruction-response pairs and a point cloud instruction tuning dataset with 10,262 point cloud-language instruction-response pairs. Motivated by LLaVA [15] and GPT-4V [10], we collect images and point clouds from publicly available datasets and use the GPT-API through self-instruction [16] methods to generate instructions and responses based on the original labels from these datasets. The resulting dataset has three appealing properties: 1) To emphasize fine-grained and dense information, we add more visual information, such as visual relationships and fine-grained categories as input for the GPT-API. 2) We observe on our benchmark that existing MLLMs may struggle to understand vision task instructions. To address this, we designed a method to convert vision task annotations into instruction-response pairs, which enhances MLLMs' understanding and generalization of vision task instructions. 3) Considering the vulnerability of LLMs to the hallucination on factual knowledge, our dataset also includes data pairs for commonsense knowledge question answering by incorporating a hierarchical knowledge graph label system from the Bamboo [17] dataset and the corresponding Wikipedia description.
Benchmark We evaluate 9 common image tasks, using a total of 11 datasets with over 62,439 samples, and 3 common point cloud tasks, by utilizing 3 datasets with over 12,788 data samples, while existing works only provide quantitative results on fine-tuning and evaluating specific datasets such
as ScienceQA, and most works only conduct demonstration or user studies. 1) We are the very first attempt to establish a benchmark for MLLMs. We conducted a comprehensive benchmark to quantify the zero-shot and fine-tuning performance of existing multi-modal language models on various computer vision tasks and compare them against state-of-the-art methods of these tasks, including classification, object detection, pose estimation, visual question answering, facial classification, optical character recognition, object counting. 2) We also attempted two novel evaluation strategies designed explicitly for MLLMs. Specifically, as for language performance on text generation, we established a scoring logic based on the GPT-API. And for tasks involving interactions between localization points and query images, such as object detection and pose estimation, we proposed an object-locating evaluation method.
**Framework** To validate the effectiveness of our dataset, we propose a primary but potential MLLM training framework. To avoid modality conflicts caused by introducing multiple modalities, we differentiate the encoder, projector, and LLM finetuning blocks for different modalities in the framework design. Meanwhile, by adding encoders and decoders for other modalities, our framework can flexibly extend to cover more modalities and tasks, such as video understanding, image synthesis, and so on. We provide the results of our baseline models trained using this framework on our benchmark and present various observations to accelerate future research.
Multimodal Large Language Model. With the rapid development of Large Language Models (LLM) such as ChatGPT, GPT-4 [1], many studies manage to explore incorporating other modalities based on LLM and they can be categorized into two perspectives. 1) System Design Perspective: Visual ChatGPT [18] and MMREACT [19] invoke various vision foundation models by processing user query to investigate the visual roles of ChatGPT with the help of Visual Foundation Models. ViperGPT [20] instructs LLM to parse visual queries into interpretable steps expressed by Python code. HuggingGPT [21] extends its framework to more modalities by integrating more expert models on Huggingface. 2) End-to-End Trainable Model Perspective: The other methodology is to connect models for different modalities into an end-to-end trainable model, also known as multimodal large language model. Flamingo [22] proposes a unified architecture for language and vision modeling, while BLIP-2 [23] introduces a Querying Transformer to connect information from image to text modality. Kosmos [4] and PaLM-E [24] build an end-to-end trainable framework on web-scale multi-modal corpora. With the open-sourced LLaMA [6], Mini-GPT4 [25] optimizes a trainable projection matrix only, which connects pre-trained BLIP-2 style vision encoder and large language model, while LLaVA [15] and mPLUG-OwL [26] also finetune LLM. Besides feeding visual info to LLM as input only, LLaMA-Adapter [27], Multi-modal GPT [28] and Otter [29] also integrate multi modal information with intermediate features in LLM.
Instruction Tuning. Instruction tuning [30] is a method proposed to improve the ability of large language models to follow instructions and enhance downstream task performance. Instruction-tuned models like InstructGPT [31], OPT-IML [32], Alpaca [8], have shown promising improvement compared to their based model. The existing instruction tuning datasets are primarily derived from collections of academic datasets like FLAN [30], chatbot data collected from ChatGPT usage such as ShareGPT, or constructed using self-instruction [16] methods like Alpaca. Apart from pure text instruction tuning datasets, Multi-Instruct [33] covers 47 multi-modal tasks. Mini-GPT4 [25] constructs instruction following dataset by composing image-text datasets and handwritten instruction templates. Moreover, LLaVA [15] feeds captions and bounding boxes as the context of COCO images to GPT-4 and therefore get 150K instruction data. Otter [29] builds such instruction tuning datasets from multi-modal MMC4 dataset [34] and incorporates in-contextual examples into instruction tuning by grouping similar instructions together.
We introduce a comprehensive multi-modal instruction tuning dataset, which involves images and point clouds from publicly available datasets for diverse vision tasks, as well as high-quality instructions and responses based on the GPT-API and self-instruction methods [16]. To be specific, our dataset contains 186K language-image instruction-response pairs, and 10K lanuage-3D instruction-response pairs. Figure 1 provides an overview of its construction process. We provide detailed information on how to construct the multi-modal instruction tuning dataset to guide the academic community, facilitating the replication and further development of our work. We showcase additional demonstrations of sample data and provide a complete prompting method in the Appendix.
Figure 1: Overview of our dataset, demonstrating the process of constructing our Instruction Tuning dataset using the GPT-API. By designing different system messages, in-context learning pairs, and queries, we have created the dataset that covers almost all high-level vision tasks for both 2D and 3D vision. The dataset includes four distinct groups: n-round Daily Dialogue, n-round Factual Knowledge Dialogue, 1-round Detailed Description, and 1-round Visual Dialogue. It is worth noting that for the introduction of vision tasks, we only used the GPT-API to generate instruction-response templates and did not directly generate dialogue data. Finally, some examples of the dataset are presented below, including 2D and 3D scenes and their corresponding instruction-response pairs.
We design four kinds of multi-modal instruction-response pairs: 1)  $C1$ :  $n$ -round daily dialogue focuses on multi-modal daily conversations. 2)  $C2$ :  $n$ -round factual knowledge dialogue aims at dialogues requiring factual knowledge reasoning. 3)  $C3$ :  $1$ -round detailed description aims to elaborate images and 3D scenes in texts. 4)  $C4$ :  $1$ -round visual task dialogue transfers vision tasks into instruction-response pairs, aiming at enhancing generalization ability towards visual tasks.
We include diverse 2D and 3D vision tasks into the dataset, such as captioning, scene graph recognition and VQA that are directly compatible with natural languages, as well as classification, detection, counting and OCR that output labels, bounding boxes, digits and a list of words instead. Note that the point-cloud instruction tuning dataset does not include data in the  $C2$ : n-round factual knowledge dialogue category. This is due to the current lack of publicly available 3D datasets with a well-defined labeling system containing factual knowledge. In our dataset, the instruction-response pairs are gathered from 8 image datasets and 4 point cloud datasets, which are referred in Figure 1.
The first three types of instruction-response pairs are generated by inputting several special designed prompts to the GPT-API, namely system messages, in-context learning pairs and queries: (1) System messages are to inform the GPT-API about the task definitions and requirements. (2) Several in-
Figure 2: An overview of our Benchmark. It includes both 2D and 3D pipelines, covering multiple computer vision tasks. For each task, we provide the task definition, output structure, and a set of questions as instructions to the MLLM model. Then the entity extraction is applied on the output to extract the key answer. The LAMM Evaluation is used to evaluate the model's performance, which includes traditional metrics, binary-location metric and the GPT Metric.
context learning pairs are manually annotated to ensure that the rest instruction-response pairs can be generated by a similar fashion. (3) Queries include comprehensive annotations of captions, bounding boxes of objects, relations between objects, factual knowledges from the Bamboo's label system and their Wikipedia descriptions.
The last type of instruction-response pairs also apply the system messages and in-context learning pairs, but use GPT-API to generate a pool of templates of instruction-response pairs instead. In this way, ground-truth annotations of many vision tasks, such as object/keypoint detection, OCR, counting and etc., can be inserted into these templates, and thus are easier to be converted into reliable language responses, rather than aforementioned query-based conversion.
Different from LLaVA [15], MiniGPT4 [25] and mPLUG-owl [26] that only provide demos and user studies to qualitatively evaluate the performances of their MLLMs, we propose the first benchmark of MLLMs, which instead evaluates the quantitative performance of MLLMs on various 2D and 3D vision tasks. It includes an inference pipeline and a set of evaluation metrics. To be specific, the benchmark on 2D vision tasks evaluates 9 common image tasks, using a total of 11 datasets with over 62,439 samples. The benchmark on 3D vision tasks evaluates 3 common point cloud tasks, by utilizing 3 datasets with over 12,788 data samples.
Inference Pipeline. It ensures that the MLLMs can produce reasonable responses that can be fairly evaluated, which includes the way of processing input instructions and the extracting output entities. We construct the Inference Instruction to help the model better understand the task it is performing and the output structure that is required, aim to improve the stability and reliability of the benchmarking process. Inference Instruction includes Task Definition, Output Structure and the usually employed Query Questions, as shown in Figure 2. Inspired by chain-of-thought prompting methods [35], we also prompt the MLLM to perform complex reasoning followed by the final answer, so as to obtain a more reliable answer. Then, we employ the Natural Language Toolkit (NLTK) and regular expression matching to extract entities from the output text. These entities act as the results.
Evaluation Metrics. The set of evaluation metrics includes Traditional Metrics, Binary Locating Metric, and GPT Metric. The Traditional Metrics are task-specific metrics from the listed 2D and 3D
Figure 3: Framework of multi-modality language model. Each modality is encoded by corresponding pre-trained encoder and decoded by LLM. LLM is shared among modalities and trainable projection layers and LoRA parameters are modality-specific.
vision tasks, which are the most rigorous to evaluate how MLLMs handle vision tasks. In the Binary Locating Metric, the model needs to output an approximated location of a recognized object through the instruction "output the position of the object", whose result is considered true if it is within the object's groundtruth bounding box. It is a straightforward metric to compare the localization ability of an MLLM model. To evaluate the understanding and question-answering abilities of MLLM models, we utilize the GPT metric to evaluate the answers' relevance and accuracy to the groundtruth. To be specific, we prompt GPT to assign scores to the outputs generated by each model through the instruction described in Figure 2. The scoring criteria were based on accuracy, relevance, fluency, logical coherence, and information richness.
Evaluation Settings. All 2D and 3D vision tasks can be evaluated in a zero-shot manner, where the testing data have no intersection with MLLM's training data. Moreover, we also evaluate the finetuning ability of MLLMs on the test dataset about several mainstream tasks, such as detection, classification and VQA in 2D tasks, as well as detection, grounding and VQA in 3D tasks.
5.1 Framework
The overall framework of our baseline MLLM is depicted in Figure 3. Each modality, image or point cloud, is processed by corresponding encoder, whose features are then projected to the same feature space as the text embeddings by a trainable projection layer. Instructions are directly tokenized by SentencePiece tokenizer [36], then the vision and text tokens are concatenated to feed into the LLM model. To finetune LLM efficiently, we add LoRA [37] parameters to all projection layers in the self-attention layers. LoRA parameters for different vision modalities are not shared. Multi-modal tokens are decoded by a shared LLM model and the corresponding LoRA parameters. As shown in Figure 3, only feature projectors and LoRA parameters are optimized during training. We use Vicuna-13B [9], as our LLM. Rank of LoRA modules are set to 32. We train all parameters including projection layers and LoRA modules in a one-stage end-to-end fashion with 4 A100 GPUs.
Input images are resized to be  $224 \times 224$  and split into 256 patches. We use CLIP [38] pre-trained ViT-L/14 and use image patch features output from transformer layers as image representations. We follow the design of FrozenCLIP [39] to encode point clouds, in which point cloud is tokenized to be 256 tokens by PointNet++ [40] and further encoded by CLIP pretrained ViT-L/14.
5.2 Results on Traditional Metrics
Zero-shot Setting on 2D Vision Tasks. Table 1 shows the results of MLLM on 2D vision tasks by the Traditional Metrics. All the MLLM models were tested in a zero-shot setting. Although MLLM models demonstrated certain abilities of recognizing open-vocabulary classes, understanding images, and answering questions, they performed poorly on tasks involving object localization, including object detection, counting and keypoints detection. Localization-aware Tasks: In detection tasks, our baseline model demonstrated stronger localization ability, but there is still a significant gap between the predicted and the ground-truth bounding boxes, indicating MLLMs' weakness to output certain digits representing points and reasoning spatial information. In counting tasks, the MLLM models showed a significant gap between the predicted and ground truth number of objects. MiniGPT4 failed in this task as it is unable to provide a specific number for most of the
Table 1: Comparison of Multi-modal Large Language Models on 2D vision tasks.
Task	Dataset	Metric	LLaVA[15]MiniGPT4[25]mPLUG-owl[26]LAMM
Classification	CIFAR10 [41]	Acc ↑	60.83	46.22	42.5	37.9
Detection	VOC2012 [42]	mAP ↑	1.42	0.92	0.158	7.20
VQA	SQA image [43]	Acc ↑	40.5	43.43	36.39	49.88
AI2D [44]	18.13	Failed	19.31	20.92
Image Caption	flickr30k [45]	BLEU4 ↑	6.65	5.1	2.74	2.56
F-g classification	UCMerced [46]	Acc ↑	47	33.6	32.5	18.23
Counting	FSC147 [47]	MAE ↓	56.2	Failed	60.67	46.88
OCR	SVT [48]	Word Acc ↑	37.78	16.97	30.39	29.14
Facial Classification	CelebA(Smile) [49]	Acc ↑	Failed	66.36	Failed	57.50
CelebA(Hair) [49]	46.42	43.47	40.93	56.96
Keypoints Detection	LSP [50]	PCK ↑	Failed	Failed	Failed	Failed
Table 2: Results of our baseline model on selected 2D vision tasks. Both zero-shot test result and finetuned results reported. Metrics for classification and VQA is accuracy, and that for object detection is mAP@0.5.
Task	Dataset	LAMM (Zero-Shot)	LAMM (Finetune)
Classification	CIFAR10 [41]	37.9	91.2
Object Detection	VOC2012 [42]	7.20	13.48
VQA	SQA image [43]	49.88	74.27