SlowGuess/NeurIPS_2023_OCR huggingface dataset:
data. As for the keypoints detection task, we asked the MLLM models to predict the position of each human keypoint in turn. However, all the predicted positions were not in an acceptable range. The MLLMs show a significant gap in this task, indicating that they have difficulty in accurately predicting the locations of the keypoints. VQA Tasks: Our baseline model demonstrated certain advantages in image understanding and multiple-choice question answering compared to other models. Note that the LLaVA model we compared to was evaluated in the zero-shot setting. Additionally, we removed the random choice process from the LLaVA evaluation to obtain a more straightforward evaluation. Captioning Tasks: All MLLM models performed poorly on image captioning. We argue that BLEU4 is not an appropriate metric since longer captions may lead to lower scores, and MLLMs tend to output detailed description. Classification Tasks: In fine-grained classification tasks and face classification tasks, all MLLMs performed poorly. Specifically, on the CelebA (Smile) dataset, the LLaVA model outputs "yes" to all the queries, while the mPLUG model randomly gives predictions. However, regarding the CelebA (Hair) dataset, the MLLMs can recognize hair color since the ability to infer visual knowledge for color recognition is relatively straightforward. These results suggest that the MLLM models may have difficulty in tasks that require fine-grained distinctions. OCR Tasks: As for OCR tasks, LLaVA can recognize and extract text from images. However, our baseline model performed poorly on this task. We provide more analysis of the results and identify several potential reasons for the performance gap in the Appendix.
Fine-tuning Setting on Image Tasks. We also fine-tuned our baseline model on several vision datasets, including CIFAR10, VOC2012, and SQA image. The results are shown in Table 2. The fine-tuned baseline achieved an accuracy of  $91\%$  on CIFAR10. It also achieved an mAP of  $13\%$  on VOC2012, in comparison with  $4.8\%$  in the zero-shot setting. These results indicate that our baseline models can receive the ability of localizing objects after being fine-tuned on detection data.
Zero-shot Setting on Point Cloud Tasks. Table 3 shows the result of our baseline model on 3D scene understanding tasks, under the zero-shot and fine-tuning settings, respectively. The results after finetuning are significantly better than the zero-shot setting, in all test tasks. Our baseline model finetuned on ScanQA multiple choice data almost achieves  $100\%$  accuracy, which may have an overfitting issue due to the narrow training/test gap and small scale of 3D dataset.
5.3 Results of Binary Locating Metric and GPT Metric
Binary Locating Metric. Table 4 shows the zero-shot results of the MLLMs on the proposed Binary Locating Metric and GPT Metric. The Binary Locating Metric covers the data from VOC2012,
Table 3: Results of 3D tasks. Metrics for 3D object detection and visual grounding is mAP@0.5, and that for 3D VQA is accuracy of multiple choice problem.
Task	Dataset	LAMM (Zero-Shot)	LAMM (Finetune)
3D Object Detection	ScanNet[51]	9.3	11.89
Visual Grounding	ScanRefer[52]	Failed	3.38
3D VQA	ScanQA[53]	26.54	99.89
Table 4: Comparison of results of Binary Locating Metric and GPT Metric of existing MLLMs. The Binary-Locating Metric is the accuracy of the predicted position, and the GPT Metric is the score from GPT response.
LLaVA	MiniGPT4	mPLUG-owl	LAMM
Binary-Loc Metric	14.73	13.12	4.42	31.2
GPT Metric	50.16	7.28	41.88	48.44
FSC147, and LSP. Since the our baseline model has been trained on a small amount of data with detection instructions, it significantly improves in localizing accuracy.
GPT Metric. We calculated GPT scores using a variety of tasks, including VQA, classification, captioning, as well as a small number of detection and counting tasks. As shown in Table 4, LLaVA surpasses other models in performance, while LAMM, although slightly lower than LLaVA, still outperforms Minigpt4 and mPLUG-owl by a wide margin.
5.4 Observation and Analysis
We conducted dozens of experiments and observations on the MLLM model across various tasks to summarize its current capabilities and limitations.
Better Performance in Counting Tasks with Small Number of Objects. As shown in the Table 1, recent MLLMs perform poorly on counting tasks. In the FSC147 dataset, there are data samples with dozens or even hundreds of objects, and the MLLMs would reply with "I cannot accurately count the number" for such data samples. Therefore, we conducted tests on the subset of the FSC147 dataset with less than 10 objects to evaluate the performance of the models on simple data, as shown in Figure 5 (b). The results show that the MLLMs are able to roughly estimate the number of specified objects in the image, but it is still unable to provide an exact numerical value.
GPT Metric is More Appropriate Than BLEU. Figure 4 illustrates the comparison between the generated captions by LLaVA and LAMM on a sample data from the Flickr30k dataset. It is evident that LAMM model produces more detailed image descriptions. However, a notable drawback is the low correlation between its generated sentences and the ground truth sentences, which consequently results in the low BLEU scores indicated in Table 1. Thus, we tried to adopt the GPT Metric to assess the relevance and accuracy of the model's output captions to the ground truth captions. GPT gives a higher score to LAMM model, compared to LLaVA, suggesting that our model is more able to generate high-quality, image-relevant text outputs. This observation also raises the possibility that using GPT-based metrics for evaluating captioning tasks instead of BLEU might offer a more effective evaluation criterion.
Capable of Object Localization but Struggles with Precise Bounding Box Prediction. We visualize the results of LLaVA on VOC2012 dataset. Figure 4 (a) shows that the LAMM model was able to roughly point out the bird in the image, but was unable to accurately locate the entire object.
LAMM Model Exhibits Fine-Grained Classification Ability on CIFAR10. As shown in Figure 4, when presented with a  $32 \times 32$  pixel image of a car, the model's prediction was a more granular category: "Fiat 500L 2012", which accurately identifies the car's brand and model. The left sub figure in Figure 4 (b) shows the image of Fiat 500L 2012 on Autoevolution [54], revealing that it has very similar features to the input image from CIFAR10. These results demonstrate that the MLLM trained with our dataset has the ability to perform more fine-grained classification, and is capable of recognizing subtle differences in images and assigning them to more specific categories.
Instruction and Reasoning Enhance Performance on SQAimage Data Following LLaVA [15], we conducted experiments on the SQAimage dataset using different inference approaches, including prompts with or without reasoning or instruction. The prompts with reasoning make the MLLM
Instruction: Give the bounding box of all the objects in the image.
Response: A bird flying over the water with its wings spread out. The bird appears to be a seagull, with a white body and black wings. To provide the precise coordinates, the bird's bounding box would be (0.156, 0.477, 0.767, 0.523).
Instruction: Give the position of all the objects in the image.
Response: In the image, there is a bird flying over the water. The bird is flying low to the water, with its beak open as if it is in the process of catching a fish. The bird's position is approximately [0.55, 0.53]. (a)
Autoevolution
Instruction: What is the category for this image?
Ground Truth: car
Response: Fiat 500L 2012
Instruction: What is the caption of this image?
Ground Truth:
1. A male volleyball player in red trunks tries to block the shot of the opposing player during a professional volleyball game.
2. Men playing volleyball, with one player missing the ball but hands still in the air.
3. A group of spectators watch a men's sand volleyball game.
4. Volleyball players compete on a sunny day at the beach.
5. Men playing volleyball in the sand.
LLaVA's Response:
The image shows a group of people playing a game of volleyball on a beach. They are actively engaged in the game, with one man jumping in the air to hit the ball. The scene is lively and energetic, capturing the spirit of outdoor sports and beach activities.
LAMM's Response:
The image captures an engaging scene from a professional sand volleyball game. Two men can be seen in the center of the court, fully immersed in the game. Both players are dressed in casual sportswear, with one wearing a white shirt and the other in a shirt of the same color. The sand court stretches across the image, providing ample space for the players to move around. There are multiple flags scattered across the scene, likely serving as markers or boundaries for the game. In the background, the sky provides a clear, light-colored backdrop to the lively action on the court. (c)
Figure 4: Observation and analysis on various tasks. (a) Visualization results on VOC2012. (b) Visualization results on CIFAR10. The right subfigure is from [54]. (c) Results on Flickr30k.
(a)
(c)
Figure 5: (a) Zero-shot Accuracy of LLaVA with different inputs on SQA image. R. indicates reasoning and inst. indicates instruction. (b) Counting Performance on FSC147 of MLLMs. (c) Zero-shot accuracy of LAMM model trained on various data combinations on SQA image. (d) Zero-shot accuracy of LAMM model trained additional instruction data in our dataset.
output the reasoning process before presenting the final results. The prompts with instruction give MLLM the task definition and output structure to the question to help the model better understand the task. The results in Figure 5 (a) shows that the instruction and reasoning both improve the MLLM's VQA ability. These results highlight the importance of incorporating task-specific information and reasoning process into MLLMs.
Difficulty in Comprehending Visual Information for Domain Shifted Data. We conducted an analysis on several datasets that exhibit significant deviations from the training dataset, including UCMerced, CelebA, and LSP. The UCMerced dataset consists of top-down views of scenes, CelebA is a facial dataset that can describe the expressions and hair colors, and the LSP dataset involves 14 key points of the human body, they are significantly different from the COCO dataset during the training phase. These results suggest that the performance of the MLLM model may degrade significantly on datasets that exhibit significant deviations from the training dataset.
Difficulty in Reading Text on SVT data. We analyzed the performance of our baseline model on the SVT dataset and observed unsatisfactory results in Table 1. A possible explanation is that we used the
TextVQA [55] dataset to generate visual task dialogue, which is more geared towards conversational text rather than OCR-related vision tasks. This mismatch in dataset characteristics may have resulted in suboptimal generalization of our model to the SVT dataset. To address this issue, we intend to conduct further investigations and incorporate more appropriate OCR data during the training process to improve our model's performance on OCR-related vision tasks.
Data volume validation on SQA image data. As shown in Figure 5 (c) (d), our four types of image instruction tuning datasets outperform LLaVA[15] on all subsets, resulting in a  $7\%$  overall performance improvement for the complete dataset. Furthermore, we investigated the impact of sampling Daily Dialogue and Detailed Description data at different proportions. Notably, even with the small size of 10k examples, our dataset achieved comparable results to LLaVA-Dataset. As the dataset size increased, the overall performance of our model continuously improved, indicating that our dataset is scalable and can be further optimized by adding more data.
In this part, we discuss limitation and social impact of this work from perspectives of dataset, benchmark and framework.
Dataset In our study, we utilized GPT-API, a state-of-the-art language model, to generate the multimodal instruction data. To achieve the desired format, which includes multi-round dialogue and one-round detailed descriptions, we provided system messages and example dialogues as guidance for the data generation process using GPT-API. The use of GPT-API for generating text-based conversations has been widely adopted in Natural Language Processing, and previous work in multi-modal data [8, 15, 16] has demonstrated promising results in various tasks.
However, it is important to acknowledge the limitations inherent to the underlying GPT model, which are not altered by the use of GPT-API. GPT-API lacks direct access to visual information and relies solely on textual context such as captions and attributes, which restricts its understanding of images and may result in missing detailed information. While GPT-API excels at generating coherent and contextually relevant responses, it can occasionally produce responses that appear plausible but are factually incorrect or lack proper context. It may also struggle with understanding complex or ambiguous queries. Moreover, the generated data used for training may inadvertently reflect inherent biases and other truthworthy issues of GPT-API. To address ethical concerns regarding data generated with GPT-API, we performed manual sampling to examine the data, ensuring that the generated data aligns with societal values, privacy, security, toxicity, and fairness requirements and expectations. In Appendix, we provide an evaluation of the data quality and showcase additional data samples. We also transparently provide the complete prompts used to invoke GPT-API, ensuring transparency throughout our work.
Benchmark LAMM evaluates MLLMs on formatted computer vision tasks and datasets. Due to the diversity of language models' outputs, metrics may fluctuate across experiments. Additionally, LAMM currently adopts metrics such as GPT-eval and binary localization as an initial attempt to evaluate MLLMs' performance. Further research is needed to enhance the stability of benchmark results and design more appropriate metrics, which can be a promising direction for future investigations.
**Framework** Our work establishes a simple MLLM framework to build up a baseline model for our dataset and benchmark. However, there is potential for further development and careful design of MLLMs for future work to enhance their capabilities and performance.
In conclusion, our work presents LAMM, an open-source endeavor in the field of multi-modal large language models. We introduce the image and point-cloud instruction tuning dataset and benchmark, aiming to establish LAMM as a thriving ecosystem for training and evaluating MLLMs. We also provide an extensible framework to facilitate the extension of MLLMs to additional modalities. Our research showcases the effectiveness of MLLMs in handling visual modalities, including images and point clouds, and highlights their potential for generalization via instruction tuning. By making our codebase, baseline model, instruction tuning dataset, and evaluation benchmark publicly available, we aim to foster an open research community for MLLMs. We believe that our work will contribute to the advancement of MLLMs and the development of general-purpose multi-model agents.
This work is done during Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi and Dingning Liu's internship at Shanghai Artificial Intelligence Laboratory. This work is supported in part by the National Key R&D Program of China (NO. 2022ZD0160100), and National Natural Science Foundation of China (62132001).
[1] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. 2, 3
[2] Hao Fu, Yao; Peng and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. *Yao Fu's Notion*, Dec 2022. 2
[3] Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336, 2022.2
[4] Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045, 2023. 2, 3
[5] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. 2
[6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2, 3
[7] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2
[8] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023. 2, 3, 10
[9] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with  $90\%$  * chatgpt quality, March 2023. 2, 6, 23, 24
[10] OpenAI. Gpt-4v(ison) system card. https://cdn.openai.com/papers/GPTV_System/Card.pdf", 2023.2
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 2
[12] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. 2
[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021. 2
[14] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028-1040, 2020. 2
Dataset and code in LAMM has been open sourced at https://github.com/OpenLAMM/LAMM. In this Appendix, we present construction pipeline and more examples of our dataset in Sec. B. Then, Sec. C shows details of benchmark and related evaluation metrics. Sec.D presents implementation details of our framework. Training a model based on our framework takes about 20 A100 GPU hours. Finally, more examples and results are visualized in Sec. E.
B Dataset
The paper introduces a novel method for constructing instruction tuning data, which represents an innovative departure from traditional techniques that rely solely on daily dialogue and detailed description. Instead, our dataset leverages additional factual knowledge extracted from Wikipedia to improve the quality and diversity of the training data. In addition, we also explore the use of traditional vision task data, covering common tasks in both 2D and 3D fields, which is converted into instruction tuning data for training purposes. By combining our new data construction method with traditional vision task data, we aim to improve the accuracy and effectiveness of instruction-tuned models in various vision-related applications. Specifically, we delve into the design of 2D and 3D portion of our dataset in Section B.1 and B.2, respectively. We also outlined the manual approach for checking the quality of the generated data in Section B.3. Finally, we provide a comprehensive explanation of the license and social impact information of our dataset in Section B.4.
B.1 Image Instruction Tuning Dataset
C1: n-round Daily Dialogue & C3: 1-round Detailed Description. The first step of our approach involves incorporating more visual information, such as visual relationships and fine-grained categories as input to GPT-API, providing dense visual context to the generated responses. To construct the C1: n-round Daily Dialogue and C3: 1-round Detailed Description data, we use the COCO images [56], similar to the LLaVA [15] approach. However, we further extract object attributes and relationships from the Visual Genome dataset [57] to emphasize fine-grained and dense information in the generated responses. Specifically, Our approach leverages image scene graph information to provide a structured representation of the objects and their relationships within the image. By doing so, we generated multi-modal dialogue data that enables us to capture the relationships between objects in the image and generate more accurate and natural language instructions. Figures 6 and 7 display the messages utilized to generate daily dialogue and detailed description data in the GPT-API. Additionally, Figure 8 provides detailed examples of the generated results for both types of data.
C2: n-round Factual Knowledge Dialogue. In the second step of our approach, we expand the dataset by incorporating 42K classes of knowledge graph facts from Wikipedia using the Bamboo dataset. This addition enables MLLMs to generate question-answering data related to factual knowledge, which is a valuable addition to the dataset. To generate C2: n-round Factual Knowledge Dialogue data, we utilize the Bamboo dataset and Wikipedia to obtain relevant information, and then use GPT-API to generate a dialogue based on the given content. Specifically, we extract the QID labels and their corresponding Wikipedia descriptions from the Bamboo dataset to generate instruction tuning data. This approach allows us to incorporate common sense knowledge into the dataset, thereby enhancing the ability of MLLMs to generate responses that draw upon a broader range of factual knowledge. The messages used to generate factual knowledge data in the GPT-API are presented in Figure 9, while Figure 10 showcases detailed examples of the factual knowledge data generated by these messages.
C4: 1-round Visual Task Dialogue. In addition to the three types of data discussed earlier, we also incorporate established computer vision tasks, such as image classification, object detection, keypoint detection, OCR, and object counting, into our dataset. This enables MLLMs to handle traditional computer vision tasks and generate responses that incorporate both language and visual information. The typical computer vision dataset consists of a set of images or videos, along with their corresponding labels or annotations that represent the desired output of the computer vision task, such as the class of objects present in the image or the location of an object. However, these discrete
messages = ["You are an AI visual assistant that can analyze a single image. You receive five sentences, each describing the same image you are observing. In addition, specific object locations within the image are given, along with detailed coordinates. These coordinates are in the form of bounding boxes, represented as (x1, y1, x2, y2) with floating numbers ranging from 0 to 1. These values correspond to the top left x, top left y, bottom right x, and bottom right y. It is worth noting that Attributes and Relationships of different objects are also given.