LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
Authors: Zhenfei Yin
Jiong WANG
Jianjian Cao
Zhelun Shi
Dingning Liu
Mukai Li
Xiaoshui Huang
Zhiyong Wang
Lu Sheng
LEI BAI
Jing Shao
Wanli Ouyang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Multi-modality
Large Language Model

Abstract: Large language models have emerged as a promising approach towards achieving general-purpose AI agents. The thriving open-source LLM community has greatly accelerated the development of agents that support human-machine dialogue interaction through natural language processing. However, human interaction with the world extends beyond only text as a modality, and other modalities such as vision are also crucial. Recent works on multi-modal large language models, such as GPT-4V and Bard, have demonstrated their effectiveness in handling visual modalities. However, the transparency of these works is limited and insufficient to support academic research. To the best of our knowledge, we present one of the very first open-source endeavors in the field, LAMM, encompassing a Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs, with a specific focus on facilitating AI agents capable of bridging the gap between ideas and execution, thereby enabling seamless human-AI interaction. Our main contribution is three-fold: 1) We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision. Extensive experiments validate the effectiveness of our dataset and benchmark. 2) We outline the detailed methodology of constructing multi-modal instruction tuning datasets and benchmarks for MLLMs, enabling rapid scaling and extension of MLLM research to diverse domains, tasks, and modalities. 3) We provide a primary but potential MLLM training framework optimized for modality extension. We also provide baseline models, comprehensive experimental observations, and analysis to accelerate future research. Our baseline model is trained within 24 A100 GPU hours, framework supports training with V100 and RTX3090 is available thanks to the open-source society. Codes and data are now available at https://openlamm.github.io.

LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
Jianjian Zhenfei Yin Jiong Jianjian Cao*: Zhelun Shit. Dingning Liu Mukai Li"
Xiaoshui Huang'. Zhiyong Wang'. Lu Sheng". Lei Bai', Jing Shao'! Wanli Ouyang'
'Shanghai Artificial Intelligence Laboratory
Beihang University 'The University of Sydney "Fudan University DDaian University of Technology (yinzhenfei, bailei, shaojing @pjlab. org.cm
Abstract
Large language models have emerged as promising approach towards achieving general-purpose Al agents. The thriving open-source L.M community has greatly accelerated the development of agents that support human human-machine machine dialogue in- teraction through natural language processing. However, human interaction with the world extends beyond only text as a modality, and other modalities such as vision also crucial. Recent works multi-modal large language models, such GPT-4V and Bard, have demonstrated their effectiveness in handling visual modalities. However, the transparency of these works is limited and insufficient support academic research. To the best of our knowledge, we present one of the very first open-source endeavors in the field, LAMM, encompassing a Language- Assisted Multi-Modaa instruction tuning dataset, framework, and benchmark. Our aim is establish LAMM as a growing ecosystem for training and evaluating MLLMs, with specific focus on facilitating Al agents capable of bridging the gap between ideas and execution, thereby enabling seamless human-AL Al interac- tion. Our main contribution is three-fold: 1) We present a comprehensive dataset and benchmark, which cover wide range of vision tasks for 2D and 3D vision. Extensive experiments validate the effectiveness of our dataset and benchmark. 2) We outline the detailed methodology of constructing multi-modal instruction tuming datasets and benchmarks for MLLMs, enabling rapid scaling and extension MILMM research to diverse domains, tasks, and modalities. 3) We provide a primary but potential MILM training framework optimized for modality exten- sion. We also provide baseline models, comprehensive experimental observations, and analysis accelerate future research. Our baseline model is trained within 24 A100 GPU hours, framework supports training with V100 and RTX3090 is available thanks to the open-source society. Codes and data are now available https: / htwpe/nllllle amm, github. io/.
1 Introduction
Humans interact with the real world through multi-modal information, such as vision and language, since each modality possesses unique capabilities to describe the world, thereby providing with richer information to construct our world model. Developing Al agents capable of processing such multi-modal information, learning and memorizing world knowledge from it, and comprehending
'Equal Contribution
CCorresponding Authors: Jing Shao (shaojing @ pjlab.org.cn) on) and Lei Bai Daailei@pjlab.org.cm
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

open- instructions from humans to take actions and complete complex tasks has long been a core aspiration in artificial intelligence.
Large Language Models (LLM) have made remarkable progress toward achieving that aspiration. ChatGPT and GPT-4 [1] model can directly comprehend user intents and generalize unknown real- world tasks [2]. LLM has become universal task interface for general purposes. Almost all natural language understanding and generation tasks can be transformed into instruction inputs, enabling a single LLM to perform zero-shot generalization on various downstream applications [3, 4, 5].
Within the realm of open-source models, the LLaMA series [6. 7] stands out for its performance and transparency. Building upon the LLaMA ecosystem, models like Alpaca [8] and Vicuna [9] employ different strategies, such as utilizing various machine-generatie high- qughhtualtty
cneruction-follwinng samples, to enhance the performance of LLMs, showcasing impressive results. Notably, these efforts are all text-only. While Multi-model Large Language Models (MILM) like GPT-4V [10] and Bard [10] demonstrate remarkable capabilities in processing visual inputs, unfortunately, they are not currently available for use within the open-source academic community.
Hence, we present LAMM, encompassing the Language-Assister Multi-Modal instruction tuning dataset, framework, and benchmark. As one of the very first open-source endeavors in MLLMs, our aim is to establish LAMM as a thriving ecosystem for training and evaluating MLLMs, and further empower us to cultivate multi-modal Al agents capable of bridging the gap between ideas and execution, facilitating seamless interaction between humans and Al machines. In this work, LLMs serve as the universal task interface, with inputs from vision tokens provided by pre-trained multi- modal encoders and language instructions. The powerful modeling capability of LLMss, combined with unified optimization objective, can help align the model to various modalities. This design sets LAMM apart from visual foundation models [11, 12]. where each model is finely tuned for specific task, or from multi-modal visual language foundation models that can only be used as pre-trained models for visual tasks or possess limited zero-shot capabilities [13], or from multi-task foundation models struggle in tag-of-war problems [14].
Thoroughly, we present a novel instruction tuning dataset, which extends the research of MLLMs to both image and point cloud. Our dataset emphasizes fine- tne-grained information and factual knowl- edge. Additionally, we introduce the very first attempt of benchmark for MLLMs that offers a comprehensive evaluation of existing open-source models on various computer vision tasks, with two new evaluation strategies designed explicitly for multi-modal language models. We conduct over 200 experiments to provide extensive results and valuable observations on the capabilities and limitations of MLLMs. Also, we establish an extensible framework to facilitate the extension of multi-modal language models to additional modalities. Our baseline model surpasses existing multi-modal language models in downstream tasks related to images, demonstrating the effectiveness of our framework and dataset. Above all, we have open-soureed our complete codebase for training and evaluating MLLMs, instruction tuning dataset covering both image and point cloud. various bascline models trained with our dataset and framework utilizing different settings to promote the development of an open research community for MLLMs.
Dataset We include an image instruction tuning dataset containing 186,098 image-language instruction-response pairs and point cloud instruction tuning dataset with 10,262 point cloud- language instruction-respone pairs. Motivated by LLaVA [15] and GPT-4V [10], we collect images and point clouds from publicly available datasets and use the GPT-API through self-instruction [16] methods to generate instructions and responses based on the original labels from these datasets. The resulting dataset has three appealing properties: 1) To emphasize fine-grained and dense information, we add more visual information, such as visual relationships and fine- -raiinninedd categories as input for the GPT-API. 2) We observe our benchmark that existing MLLMs may struggle to understand vision task instructions. To address this, we designed a method to convert vision task annotations into instruction-responsee pairs, which enhances MILMs' understanding and generalization of vision task instructions. 3) Considering the vulnerability of LLMS to the hallucination on factual knowledge, our dataset also includes data pairs for commonsense knowledge question answering by incorporating a hierarchical knowledge graph label system from the Bamboo [17] dataset and the corresponding Wikipedia description.
Benchmark We evaluate 9 common image tasks, using a total of datasets with over 62,439 samples, and 3 common point cloud tasks. by utilizing 3 datasets with over 12,788 data samples, while existing works only provide quantitative results on fine-tuning and evaluating specific datasets such
2

---

as ScienceQA, and most works only conduct demonstration or user studies. 1) We are the very first attempt to establish benchmark for MLLMs. We conducted a comprehensive benchmark to quantify the zero-shot and fine-tuning performance of existing multi-modal language models on various computer vision tasks and compare them against state-of-the-art methods of these tasks, including classification, object detection, pose estimation, visual question answering, facial classification, optical character recognition, object counting. 2) We also attempted two novel evaluation strategies designed explicitly for MLLMs. Specifically, as for language performance on text generation, we established a scoring logic based on the GPT-API. And for tasks involving interactions between localization points and query images, such object detection and pose estimation, we proposed an object-locating evaluation method.
F Framework To validate the effectiveness of our dataset, we propose primary but potential MLLM training framework. To avoid modality conflicts caused by introducing multiple modalities, we differentiate the encoder, projector, and L.LM finetuning blocks for different modalities in the framework design. Meanwhile, by adding encoders and decoders for other modalities, our framework can flexibly extend to cover more modalities and tasks, such video understanding, image synthesis, and so on. We provide the results of our baseline models trained using this framework on our benchmark and present various observations to accelerate future research.
2 Related Work
Multimodal Large Language Model. With the rapid development of Large Language Models (LLM) such as ChatGPT, GPT-4 [1], many studies manage explore incorporating other modalities based on LLM and they can be categorized into two perspectives. 1) System Design Perspective: Visual ChatGPT [18] and MMREACT [19] invoke various vision foundation models by processing user query to investigate the visual roles of ChatGPT with the help of Visual Foundation Models. ViperGPT [20] instructs LLM to parse visual queries into interpretable steps expressed by Python code. HuggingGPT [21] extends its framework to more modalitics by integrating more expert models on Huggingface. 2) End-to-End Trainable Model Perspective: The other methodology is to connect models for different modalities into an end-to-end trainable model, also known as multimodal large language model. Flamingo [22] proposes unified architecture for language and vision modeling, while BLIP-2 [23] introduces a Querying Transformer to connect information from image to text modality. Kosmos [4] and PaLM-E [24] build an end-to-end trainable framework on web-scale multi-modal corpora. With the open-sourced LLaMA [6]. Mini-GPT4 [25] optimizes a trainable projection matrix only, which connects pre-trained BLIP-2 style vision encoder and large language model, while VL [15] and mPLUG-Ow [26] also finetune LLM. Besides feeding visual info to LLM as input only, ALaMA-Adapter [27], Multi-modal GPT [28] and Otter [29] also integrate multi modal information with intermediate features in LLM.
Instruction Tuning. Instruction tuning [30] is method proposed to improve the ability of large language models to follow instructions and enhance downstream task performance. Instruction-tunee models like InstructGPT [31], OPT-IML [32], Alpaca [8], have shown promising improvement compared to their based model. The existing instruction tuning datasets are primarily derived from collections of academic datasets like FLAN [30], chatbot data collected from ChatGPT usage such as ShareGPT, or constructed using self-instruction [16] methods like Alpaca. Apart from pure text instruction tuning datasets, Multi-Instrucc [33] covers 47 multi-modal tasks. Mini-GPT4 [25] constructs instruction following dataset by composing image-text datasets and handwritten instruction templates. Moreover, LLaVA [15] feeds captions and bounding boxes as the context of COCO images to GPT-4 and therefore get 150K instruction data. Otter [29] builds such instruction tuning datasets from multi-modal MMC4 dataset [34] and incorporates in-contextual examples into instruction tuning by grouping similar instructions together.
3 Dataset
We introduce comprehensive multi-modal instruction tuning dataset, which involves images and point clouds from publicly available datasets for diverse vision tasks, as well as high-quality instruc- tions and responses based on the GPT-API and self-instruction methods [16]. To be specific, our dataset contains 186K language-image instruction-response pairs, and 10K lanuage-3D instruction- response pairs. Figure provides an overview of its construction process. We provide detailed information on how to construct the multi-modal instruction tuning dataset to guide the academic community, facilitating the replication and further development of our work. We showcase additional demonstrations of sample data and provide complete prompting method in the Appendix.
3

---

Inputs
GPTARI
Outputs
1
System Message
niounn Daily Dialogue
You are an visuel assistant...
Garouddndd Fectual Knowledge Dialogue
C1. You need generate conversition
Cg. a-rundd Detailed Description
generate multiple rounds conversations
C4:1/round Visual Task Dialogue
based on Bamboo label and Wikipedia description
C3 You need to descrihe the scene in detail
C4, Give me 20 similarnstruccte instruction respooon prompts.
2D &D VisionTasks
Instruction Pool & Response Template
In-context Learning Pairs
Instructions: aWh the names of the objects in the mage, and where
Human- nabeen-
resporises pairs
Object Detection
pasttioned?,
Query
Image Classilictioon
How many itemn in image belong to particular category? ...]
Keypoint Detection
CL <Cpptoon> -Boxe>> + <Relationships>
Counting
Responses: (There an object the (P] position of the mage, and category (C), There image... ...]
<Bemboo lobels> -Wiipeedis descriptions
OCR 3D indoor Detectinn
eCaptions dRelationships>
C4.NA
Dataset
Bemooc ELS
COCDEET
Visal GEBAIT
Shapetae
Ca: i-roonn Daily Dialogue
C3: 1-ouud Detailed Desrrleed
Instruction:ddd dimggeo What are
woman dsinn?
explanation of
street, with man holting a phone.
Resporse:
and woran
image walcig
down
Resporse: This
donning sung and handbag, clothed shirt yellow
Instruction: Whht does interactioneer bey about their rdationsiia!
Resporis:: The mage
eman weman wallon long arm arm, suggesting
they close roldions
Intruction: opeint cloud What is this parnt cloud about?
Instruction: poiitt doud's there a cabline in roon?
Responde:
Yes, The doud desoriis slod comerr cabinet with
Resporse: The point shows living space with a large gray shaeed couch pos tioned in the in riddle of the room. The couch ad scent to the kitchon area and spioss from the room area, providigg a coay and gethering space...
lorge opering in the center tora televition.
Instruction: Where the derk located mo
Response: The denk a lleeated in front bark office chair with wheels
n-rond Factual Knttondddectttoooooeeegegggg Dialogue C4:2-round VisualTask Dialogue
Instruction: cimages Can you ocott
sbjects Indgeand
got
Response Upon
mage,
object pos
Detection
anob
catagory.
30 Detection
cimages type depicted
Instruction: spoint clarts Provcie a detuiled srcount oftice sbjett shsecoed gaint cinud, arrding their precise lncations?
image?
cimages What
Resporse:
image
When where Tupolay
Response: obsenvig the and soight
categury can ona, 1412, 8.4 san, a grt, 1.08)] positionin the plint Positioned at the
Resporae: ciject within
cluerved
The
the
1.95, 0.222, 0.42) Gass 0:40), 6.2))) location, object the window catoogor....
Tupolev Design the Union
OCR
Figure 1: Overview of our dataset, demonstrating the process of constructing our Instruction Tuning dataset using the GPT-API. By designing different system messages, in-context learning pairs, and queries, have created the dataset that covers almost all high-level vision tasks for both 2D and 3D vision. The dataset includes four distinct groups: n-round Daily Dialogue, n-round Factual Knowledge Dialogue, 1-round Detailed Description, and 1-round Visual Dialogue. It is worth noting that for the introduction of vision tasks, we only used the GPT-API to generate instruction instruction-response templates and did not directly generate dialogue data. Finally, some examples of the dataset pairs.
response
presented below, including 2D and 3D scenes and their corresponding
We design four kinds of multi-modal instruction-responee pairs: 1) Cl: n-round daily dialogue focuses on multi-modal daily conversations. 2) C2: n-round factual knowledge dialogue aims at dialogues requiring factual knowledge reasoning. 3) C3: 1-round detailed description aims to elaborate images and 3D scenes in texts. 4) C4: 1-round visual task dialogue transfers vision tasks into instruction-responnee pairs, aiming at enhancing generalization ability towards visual tasks.
We include diverse 2D and 3D vision tasks into the dataset, such as captioning, scene graph recognition and VQA that are directly compatible with natural languages, well as classification, detection, counting and OCR that output labels, bounding boxes, digits and a list of words instead. Note that the point-cloud instruction tuning dataset does not include data in the C2: n-round factual knowledge dialogue category. This is due to the current lack of publicly available 3D datasets with a well-defined labeling system containing factual knowledge. In our dataset, the instruction-responss pairs gathered from 8 image datasets and 4 point cloud datasets, which are referred in Figure 1.
The first three types of instruction-responss pairs are generated by inputting several special designed prompts to the GPT-API, namely system messages, in-context learning pairs and queries: (1) System messages are to inform the GPT-API about the task definitions and requirements. (2) Several in-

---

aD Detection
Traditional Metrics
You are performing an object dstattonn takt Your qcalla locate objecta the mraes and uie the comaspnning coard rates
Reaponse: There
doglo.s u.e, a.A,
Classfication
Accuracy
Your cotput ahou zs aigll rame. Your ansver should the db sct nurr and ite bourding bos, repreanted a ths, vn, ni, win muunegrummet numse vneging from
7.7%, fromion (oyo)t lusl
Detection
MAP
Caption
RLEU
2D Counting
count abgect enin www.aaaa.a......................
category
tlom the tmuse""
Counting
gow
counting computer visian task that invo vas desocting
counting the number intentanee of specifi the objects within image.
Respors:: There eightssaind's thi Beraae
II. Binary-Loc. Metric
The anower reedeso Arab sumera.
Insi bbos
Hew seathe ine mrape?
Posilbv
Extraction
LAMM Evaluation
Entity
Outsides abox
Task Definition Output Structure Instruction
MLLM
30 Grounding
Negathre
are parforming apo nt cloud greun ctingg and goal
Resporon: kiccasas
thould unting represented aeDo
at (6-755 5-477
so, wilwith floating numberr meging from to
6-ghh, 68), gigg..
spentt cloud- Thare a lnshee It the roll Sae my the boarding hon of in paint clecd.
IlI. Metric
2D VOA
www............................. www..a..........................
Task Dufinition
Enal-nnctruck
pealisto penerate nacural lingunge ar swers accursndyy salve the quatton
Respores: Upon
Modal
GPTT Stteting
thorough examinatinn
Peigemo
Model A:
Garna du complex rezventr garooas and accurols arowers
beinfeeed the cusot i door.
Reipuiso
opter deue Whar plared
Rripeme
Model IS
Figure 2: An overview of our Benchmark. It includes both 2D and 3D pipelines, covering multiple computer vision tasks. For each task, we provide the task definition, output structure, and set of questions as instructions to the MLLM model. Then the entity extraction is applied on the output to extract the key answer. The LAMM Evaluation used to evaluate the model's ss performance, which includes traditional metrics, binary-location metric and the GPT Metric.
context learning pairs are manually annotated to ensure that the rest instruction- response pairs can be generated by a similar fashion. (3) Queries include comprehensive annotations of captions, bounding boxes of objects, relations between objects, factual knowledges from the Bamboo's ' label system and their Wikipedia descriptions.
The last type of instruction- response pairs also apply the system messages and in-context learning pairs, but use GPT-API to generate a pool of templates of instruction- response pairs instead. In this way, ground tround-truth annotations of many vision tasks, such as object/keypoint detection. OCR, counting and etc., can be inserted into these templates, and thus are easier to be converted into reliable language responses, rather than aforementioned query- query-based conversion.
Benchmark
Different from LLaVA [15], MiniGPT4 [25] and mPLUG-owl [26] that only provide demos and user studies qualitatively evaluate the performances of their MLLMs, we propose the first benchmark of MLLMs, which instead evaluates the quantitative performance of MLLMs on various 2D and 3D vision tasks. It includes inference pipeline and a set of evaluation metrics. To be specific, the benchmark on 2D vision tasks evaluates 9 common image tasks, using a total of 11 datasets with over 62,439 samples. The benchmark 3D vision tasks evaluates 3 common point cloud tasks, by utilizing 3 datasets with over 12,788 data samples.
Inference Pipeline. It ensures that the MILMSs can produce reasonable responses that can be fairly evaluated, which includes the way of processing input instructions and the extracting output entities. We construct the Inference Instruction to help the model better understand the task it is performing and the output structure that is required, aim to improve the stability and reliability of the benchmarking process. Inference Instruction includes Task Definition, Output Structure and the usually employed Query Questions, as shown in Figure 2. Inspired by chain-of-thought prompting methods [35], also prompt the MLLM to perform complex reasoning followed by the final answer, so as to obtain a more reliable answer. Then, we employ the Natural Language Toolkit (NLTK) and regular expression matching to extract entities from the output text. These entities act the results.
Evaluation Metrics. The set of evaluation metrics includes Traditional Metrics, Binary Locating Metric, and GPT Metric. The Traditional Metrics are task-specific metrics from the listed 2D and 3D
5

---

Instruction
Image + Encoder
Image Projector
LoRA
Response
How many fruits are there in the <image and what are they?
There's 'e 15 fruits. There are 3 apples, 2 pears and...
Large Language Model
Response
Instruction
Point Cloud Encoder
Point Cloud Projecter
The room shown can be bathroom with a separate toilet...
What's the type of room in the spoint cloud>?
LoRA
Figure 3: Framework of multi- modality language model. Each modality is encoded by corresponding pre-trained encoder and decoded by LLM. LLM shared among modalities and trainable projection layers and LoRA parameters are modality
vision tasks, which are the most rigorous to evaluate how MLLMs handle vision tasks. In the Binary Locating Metric, the model needs output an approximated location of recognized object through the instruction "output the position of the object", whose result is considered true if is within the object's groundtruth bounding box. is a straightforward metric compare the localization ability of answering abilitis of MLLM models,
an MLLM model. To evaluate the understanding and we utilize the GPT metric evaluate the answers' relevance and accuracy the groundtruth. To be specific, we prompt GPT to assign scores to the outputs generated by each model through the instruction described in Figure 2. The scoring criteria were based on accuracy, relevance, fluency, logical coherence, and information richness.
Evaluation Settings. All 2D and 3D vision tasks can be evaluated in a zero-shot manner, where the testing data have no intersection with MLLM's training data. Moreover, we also evaluate the finetuning ability of MLLMs the test dataset about several mainstream tasks, such detection, classification and VQA in 2D tasks, as well as detection, grounding and VQA in 3D tasks.
5
Experiments and Results
5.1
Framework
The overall framework of our baseline MILM is depicted in Figure 3. Each modality, image or point cloud, is processed by corresponding encoder, whose features are then projected to the same feature space the text embeddings by a trainable projection layer. Instructions are directly tokenized by SentencePiece tokenizer [36], then the vision and text tokens are concatenated to feed into the LLM model. To finetune LLM efficiently, we add LoRA [37] parameters all projection layers in the self-attention layers. LoRA parameters for different vision modalities are not shared. Multi-modal tokens are decoded by shared LLM model and the corresponding LoRA parameters. As shown in Figure 3, only feature projectors and LoRA parameters are optimized during training. We use Vicuna- 13B [9], as our LLM. Rank of LoRA modules are set 32. We train all parameters including projection layers and LoRA modules in a one-stage end-to-end fashion with 4 A100 GPUs.
Input images resized be 224x 224 and split into 256 patches. We CLIP [38] pre-trained ViT-L/14 and use image patch features output from transformer layers as image representations. We follow the design of FrozenCLIP [39] encode point clouds, in which point cloud is tokenized to be 256 tokens by PointNet++ [40] and further encoded by CLIP pretrained ViT-L/14.
5.2 Results on Traditional Metrics
Zero-shot Setting on 2D Vision Tasks. Table 1 shows the results of MLLM on 2D vision tasks by the Traditional Metrics. All the MLLM models were tested in a zero-shot setting. Although MLLM models demonstrated certain abilities of recognizing open-vocabulary classes, understanding images, and answering questions, they performed poorly on tasks involving object localization, including object detection, counting and keypoints detection. Localization- eiriiaatioo-wwaeee Tasks: detection tasks, our baseline model demonstrated stronger localization ability, but there is still a significant gap between the predicted and the ground-truth bounding boxes, indicating MLLMs' weakness to output certain digits representing points and reasoning spatial information. In counting tasks, the MILMM models showed significant gap between the predicted and ground truth number of objects. MiniGPT4 failed in this task as it is unable to provide specific number for most of the

---

Table 1: Comparison of Multi-modal Large Language Models on 2D vision tasks.
Task
Dataset
Metric
Classification
CIFAR10 [41]
Ace
60.83
46.22
42.5
37.9
Detection
VOC2012 [42]
mAP t
1.42
0.92
0.158
7.20
SQAimage [43]
40.5
43.43
36.39
49.88
VQA
Acc t
AI2D [44]
18.13
Failed
19.31
20.92
Image Caption
flickr30k [45]
BLEU4 f
6.65
5.1
2.74
2.56
F-g classification
UCMerced [46]
Acc
47
33.6
32.5
18.23
Counting
FSC147 [47]
MAE
56.2
Failed
60.67
46.88
OCR
SVT [48]
Word Acc 1
37.78
16.97
30.39
29.14
CelebA(Smilc) [49]
Failed
66.36
Failed
57.50
Facial Classification
Ace
CelchA(Hair) [49]
46.42
43.47
40.93
56.96
Keypoints Detection
L.PP [50]
PCK 1
Failed
Failed
Failed
Failed
Table 2: Results of our baseline model on selected 2D vision tasks. Both zero-shot test result and finetuned results reported. Metrics for classification and VQA is accuracy, and that for object detection is m.5.0.5.
Task Classification Object Detection VQA
Dataset
LAMM (Zero-Shot)
LAMM (Finetune)
CIFARIO [41]
37.9
91.2
VOC2012 [42]
7.20
13.48
SQAimage [43]
49.88
74.27
data. As for the keypoints detection task, we asked the MLLM models to predict the position of each human keypoint in turn. However, all the predicted positions were not in an acceptable range. The MLLMs show a significant gap this task, indicating that they have difficulty in accurately predicting the locations of the keypoints. VQA Tasks: Our baseline model demonstrated certain advantages image understanding and multiple-choie question answering compared to other models. Note that the LLaVA model we compared to was evaluated in the zero-shot setting. Additionally, we removed the random choice process from the LLaVVA evaluation to obtain a more straightforward evaluation. Captioning Tasks: All MILLM models performed poorly on image captioning. We argue that BLEUU4 not an appropriate metrie since longer captions may lead to lower scores, and MI..Mss tend to output detailed description. Classification Tasks: In fine-grained classification tasks and face classification tasks, all MLLMs performed poorly. Specifically, on the CelebA (Smile) dataset, the LLaVA model outputs "yes" to all the queries, while the mPLUG model randomly gives predictions. However, regarding the CelebA (Hair) dataset, the MLLMS can recognize hair color since the ability to infer visual knowledge for color recognition is relatively straightforward. These results suggest that the MLLM models may have difficulty in tasks that require fine nee-grained distinctions. OCR Tasks: As for OCR tasks, LLaVA can recognize and extract text from images. However, baseline model performed poorly on this task. We provide more analysis of the results and identify several potential reasons for the performance gap in the Appendix.
Fine-tuning Setting on Image Tasks. We also fine-tuned our baseline model on several vision datasets, including CIFARIO, VOC2012, and SQAimage. The results shown in Table 2. The fine-tuned baseline achieved an accuracy 91 % on CIFARIO. 10. also achieved an mAP of 13% on VOC2012, in comparison with 4.8% in the zero-shot setting. These results indicate that our baseline models can receive the ability of localizing objects after being fine-tuned on detection data.
Zero-shot Setting on Point Cloud Tasks. Table 3 shows the result of our bascline model on 3D scene understanding tasks, under the zero-shot and fine-tuning settings, respectively. The results after finetuning are significantly better than the zero-shot setting, in all test tasks. Our baseline model finetuned on ScanQA multiple choice data almost achieves 100% accuracy, which may have an overfitting issue due to the narrow training/test gap and small scale of 3D dataset.
5.3 Results of Binary Locating Metric and GPT Metric
Binary Locating Metric. Table shows the zero-shot results of the MILMSs on the proposed Binary Locating Metric and GPT Metric. The Binary Locating Metric covers the data from VOC2012.
7

---

Table 3: Results of 3D tasks. Metries for 3D object detection and visual grounding is mAP@0.5, and that for 3D VQA is accuracy of multiple choice problem.
Task
Dataset ScanNei[51]
LAMM (Zero-Shot)
LAMM (Finetune)
3D Object Delection
9.3
11.89
Visual Grounding
ScanRefer[52]
Failed
3.38
3D VQA
ScanQA[53]
26.54
99.89
Table 4: Comparison of results of Binary Locating Metric and GPT Metric of existing MLLMs. The Binary-Locating Metric is the accuracy of the predicted position, and the GPT Metric is the score from GPT response.
LLaVA MiniGPT4 mPLUG-ow]
LAMM
Binary-Loe Metric GPT Metric
14.73
13.12
4.42
31.2
50.16
7.28
41.88
48.44
FSC147, and LSP. Since the our baseline model has been trained on a small amount of data with detection instructions, it significantly improves in localizing accuracy.
GPT Metric. We calculated GPT scores using variety of tasks, including VQA, classification, captioning, as well as a small number of detection and counting tasks. As shown in Table 4, LLaVA surpasses other models in performance, while LAMM, although slightly lower than LLaVA, still outperforms Minigpt4 and mPLUG-owl by wide margin.
5.4 Observation and Analysis
We conducted dozens of experiments and observations on the MLLM model across various tasks to summarize its current capabilities and limitations.
Better Performance in Counting Tasks with Small Number of Objects. As shown in the Table 1, recent MILMS perform poorly on counting tasks. In the FSC147 dataset, there are data samples with dozens or even hundreds of objects, and the MLLMs would reply with "I cannot accurately count the number" for such data samples. Therefore, we conducted tests on the subset of the FSC147 dataset with less than 10 objects to evaluate the performance of the models on simple data, as shown in Figure (b). The results show that the MLLMs are able to roughly estimate the number of specified objects in the image, but it is still unable to provide an exact numerical value.
GPT Metric is More Appropriate Than BLEU. Figure 4 illustrates the comparison between the generated captions by LLaVA and LAMM on a sample data from the Flickr30k dataset. is evident that LAMM model produces
detailed image descriptions. However, notable drawback the low correlation between its generated sentences and the ground truth sentences, which consequently results in the low BLEU scores indicated in Table 1. Thus, tried adopt the GPT Metric to assess the relevance and accuracy of the model's output captions to the ground truth captions. GPT gives higher score to LAMM model, compared LLa suggesting that our model more able to generate high-quality, image-relevant text outputs. This observation also raises the possibility that using GPT-based metries for evaluating captioning tasks instead of BLEU might offer a more effective evaluation criterion.
Capable of Object Localization but Struggles with Precise Bounding Box Prediction. We visualize the results of LLaVA on VOC2012 dataset. Figure 4 (a) shows that the LAMM model was able to roughly point out the bird in the image, but unable accurately locate the entire object.
LAMM Model Exhibits Fine-Graide Grained Classification Ability on CIFARIO. As shown in Figure 4, more granular category: "Fiat 500L 2012", which accurately identifies the car's brand and model. The left sub figure in Figure 4 (b) shows the image of Fiat 500L 2012 on Autoevolution [54], revealing that has very similar features to the input image from CIFAR 10. These results demonstrate that the MILM trained with our dataset has the ability perform more fine- ine-grained classification, and is capable of recognizing subtle differences in images and assigning them to more specific categories.
when presented with 32x32 pixel image of car, the model's prediction
Instruction and Reasoning Enhance Performance on SQAimage Data Following LLaVA [15]. we conducted experiments on the SQAimage dataset using different inference approaches, including prompts with without reasoning instruction. The prompts with reasoning make the MLLM

---

bounding
oblects image:
wings spread The bird appears to be seagul, with white body and black wings. provide precise coordinates, box would be (0.356, 0.477, 0,y6,, o.s2g).
Resporse: flying over
bounding
Instructio:: Give the position of all the objects in the image.
CIFAR1o
Autoevolution
Instruction: What the calegoay for this image?
Resporse: the image, there a birdd flying over the water. The
bird is flying low water, with beak as ifit the
Ground Truth:
process catching a fish. The bird's position approximately [o-55, o.5tl.
(a)
Responte: Flat
(b)
Instruction: What is caption of image?
Ground
male valleyball player in red trunks tike to black the shot the opposing player during a professional
volleybal! game.
Men playing volleyball, with one player missing the ball but hands still in the air.
3. group of spectators
sand vaileybali game. Volleyball players compete on sunny day the beach.
Men playing valicyball sand.
LLavA's Response:
The image shews group of people playing game of volleyball on beach. They are actively engaged the garne, with man jumping in ball. The lively energetic, capturing spirit outdoor sports and activitiss
Response:
The image capoures an engaging scene from a professional sand volleyball geme. Two men can be seen in the center of the court, fully immersed game, Bath players dressed casual sportswear, with wearing write shirt the other shirt The sand court stretches across the image, providing ample space for the players to move are und. There are multiple flags scattered across the light-colored backdrop the lively
scene, likelly se serving markers boundarles the game. In the brougd the provides action on the court.
(c)
Figure 4: Observation and analysis various tasks. (a) Visualization results VOC2012. (b) Visualization results on CIFAR1O. The right subfigure is from [54]. (c) Results on Flickr30k.
0.4
Low
High
Ca Added
40.500
Acc on SQAimage
39.500
Acc on A0ge
Added
38
37-330
0.2
0.350
Detaa Desic
35
35-440
0.00
0.00
w.R.
W/o R.
WR
Wic R.
W inst.
inst.
wio inst.
w/o inst.
0.0
LLaVA
MiniGPT, mPLUG owl
(a)
(b)
(c)
(d)
Figure 5: (a) Zero-shot Accuracy of LLaVA with different inputs on SQAimage. R. indicates
reasoning and inst. indicates instruction. (b) Counting Performance FSC147 of MILMs. (c) Zero- shot accuracy of LAMM model trained on various data combinations on SQAimage. (d) Zero-shot accuracy of LAMM model trained additional instruction data in our dataset.
output the reasoning process before presenting the final results. The prompts with instruction give MLLM the task definition and output structure to the question help the model better understand the task. The results in Figure 5 (a) shows that the instruction and reasoning both improve the MLLM's VQA ability. These results highlight the importance of incorporating task-specific information and reasoning process into MI.LMs.
Difficulty in Comprehending Visual Information for Domain Shifted Data. We conducted an analysis on several datasets that exhibit significant deviations from the training dataset, including UCMerced, CelebA, and LSP. The UCMerced dataset consists of top-down views of scenes, CelebA a facial dataset that can describe the expressions and hair colors, and the LSP dataset involves 14 key points of the human body, they significantly different from the COCO dataset during the training phase. These results suggest that the performance the MLLM model may degrade significantly datasets that exhibit significant deviations from the training dataset.
Difficulty in Reading Text on SVT data. We analyzed the performance of our baseline model on the SVT dataset and observed unsatisfactory results in Table 1. A possible explanation is that used the

---

TextVQA [55] dataset to generate visual task dialogue, which is more geared towards conversational text rather than OCR-related vision tasks. This mismatch in dataset characteristics may have resulted in suboptimal generalization of our model to the SVT dataset. To address this issue, we intend to conduct further investigations and incorporate more appropriate OCR data during the training process to improve our model's 's performance on OCR-related vision tasks.
Data volume validation on SQAimage data. As shown in Figure 5 (c) (d), our four types of image instruction tuning datasets outperform LLaVA[15] on all subsets, resulting in 7% overall performance improvement for the complete dataset. Furthermore, we investigated the impact of sampling Daily Dialogue and Detailed Description data at different proportions. Notably, even with the small size of 10k examples, our dataset achieved comparable results LLavA-Dataset As the dataset size increased, the overall performance of our model continuously improved, indicating that our dataset scalable and can be further optimized by adding more data.
Limitations
In this part, we discuss limitation and social impact of this work from perspectives of dataset, benchmark and framework.
Dataset In our study, utilized GPT-API, a taite--ff-heeeeee language model, to generate the multi- modal instruction data. To achieve the desired format, which includes multi- dialogue and one-round detailed descriptions, we provided system messages and example dialogues guidance for the data generation process using GPT--PI.. The use of GPT-API for generating text-based conversations has been widely adopted in Natural Language Processing, and previous work in multi-modal data [8, 15, 16] has demonstrated promising results in various tasks.
However, it is important to acknowledge the limitations inherent to the underlying GPT model, which are not altered by the use of GPT-API. GPT-API lacks direct access to visual information and relies solely textual context such captions and attributes, which restricts its understanding of images and may result in missing detailed information. While GPT-API excels at generating coherent and contextually relevant responses, it can occasionally produce responses that appear plausible but are factually incorrect or lack proper context. It may also struggle with understanding complex or ambiguous queries. Moreover, the generated data used for training may inadvertently reflect inherent biases and other truthworthy issues of GPT-API. To address ethical concerns regarding data generated with GPT-API, we performed manual sampling to examine the data, ensuring that the generated data aligns with societal values, privacy, security, toxicity, and fairness requirements and expectations. In Appendix, we provide an evaluation of the data quality and showcase additional data samples. We also transparently provide the complete prompts used to invoke GPT-API, ensuring transparency throughout our work.
Benchmark LAMM evaluates MLLMs on formatted computer vision tasks and datasets. Due to the diversity of language models' outputs, metrics may fluctuate across experiments. Additionally, LAMM currently adopts metrics such as GPT-eval and binary localization as an initial attempt to eval- uate MILMMs' performance. Further research is needed enhance the stability of benchmark results and design more appropriate metrics, which can be a promising direction for future investigations.
Framework Our work establishes simple MLLM framework to build up a baseline model for our dataset and benchmark. However, there is potential for further development and careful design of MLLMs for future work to enhance their capabilities and performance.
7 Conclusion
In conclusion, our work presents LAMM, an open-source endeavor in the field of multi-modal large language models. We introduce the image and point-cloud instruction tuning dataset and benchmark, aiming to establish LAMM as thriving ecosystem for training and evaluating MLLMs. We also provide an extensible framework to facilitate the extension of MLLMs to additional modalities. Our research showcases the effectiveness of MLLMs in handling visual modalities, including images and point clouds, and highlights their potential for generalization via instruction tuning. By making our codebase, baseline model, instruction tuning dataset, and evaluation benchmark publicly available, we aim to foster an open research community for MLLMs. We believe that our work will contribute to the advancement of MLLMs and the development of purpose multi-model agents.
10

---

Acknowledgement
This work is done during Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi and Dingning Liu's is in- ternship at Shanghai Artificial Intelligence Laboratory. This work is supported in part by the National Key R&D Program of China (NO. 20222D0160100), and National Natural Science Foundation of China (62132001).
References
[1] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. 2,
[2] Hao Fu, Yao; Peng and Tushar Khot. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu's it's Notion, Dec 2022.
[3] Yaru Hao, Haoyu Song, Li Dong. Shaohan Huang, Zewen Chi, Wenhui Wang. Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06. 063366 2022.
[4] Shaohan Huang. Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023. 2,
[5] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation of language models. arXiv preprint arXiv:221 1.09/ 10, 2022.
[6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efffccintt foundation language models. arXiv preprint arXiv:2302.13377 2023. 2,
[7] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023.
[8] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cepstruction-folloiing Ilama model. https:/// // / thub. com t su- lab/ stanford_aaaaaaaeo
2023. 2, 3, 10
[9] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 2, 6, 23, 24
[10] OpenAL. Gpt-4v(ision) system card. htttss://cdh.ppene http:/chh.ppppai..ccmpaappppeeee wwt.c/oooppppccceeeco
Sysstm_Cardd.ppe 2023. 2
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[12] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx. Michael Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunitics and risks of foundation models. arXiv preprint arXiv:2108, 2iti::2008.02588 2021. 2
[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8788 8763. PMLR, R, 2021. 2
[14] Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Contin- ual learning in deep neural networks. Trends in cognitive sciences, 24(12):1028- 1040, 2020.
2
11

---

[15] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
2, 5, 7, 8, 10, 15, 21, 23
[16] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprini arXiv:2212.
2022. 2, 3, 10
[17] Yuanhan Zhang. Qinghong Sun, Yichun Zhou, Zexin He, Zhenfei Yin, Kun Wang, Lu Sheng, Yu Qiao, Jing Shao, and Ziwei Liu. Bamboo: Building mega- scale vision dataset continually with human-machinn synergy, 2022. 2
[18] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint drXiv:2303.0667,, 2023.
[19] Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. /1381, 2023. 3
[20] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08123 2023. 3
[21] Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint wrXv::2303.17580. 2023.
[22] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736 2022. 3
[23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[24] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint 2303:323003.3337,, 2023. 3
[25] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing dision-lanuuaa understanding with advanced large language models. arXiv preprint 2304 10592, 2023. 5,
[26] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang. Anwen Hu, Pengeheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality. 2023. 5,7
[27] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arX3v:2333.16/99 2023. 3
[28] Tao Gong, Chengqi I Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.047900 2023.
(29] Bo L.i, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.033226 2023.
3
[30] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0r 01652, 2021.
12

---

[31] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022. 3
[32] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang. Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. Opt-iml: Scaling language model instruction meta learning through the lens of generalization, 2023.
[33] Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Improving multi- zero-shot learning via instruction tuning. arXiv preprint
2022. 3
[34] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo, March 2023. 3
[35] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thougll prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 2022. 5
[36] Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics.
[37] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
2021. 6
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748-8766 8763. PMLR, 2021. 6, 24
[39] Xiaoshui Huang, Sheng Li, Wentao Qu, Tong He, Yifan Zuo, and Wanli Ouyang. Frozen clip model is efficient point cloud backbone. arXiv preprint 22ti::2212.00099 2022. 6
[40] Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv:1706.0244 2017. 6
[41] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook
of Systemic Autoimmune Diseases, 1(4), 2009. 7, 21
[42] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-
www..rr///ccaaleeeeeeoeeoe wtweoe/2212wwww/hooo//eee h/workkshp//indxx...iie html. 2.
[43] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wer Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Process- ing Systems (NeurIPS), 2022. 71
[44] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth A dozen images. CoRR, Abs//600.00736, 2016. 7, 21
[45] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014. 71
[46] Yi Yang and Shawn Newsam. visul and spatial extensions for land-use clas- sification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pages 270-279, 2010. 7, 21
13

---

[47] Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai. Learning to count everything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 21
[48] Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In 2011 International conference on computer vision, pages 1457 1457-1464. IEFE, 2011. 7,21
[49] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018. 7, 22
[50] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In British Machine Vision Conference, 2010. 7, 22
[51] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Puttern Recognition (CVPR), IEEE, 2017. 8, 23
[52] Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Computer Rision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX, pages 202-221. Springer, 2020. 8, 23
[53] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 8, 23
8.9
[55] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019, 10
[56] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context, Jan 2014. 15
[57] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. CoRR, abs/1602.07332, 2016. 15
[58] Xu Yan, Zhihao Yuan, Yuhao Du, Yinghong Liao, Yao Guo, Zhen Li, and Shuguang Cui. Clevr3d: Compositional language and elementary visual reasoning for question answering in 3d real-world scenes. arXiv preprint arXiv:2112.11669 2021. 18
[59] Johanna Wald, Armen Avetisyan, Nassir Navab, Federico Tombari, and Matthias Niessner. Rio: 3d object instance re-localization in changing indoor environments. In Proceedings IEEE International Conference on Computer Vision (ICCV), 2019. 17
(60) Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scene- graphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7515-7525 2021. 18
[61] https: http::///e..kkkiieeiss.oo///////
[62] http:://eobbreppt..ccca///ttt//s 24
httpps//e...wiiipdddaaoooxhhssss htapsesauuttMMtteeee_e 24
[64] httpps///ennukkkppldaaagg//hh//p 24
14

---

Appendix
A Overview
Dataset and code in LAMM has been open sourced at LAMM. In this Appendix, we present construction pipeline and more examples of our dataset in Sec. B. Then, Sec. C shows details of benchmark and related evaluation metrics. Sec.D presents implementation details of our framework. Training model based on our framework takes about 20 A100 GPU hours. Finally, more examples and results are visualized in Sec. E.
com/ www/ OpMMLAMM/
B Dataset
The paper introduces novel method for constructing instruction tuning data, which represents an innovative departure from traditional techniques that rely solely on daily dialogue and detailed description. Instead, our dataset leverages additional factual knowledge extracted from Wikipedia to improve the quality and diversity of the training data. In addition, we also explore the use of traditional vision task data, covering common tasks in both 2D and 3D fields, which is converted into instruction tuning data for training purposes. By combining our new data construction method with traditional vision task data, aim to improve the accuracy and effectiveness of Instruction-tuncd models in various vision-related applications. Specifically, we delve into the design of 2D and 3D portion of our dataset in Section B.1 and B.2, respectively. We also outlined the manual approach for checking the quality of the generated data in Section B.3. Finally, we provide comprehensive explanation of the license and social impact information of our dataset in Section B.4.
B.1 Image Instruction Tuning Dataset
Cl: n-round Daily Dialogue & C3: I-round Detailed Description. The first step of our approach involves incorporating more visual information, such as visual relationships and fine-grained cate- gories as input to GPT-API, providing dense visual context to the generated responses. To construct the Cl: n-round Daily Dialogue and C3: I-round Detailed Description data, we use the COCO images [56]. similar to the LLaVA [15] approach. However, we further extract object attributes and relationships from the Visual Genome dataset [57] to emphasize fine-grained and dense information in the generated responses. Specifically, Our approach leverages image scene graph information to provide a structured representation of the objects and their relationships within the image. By doing so, we generated multi-modal dialogue data that enables us to capture the relationships between objects the image and generate more accurate and natural language instructions. Figures 6 and display the messages utilized to generate daily dialogue and detailed description data in the GPT-API. Additionally, Figure 8 provides detailed examples of the generated results for both types of data.
C2: n-round by incorporating 42K classes of knowledge graph facts from Wikipedia using the Bamboo dataset. This addition enables MLLMs to generate frestion-answeriig which is a valuable addition to the dataset. To generate C2: n-round Factual Knowledge Dialogue data, we utilize the Bamboo dataset and Wikipedia to obtain relevant information, and then use GPT-API to generate a dialogue based on the given content. Specifically, we extract the QID labels and their corresponding Wikipedia descriptions from the Bamboo dataset to generate instruction tuning data. This approach allows us to incorporate common sense knowledge into the dataset, thereby enhancing the ability of MILMs to generate responses that draw upon broader range of factual knowledge. The messages used to generate factual knowledge data in the GPT-API are presented in Figure 9, while Figure 10 showcases detailed examples of the factual knowledge data generated by these messages.
Factual Knowledge Dialogue. In the second step of our approach, we expand the dataset
data related to factual knowledge,
C4: I-round Visual Task Dialogue. In addition to the three types of data discussed earlier, we also incorporate established computer vision tasks, such as image classification, object detection, keypoint detection, OCR, and object counting, into our dataset. This enables MLLMs to handle traditional computer vision tasks and generate responses that incorporate both language and visual information. The typical computer vision dataset consists of a set of images or videos, along with their corresponding labels or annotations that represent the desired output of the computer vision task, such the class of objects present in the image the location of an object. However, these discrete
15

---

messuges =
[l "You are Al visual assistant that can analyze single image. You receive five sentenees, each describing the sume image you observing. In addition, specific object locations within the image are given, along with detailed coordinates. These coordinates are in the form of bounding boxes, represented as (x1, yl, x2, y2) with floating numbers ranging from 0 to I. These values correspond to the top left x, top left y. hottom right x, and bottom right y. It is worth noting that Attributes and Relationships of different objects are also given.
You need generate n-round Daily Dialogue data between two people. The answers should be lone that visual Al assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers. Include least one plausible question about the image, and provide the answer in detail. To answer such questions, should require first understunding the visual content, then based the background knowledge or reusoning. either explain why the things are happening that way, or provide guides and help to user's request. Make the question challenging by not including the visual content details in the question that the needs reason about that first.
Instead of directly mentioning the bounding box coordinates, the captions and relations, utilize those data to explain the scene using natural language. Include questions asking about the visual content of the image, including the object types, counting the ohjects, object actions, object locations, relative positions between objects, etc. Always answer as if you are directly looking at the image. "3]
for sample in fewshot, samples:
messages.appendf ("role":"user "user", "content" "sample| (oontllttoooott] !) messages.appendi ""role":"assista"" "content" "rontent"ssamplee" (respposssss !)
aessaggss.apeende (" "usse""""s""",, "content": "ut join(query)))
Figure 6: Messages used to construct n-round Daily Dialogue data for image instruction tuning.
messages [f "You are an visual assistant that can analyze single image. You receive five sentences, each describing the same image you are observing. In addition, specific object locations within the image are given, along with detailed coordinates. These coordinates are in the forn of bounding boxes, represented as (xl, yl, x2, y2) with floating numbers ranging from O to 1. These values correspond to the top left x, top left y, bottom right x, and bottom right It is worth noting that Attributes and Relationships of different objects are also given.
You need to generate a I-round Detuil Description data. You cun use the provided caption, attributes, relationships und bounding box information to describe the scene in a detailed manner.
Instead of directly mentioning the bounding box coordinates, utilize this data to explain the scene using natural language. Include details like object counts, position of the objects, relative position between the objects. When using the information from the caption and coordinates, directly explain the scene, and do not mention that the information source is the caption, relationships or the bounding box. Always answer as if you are directly looking at the image. You can include multiple paragraphs if necessary:")) *]]
for sample in fewshot samples:
messages.appendt "role"""user"" "content" ssmmeeeeeoplllllle })
append ""sule"""assistan""" "content" "oonieen"ssampleereeett "ssmmpe["rsppuss]]]] })
upssed) "uuser"""",er", "content":"n" join(qurr)))
Figure 7: Messages used to construct I-round Detailed Description data for image instruction tuning.
results are not suitable for large language model dialogues, as they do not allow for natural language interactions. To address this issue, our proposed approach involves converting computer vision tasks, such as image classification, into natural language dialogues to enable large language models to perform these tasks through dialogue interactions. In detail, we first use GPT-API to generate a template pool of questions and answers for each task. Then, we randomly select a pair from the question template pool and answer template pool to combine with a piece of data from the computer vision dataset, creating the C4: 1-round Visual Task Dialogue data. Figure 27-30 provide some examples of the dialogues generated using our proposed approach for converting computer vision tasks into natural language dialogues. This approach allows us to leverage the rich visual information in traditional computer vision datasets and incorporate it into the instruction tuning process, thereby enhancing the ability of MLLMs to understand and respond to natural language instructions related to these tasks.
In summary, the construction of 2D part in our dataset provides a comprehensive and diverse samples of real-world scenarios, incorporating fine-grained and dense information from object relationships and factual knowledge sources. The dataset contains 186K unique language-image instruction- following samples, including 49K in daily dialogues, 49K in detailed descriptions, 42K in factual Knowledge dialogues, and 46K in visual task dialogues. Our experiments showed that the use of GPT-API consistently provides higher-quality ciatucction-oolowiin data, such as spatial reasoning.
16

---

Captions: IA clock tower with church steeple and bell. clock tower has fenced bell on top. A very tall clock tower with large bell sitting above it. The cross, bell and clock on the tower over a church. There is tower clock on the wall] bell: rusty, [0.4, 0.23, 0-57, 0.33]. clock: white, black, [o.18, 0.65, 0.77, o.8g]. building: white, tall, rusty, [o.14, 0.0g, 0.9, 0-99).
Objects:
sky. blue, (a.az, 0.0, 2.9, 0.7).
n-roud Daily Dialogue
Instruction: cineaiion.cmmaees What kind of building has clock tower in the image?
Response: In the image, there is a church with a clack tower.
Instruction ls there anything special about the clack tower?
Response: Yes, there a large, fenced-in bell on top of the clock tower, which makes unique. Instruction What color is the clock on the tower?
Response: The clock
tower is white with black numerals and hands.
Instruction: Can
describe the appearance of the sky in the image?
Response: : The sky in the image is blue.
Instruction What role might this clock tower and hell play in the local ity
Response: The clock tower and bell might play significant role the local community by serving as landmark and focal point of the town or neighborhood. The clock tower, with its large, visible flockface, helps the residents keep track of time and maintain their daily schedules. In addition, the bell might be used to signal important events, such as religious services, weddings, or funerals, could also serve reminder of community values and traditions, as well symbol of unity and shared history.
C3: z-round Detailed Description
Instruction: : <image> What can you observe in this image?
The image presents tall, white clock tower that stands out against a clear blue sky. The clack face prominently visible, featuring white background and black hands, Above the clock, fenced-in, rusty bell can seen, adding of history to the tower. Furthermore, the tower also displays cross at the top, signifying its connection to church, The combination of the cross, bell, and clock creates striking visual that showcases the tower's architectural and historical significance.
Response:
Figure 8: Example of generated n-round daily dialogue and I-round detailed description data.
messages - YouYou are an Al visual ussistant that can andlyze single image, You receive a cutegory label for an image and its associuted English description describing the sume image you are looking at. The cutegory informution is the category label of the image you are seeing. In addition, The English description is detailed semuntic information for the given category.
You need to generate a n-round Factuul Knowledge Dialogue data bused on the received category lable und its English description. Pleuse ask questions uround the given cutegory information and its corresponding description information, und don't ask questions that cannot be reasoned based on the given information. Questions and answers should be concise and effective, with reasoning logic. Answer all questions as you are sceing the image.
The answers should be in a tone that u visual Al assistant is secing the image and answering the question. Ask diverse questions und give corresponding answers. Include questions asking ubout the visual content of the imuge. Only include questions that have definite unswers: (1) one cun see the content in the image that the question asks about and can answer confidently; (2) one can determine confidently from the image that is not in the image. Do not ask any question that cannot be answered confidently. Also include complex questions that are relevant to the content in the image. for example, asking about buckground knowledge of the objects in the image, usking to discuss about events happening in the image, etc. Again, do not usk about uncertain details. Provide detailed answers when unswering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-orgunized. You can include multiple paragraphs if necessary "]]
for sample in fowshot tewshot_sumpless
messages.append( ( "user""" "content" "content":sumple[ (captl//clcctttt]] 3)
(ppeend ( ""role""uusistumm", "comtent"ssappp[[ "ssampe[[respoose"" ]
gessaggssuppendd ( ""role":" userr, "content":"tt join(query)))
Figure Messages used construct Factual Knowledge Dialogue data for image instruction tuning.
These features make our dataset a valuable resource for researchers and practitioners working in the computer vision and natural language processing fields.
B.2 Point Cloud Instruction Tuning Dataset
The construction pipeline of point cloud instruction tuning data is similar to that of image instruction tuning data. However, due to the limited availability of 3D data, point cloud instruction tuning dataset only consists of three major components: n-round plain conversation and l-round detalied description data from GPT-API and 1- --round visual dialogue data converted from datasets for existing 3D vision tasks.
Cl: n-round Daily Dialogue & C3: I-round Detailed Deseription. To construct the Cl: n-round Daily Dialogue and C3: 1 1-round Detailed Description data, we choose point clouds from 3RScan [59]
17

---

Category: Women in Belarus.
Description: The modern-day characteristics of women Belarus evolved from the events that happened in the history of Belarus, particularly when the "concept of equal rights for
first developed and substantiated in the late 16th century". The so-called led Grand Duchy Charter of 1588 one of the most important legal documents Belarusian history protected the dignity of Belarusian women under law. Women in Belarus and their contribution to Belarusian society is celebrated annually.
C2: n-round Factual Knowledge Dialogue
SBD
ROGGUE
Instruction <image> What significant historical event marked the development of equal rights women Belarus?
Response:
The development of equal rights for women in Belarus was significantly
marked by the "Grand Duchy Charter of 1588", one of the most important legal documents
in Belarusian history, which protected the dignity of Belarusian women under the law.
Instruction: When is the contribution of Belarusian women to theirsociety cele ebrated annually?
Response: : The contribution of Belarusian women to their society is celebrated annually on the 8th of March, during International Women's Day.
Instruction: How have the modem-day modem-day.characteeess charcctriiist of women Belarus been influenced?
Response : The modern-day characteristics women in Belarus have been influenced by the events that have occurred in the history Belarus, particularly those related to the concept of equal rights for women.
Instruction Is there specific legal document that stands out the history of women's rights Belarus
Response: Yes, the Grand Duchy Charter of 1588 stands out significant legal document the history of woree'' rights in Belarus. This charter protected the dignity of Belarusian women under the law.
Figure 10:
The example for constructing n-round Factual Knowledge Dialogue data. The description from Wikipedia page.
messages = You are an expert in linguistics. please help me to turn these questions and answer sentences into declarative sentences, there are many question-and-answer sentences, each will be separated by ***, these is example: Are there the same number sofas and wide sinks? no
How many objects are tall beds or tall square beds? 1
There is a blue ottoman that is on the left side of the tv stand; is its shape the same as the brown tv stand? yes
There is a couch that is close by the armchair that is lying on the low white cushion; what its color? brown
the results of the two question-answer sentensetnencsss I hope you return is:
There are not the same number of sofas and wide sinks.
There is 1 object is tall bed or tall square bed.
There is a blue ottoman that is on the left side of the tv stand and its shape is the same as the brown tv stand.
There is a couch that is close by the armchair that is lying on the low white cushion and its color is brown.
Please do not add 'es or no' word at the first of the response sentence."")
Figure 11:
Message to transfer visual question answering annotations from CLEVR3D [58] to declarative sentences for 3D data.
as data source and use its original 3D bounding box annotations. Since there is no caption annotation for 3RScan, we input visual question answering (VQA) annotations from CLEVR3D [58] to GPT-API and ask it to convert the Q&A data into declarative sentences, which serves point cloud captions in further steps. Figure 11 shows the corresponding prompts. Object attributes and relationships are extracted from scene graph annotation in 3DSSG [60]. Figure 12 and 13 show the prompts to let GPT-API generate daily dialogue and detailed description data for point clouds. Since full annotation of scene point cloud may easily exceed input token limits of GPT-API, we randomly selected 10 captions and keep bounding box and relationships of corresponding objects as input contexts. For GPT-generater data, We limit the number of turns in each dialogue data to more than 10, and any data exceeding this limit will be split into different samples. Figure 14 shows an example of GPT-generated data.
C4: I-round Visual Task Dialogue. On the other hand, we also leverage annotations for existing 3D vision tasks, such as point cloud classification, 3D object detection, and CLEVR3D for 3D VQA. Similar to 2D datasets, we designed 15 templates for instruction and response by sending definitions of the corresponding tasks to GPT-API. Then instruction data are formulated by replacing keywords with corresponding annotations. Templates of 3 tasks involved are presented Figure 31, 32 and 33, respectively.
18

---

messages (IuYou Al visual assistant that analyza lyze single point cloud. receivee five sentences, each describing the point cloud you are abserving. In addition, specific object locations within the point cloud are given, along with detailed coordinates. These coordinates are in the form of bounding boxes, represented as (cx, cy, cz, ls, ly, lz) with floating numbers in unit meters. These top y, coordinaatss of bounding box ceter and length of bounding box along % Y, It worth noting that Attributes and Relationships of different objects are also given.
values correspond
Generate some questions and answers of the scene in detailed manner. Instead of directly mentioning the bounding box coordinates captions given, you shou utilize data explain question using natural language, Include details like the scenarin, object counts, position ofthe objects, relative position hetween the objects When using the information from the caption and coordinates, directly explain the scene, and do not mention that the information source is the caption, relationships or the bounding Answer questions descriptions if you really saw the whole scene, using the tone of seeing the scene to ask questions answer. Please ask questions around the given category information and its corresponding description information, and don't ask questions that cannot be reasoned based on the given information. Questions and answers should be concise and effective, with reasoning logic. Answer all questions as you are seeing point seeling the point cloud and answering the question.
answers should
tone that visual assistant
Ask diverse questions and give corresponding answers. Include questions asking about the visual content of the point cloud. Only include questions have definite answers: can the content point cloud that the question asks about answer confidently. All descriptions are attributes or relationshiis with other objects, which does not mean that there are people in the scene:"]] "I]
sample fewshot samples:
messages. agpendif"role"""uee" "content" "ppleen""semmlll"cooo mple'ccoeett]] apssaggesappendd(todee " "content":samplee sapptettssmmpllleessosssss] messages.appendiT" "userr" "content": An' jointquur))))
Figure 12: Message to generate n-round Daily Conversation Dialogue data in 3D portion of our dataset.
messages- - You are an visual assistant that can analyze a single point cloud. You receive five sentences, each describing the same point cloud you are observing. addition, specific object locations within the point cloud are given, along with detailed coordinates. These coordinates are the form of bounding boxes, represented as (cx, cy, cz, by, ly, lz) with floating numbers in unit of meters. These values corresponc to the top x, y, z coordinates of bounding box ceter and length of bounding box along x y, z axis. It is worth noting that Attributes and Re Relationships of different objects are also given.
You need use the provided caption, relationships and bounding box information to describe the scene a detailed manner. Instead directly mentioning the bounding box coordinates or captions given, you should utilize this data to explain the scene using natural language. Include details like the scenario, object counts, position of the objects, relative position between the objects. When using the information from the caption and coordinates, directly explain the scene, and do not mention that the information source the caption, relationships the bounding box. Answer questions or descriptions as you really saw the whole scene, using the tone of seeing the scene to ask questions or answer. Always answer as if you are directly looking at the point cloud. You can include multiple paragraphs necessary. Every question and answer must be related. "I] for sample in fewshot_ss saamhllsssssss: messages.appente role"" "'onten""sammle[[conte"""" })
apppedd) "aasssststtt,"""" saapeetttssmmmpeeeepppoeeee appendi( role": "user", "ronteet":"1t-jjonnqquee"
Figure 13: Message to generate I-round Detailed Description data in 3D portion of our dataset.
Finally, 3D portion of our dataset contains 10K samples in total, and the number of ShapeNet, 3RScan detection, CLEVR3D, and GPT-generated dialogue are 2K, 1.3K, 2K, and 4.9K, respectively.
B.3
Quality Check
In order to ensure the quality of the generated instruction tuning data, we implemented several measures. Firstly, we generate small amount of data cold start and conduct manual check on the generated data. This involved carefully assessing the quality and making necessary adjustments to the message information provided as input to GPT-API. The iterative process aimed to eliminate ethical concerns and establish a strong correlation between the generated data and the corresponding inputs. We repeated this process until the desired level of quality was achieved. Once satisfied, we proceeded to generate a large volume of data. Furthermore, to verify the quality of the generated dataset, we randomly select a subset of 10% data for manual checks. This step allowed us to evaluate the generated data against our specific requirements and quality standards. During this evaluation, any formatting issues or incorrect answers generated by GPT-4 were filtered out to ensure the usability and reliability of the data. By combining manual checks during the iterative generation process and
19

---

Captions:
[There is tall box and there are 6 boxes lower than it. The table that is made of same material as the tall box is narrow. door. chair that same color as the glass window tall. There are 7 objects
There are ş boxes that the same material that either wide boxes higher than the low box big shelves.]
Objects: floor: trred,,tled,,ppp placing items on, sarne shape lab_name,, [-2.25, 2.83, -1,59, 0.26, 7.81, 0.4]. shelf: placing items on, hanging, big, right box,same object type shelf, [-0.27, -2.12, -1.48, 0.16, 2.4, 1.33]. boxes: moving,placing items in,placing items [-1.21, 0.23, -1.46, 0.85, 0.52, 0.93]
Ca: n-round Daily Dialogue
Instruction:: : spoint cloud> is material of tall
point cloud?
Response: The tall box in the point cloud is made of the same material the table, but the captions do not give information on the exact material.
Instruction "How many boxes,
the point cloud?
Response: There total of boxes the point cloud, with of them made of the same material as the door and of them lower than the tall box.
Instruction there more desks windows
point cloud?
Response:
The captions mention that there aren't more desks being supported the table than windows in the point cloud.
C3: 1-round Detailed Description
Instruction spoint clouds Can you describe the scene in this cloud?
Response: : The point cloud consists of room with tiled flaoring. There are four walls with two windows supported by the leftmost wall different heights. Besides the walls, the point cloud shows shelves four walls,, varying in size and position, with boxes placed on and behind them. There is also narrow table made of the same material as the tall shelf with boxes placed on A single door hangs on the wall, and no other objects share the identical shape. There is no matching number of doors and shelves. Nine boxes made of the material
door.
Figure 14: Example of GPT-ggnerraed n-round daily dialogue and 1-round detailed description data in 3D portion of our dataset.
subsequent random manual checks on the final dataset, we strive to ensure that the generated data meets our rigorous quality standards and aligns with the specific needs of our dataset.
B.4 Social Impact
Our dataset is compilation of publicly available datasets that have been licensed under the Creative Commons license (CC-BY). We have taken great care to follow all necessary legal protocols to use this data in our research, and believe that transparency in data licensing is crucial for ensuring proper attribution and appropriate use of the data. Besides, the dataset includes images sourced from publicly available datasets and language data generated using the GPT-API. While we have taken steps to ensure appropriate content, acknowledge that problematic content may exist. If you encounter any such content, please notify us immediately, and we will make necessary modifications to maintain high-quality dataset that is free of inappropriate content. To protect the privacy of individuals and vehicles captured in the images, we plan to obfuscate sensitive information, such as faces and license plates, before publishing the dataset. We are committed to maintaining a dataset that is both high-quality and ethically responsible and pledge to uphold principles of privacy and transparency in our work.
C Benchmark
C.1 Benchmark on image tasks
We selected a set of nine commonly used CV tasks to evaluate the performance of MILMM models in our benchmark on image tasks. Our task selection criteria were based on widely studied tasks in the CV field that can showcase the MLLM model's abilities in visual interpretation, localization, and qnestion-anssering
Table provides a summary of the tasks and the corresponding common evaluation metrics, which are based on the output that the MLLM models are required to generate for each task. We utilized a prompt-basee based approach to instruct the MLLM models to understand the task definition and generate the desired output. The ability of the models to understand and interpret the given instruction was also evaluated as part of the assessment criteria. As the models' outputs are text, we use different text-processing techniques for each task to extract entities as the final answers for evaluation. For each task, We selected datasets that are distinet from the training datasets, as our benchmark evaluation is conducted in
zero-shot setting.
20

---

Table 5: CV tasks in Our Benchmark
Task
Output
Metrics
Classification
label name
Acc
Detection
list of object label and bbox option and answer
mAP50
VQA
Acc
Image Caption
captions
BLEU4
Fine-grained classification
fine-grained label name
Acc
Object counting
number
MAE
OCR
list of words
word Acc
Facial classification
answer
Acc
Keypoints detection
keypoints
PCK
3D Detection
list of object label and bbox
mAP50
3D VQA
option and answer bbox
Acc
3D Visual Grounding
MAP50
Classification This task involves predicting the most likely category label for an image. For MLLM models, the task involves performing open-vocabulary classification. We selected CIFAR-10 [41] as the test dataset for the evaluation of classification. CIFARIO contains 10000 test images across 10 common categories. We utilize NLTK to extract noun entities from the models' output text, and expand them to a synonym set for accuracy evaluation calculation.
Object Detection We selected the VOC 2012[42] datasets to evaluate the model's ability to detect objects in images while considering both its visual interpretation and localization capabilities. To evaluate the accuracy of object category predictions, we employ similar approach to classification tasks. We also use regular expression matching to extract the models' output bounding boxes for mAP50 calculation.
Visual Question Answering We selected the ScienceQA[43] and AI2D[44] datasets evaluate the MILM model's ability to answer questions about images. The ScienceQA and A12D datasets include over 2017 and 5793 multiple-choice questions with images, respectively. We extract the image-containing data from the ScienceQA dataset to create the SQAimage dataset. We then tested MLLM models the SQAimage dataset evaluate their multimodal understanding skills. As both ScienceQA and A12D datasets are presented in multiple-choie format, we evaluated the model's s performance using the accuracy metric. Following LLa VA [15], we prompt the MLLM output the complex reasoning procession, followed by the final option answer.
Image Caption The image caption task involves generating textual description of an image. We selected the Flickr30k[45] dataset to evaluate the MILM model's ability to understand images and generate descriptive captions. Flickr30k contains variety of objects and scenes with diverse captions, providing a challenging task for the MLLM model. To evaluate the quality of the models' text outputs, we split the generated text into sentences and calculate the BLEU-4 score for each. The highest score is selected as the final result.
Fine-grained classification Similar the classification task, the fine-grained classification task requires the model to make predictions across a large number fine-grained categories. We selected UCMerced Land Use dataset [46] as the test set. UCMerced Land Use contains 21 classes of land-use categories, including airports, forests, and residential areas. Similar to classification, we report Accuracy.
Object counting We selected the FSC147 dataset for object counting evaluation. FSC147[47] is a dataset of 1190 images containing various objects, including animals, vehicles, and houschold items. The images in this dataset are challenging and contain occlusions and overlapping objects, making it suitable choice to test the model's object recognition and localization capabilities. We utilize regular expression matching to extract the numeric entity and evaluate the model's I's performance using the mean absolute error (MAE) metric.
Optical Character Recognition The OCR (Optical Character Recognition) task involves recognizing and transcribing text from images. To evaluate the MILM model's ability to recognize text from images, we selected SVT dataset [48]. We extract the entities enclosed quotation marks from the generated text as the predicted word list. Word Accuracy is adopted as the evaluation metric.
21

---

(System Message]
#AR Human: <vision> [Vision Tokens] </vision> [Instruction]
### Assistant: [Response
##* ANSWER: [Response
Figure 15: Template instructions for VQA inference. "Response a" is the generated reasoning process, which is the output of the first inference. "Response b" is the output answer, which is the ouput following the prompt "### ANSWER".
Facial Classification Due to the difficulty of performing face recognition tasks using MLLM, we evaluated the model's performance facial attribute classífication tasks. We selected the CelebA[49] dataset, which contains 19962 images for testing with annotations for 40 facial attributes, including hair color and facial expression. Specifically, we evaluated the model's ability to predict whether a person in an image is smiling, named CelcbA(Smile) dataset, and the color of their hair, named CelebA(Hair) dataset. We aimed to evaluate the MLLM model's ability to understand facial images. Classification accuracy is used as the evaluation metric.
Keypoints Detection To evaluate the models' ability to perform ine-grained point localization, we utilized the LSP[50] dataset for keypoint detection. To simplify the task difficulty for MLLM models, we employed grounding approach, where we sequentially asked the model to predict the position of each human body keypoints in the image. The evaluation metric used for this task was PCK (Percentage of Correct Keypoints).
C.2 Inference Details
C.2.1 System messages for image tasks
Figure 17 shows the system messages defined for each image task. The system messages, which include the task definition and the output structure, is part of the instruction that prompt the MLLM models generated responses. This is designed to enable the model better understand the task is performing, focus on the critical aspects, and output the appropriate structure. Note that some tasks do not require a defined output structure. In such cases, the model can output any text as response.
C.2.2 Instructions for VQA
Different from other common image tasks, besides the system messages designed C.2.1, we prompt MLLM generate the reasoning process additionally, as figure 15 shows. To prompt the model to output its reasoning process, we first use conventional instruction texts to generate "Response a". We then combine the first instructions. the "Response a", and the prompt "### ANSWER" make the model generate the option as the final answer.
C.2.3 Metrics
Our benchmark includes two evaluation settings. The first is a zero-shot setting, where we selected downstream tasks that have no intersection with the MLLM's training data. We provide the zero-shot results of the current MLLM models on these datasets. The second setting involves fine-tuning mainstream task datasets, covering tasks such as detection, classification, and VQA.
C.2.4 Binary Locating Metric
The ability to accurately localize objects in an image is a crucial component of MLLM models' visual understanding skills. In addition to using conventional detection tasks to calculate mAP, attempted more direct method for evaluating the models' localization ability, namely Binary Locating Metric. Distinct from object detection, which requires the model to output a bounding box, we instructed the model with "output the position of the object" instead of "output the bounding box of the object" to output the approximate position. During the evaluation phase, the model's predicted keypoint was considered correct as long as it was within the object's bounding box. Object locating is evaluated on all datasets involving object localization, including object detection, object counting, and keypoints detection. Compared to the traditional detection evaluation methods, the object locating evaluation
22

---

method provides a more reasonable and direct approach for evaluating the localization ability of MLLM.
C.2.5 GPT Metric
To evaluate the overall understanding and sreestion-answerte abilities of MLLM models, we utilized the GPT Metric. Unlike [A[5) and Vicuna [9], we ranked the answers of multiple models using GPT. Similar to the pipeline approach, we give GPT an instruction, informing of the task definition, the question, and the answer provided by each model. We then ranked each model's response based on its relevance and accuracy with the answer. Each model received a score based on its ranking, and the average score obtained on all test data served as metric for measuring the model's overall ability. Our GPT evaluation datasets cover various visual tasks, including captioning and VQA tasks involving image description and answering, well as small number of detection and counting tasks related to object localization.
C.3 Benchmark on point cloud tasks
For benchmark on point cloud tasks, we focus on three tasks of scene perception, including 3D object detection, visual grounding, and 3D visual question answering. Figure 18 presents system messages for point cloud tasks.
3D Object Detection. As it's widely used in 3D object detection, we select ScanNetv2 [51] as the dataset evaluate MLLM's ability to locate objects in point cloud and identify semantics, whose validation set contains 312 scenes. In this task, MLLM is expected to list all objects along with bounding boxes, and we extract bounding boxes from the response text by entity extraction. Boxes whose IoU with ground truth is larger than 50% count for positive predictions and we use mean Average Precision (mAP) to evaluate performance.
Visual Grounding. This task aims to locate the object described by a given caption and output the corresponding bounding box. We test on ScanRefer [52] in this task, which provides labenll captions towards each object in ScanNet and its test set contains 9508 samples. Similar with object detection, mean average precision (mAP) reported evaluate MLLM's capacity.
3D Visual Question Answering. ScanQA [53] is proposed for 3D visual question answering before, and models are required answer the given questions based on the point cloud. It has been formatted as an attribute classification task in previous work [53]. However, MI.L's output cannot be constrained with several classes consistently and is usually long text explain details, so the original metrics in ScanQA, Exact Matching & BLEU, cannot be used for test, as long text is different from the style of given ground truth and the BLEU score inevitably decreases for long-text results. Following ScienceQA in 2D VQA task, we transfer this task to be a multiple-choice problem. First, we feed the original question-answer answer pairs GPT-API and ask for 5 confusing options. Then MLLM is expected to choose the correct option or output the correct content. Thus, a metric of accuracy is used evaluate model performance.
Evaluation Settings Similar to evaluation for 2D tasks, our 3D benchmark includes two settings for evaluation. The first one is zero-shot setting. MLLM is trained instruction data from 3D portion of our dataset, whose point clouds come from 3RScan or ShapeNet and has no overlap with ones in downstream tasks. Furthermore, finetune the models trained our 3D datasets by training set of downstream tasks and reporting metrics on the corresponding test set.
D Implementation Details
In our experiments, 2D and 3D models are trained independently, and only the feature projection layer and LoRA parameters are optimized during training while LLM can be shared among tasks.
For all experiments, trainable parameters are optimized by Adam optimizer with a learning rate initialized to be 5e-4, and scheduled using a linear decay scheduler. We For 2D experiments, models are trained for 2 epochs. For 3D experiments, we increase the number of iterations to 10,000 in case of too few samples. We use A A100-80GB to conduct experiments. Each GPU process 2 samples every iteration and the effective batch size are set to 64 by gradient accumulation. For reference, 2D
23

---

[System Message]
### Human: <vision> [Vision Tokens] </vision> [Instructions]
### Assistant: [Response]
Figure 16: Template for multi-modal data pairs. Bold words stand for corresponding text data and italic words indicate fixed templates. < visiin >> & //vision > stand for start & end token for vision contents.
experiments at most last for about 8 hours for 186K samples, while 3D experiments require about hours.
Following Vicuna [9], we format multi-modal training data as Figure 16. [SystemM specifies the corresponding task of sample, [Query] refers to position of texts from human and [Response] refers to contents expected for LLM. The special tokens < vision > & < </ /vision represents start and end positions for vision content. We use Img >< /Img > and Pcl >< /Pd > in 2D and 3D datasets, respectively. The training objective used is next token prediction loss, and only text tokens [Response] count for loss computation. As we use CLIP [38] pre-trained ViT-Large- 14 as visual encoder, the number of vision tokens are 256 and length of text tokens after vision tokens are limited to 400 in training.
E
Demonstrations
E.1
Results on CIFAR10
Figure 19 presents some examples responses from model trained by our dataset on CIFAR1O, where the model's s answers were judged as incorrect in the evaluation, but in fact, our model provided more granular classification result. The left column shows the test images from CIFAR 10, and the right column displays the images of the objects that the model classified, including toad [61], Land Rover Series II [62]. Mirage 2000D fighter aircraft [63] and police car [64]. It is evident that the fine-graine objects classified by our model have very similar features to the input images, demonstrating its ability to perform fine-gratd tme-grained classification.
E.2 More detailed information on image caption
Our model performed poorly on the Flickr30k dataset in terms of BLEU scores. This is because model's 's responses include additional details that are not captured by the ground truth captions. Figure 20 illustrates this phenomenon, where the highlighted text red represents the matching ground truth captions, while the text in orange is not matched but is still relevant to the image content. It is evident that our model is capable of providing more detailed descriptions the image, which not captured by the traditional BLEU metric.
E.3 Comparison with LLaVA detection and counting tasks
We compared the performance of model trained by our dataset with that by LLaVA on both object detection and counting tasks. Figure 21 illustrates the comparison results detection, where the leftmost images represent the ground truth bounding box, and the rightmost images show the visualizations of the responses after entity extraction.
Although LLaVA was able to identify the approximate location of the object, it was unable to provide precise bounding box coordinates. On the other hand, our model demonstrated superior detection capabilities after fine-tuning on delection-related data and was able to provide more accurate bounding box coordinates. Additionally, our model also exhibited better counting performance, shown in Figure 22. worth noting that counting is essentially task that tests the model's localization ability.
E.4 Results of binary-loc metric and GPT metric
We present the results of our model and LLaVA on the binary locating metric in Figure (a), where our model demonstrates more precise localization abilities. The green points in the image

---

the visualization of the predicted key points. In the second row of the figure, our model outputs bounding box, which we break down two position coordinates (top-left and bottom-right) during entity extraction.
In Figure 24 (b), we show the evaluation results of the two models' image captioning responses using the GPT metric. The GPT metric considers our model's responses to be more specific and accurate compared to LLaVA, resulting higher ranking. These results further demonstrate the effectiveness of the model trained on our dataset in accurately detecting, locating, and describing objects in images.
E.5
More demonstration examples
Figure 23 shows the results of our model on VQA task and Figure 25 shows its example results 3DVQA task. Figure 26 shows the results on in-the-wild images.
25

---

Classification
salyo a single image provided with an image and szecifiee which human body parts the user want you to detect To generate content images mprehand meaning Your primary sbjective assistant perform classification task accurately reliably, information is cruc all for users make informed decisions ba on image data. To simply provid ing dapendable prezise.
given image, ensure
Please provice label Uhel eccuralely descr the subject of the imaye.
Detection
You are now parforming otject detection tack, and your goal is locate all instances of objacts in immgge, suth as people, sars, animmls, or olher objeds, end yive the correspordiigg coordin ales.
bounding boxes, represented baz, ye, x2, ya) flozting numbers rangi from to Thhse values carracaond to the top laft top laft bottom rightx, and botto righty.
ccordinates
VQA
performing question answering task, and your goal
aturallanguage answers that accurately solve the question. In order to generate accurate answers to cuestions about visuel you be understand the content of images, understand meening of questions, and perform corr plex reasoning processes.
Image Caption
As Al assistant, your primary task zerform image captioning, which requires you gansrate clear and concise natural language descriptio unde istand the visual content the image, identiff its salient features, and gener coherant and von conieellalllyy releveni sap ion thei acur ausraaele stely conveys ills meening.
content, achieve this, you must be
Generate descriptions of the visual context.
Fine-grained Classification
assistant, your any perform image captioning, which requires you generate alear and concise natural language de descriptions ns of the visual content. To achieve this, you must be able to understand Uhe contert thei mage, ideniify contextually relevant caption that accurately conveys its meaning.
feelures, generrale
Please provice fine-grained labe that accurately cescribes the subject of the ima.ge
Object Counting
an Al assistant, you are performing a object counting task. Your goal is to accurately count the number of objects image. Object counting rumber instances of specific obje cts within an image. You need to analyze the inpue image and accurataly count the number obj
puter task that involves catacting counting
precise
OCR
You are performing Optical Character Recognition task, which involves recognizing and estracting test from images. To generate accurate answers questions able to accurately reoog nize and extre text from images, and understed redarddtadddd the m meaning of questions.
content images, you must
Your be words
Facial Classification
performing an Character Recogn task. which involves recognizing extracting from images. To generate accurate answers to questions about the text content of mages, you must inteyes, and undersland meening quesiioo.
dble ebely resuynice
Keypoints Detection
visusa assistant
cay points.
answers, you must que stions
es yl Both are to the entire image.
express
indicating positicn
Figure 17: System messages for benchmark on image tasks
26

---

3D object Detection
You are now performing an object detection task, and you goal to locate ins nces objects point cloud, such as furniture, transportations, objectt,,, and give the corresponding coordinates.
These coorcinates are in the form of bounding boxes, represented as (ex, ey, a,, Is, ky. ls) with floating numbers of These values correpoond y, coordinates bounding box center and length of bounding box along axis.
3D Visual Grounding
You are able to understand the visual content that the user providas, and assist the user with language.
variaty tasks using
now marforming visual grounding task, your goal is locate the instances of objects want with the given description pointcloud such furn ture, or other objects, and give the corresponding coordinates
the form of bounding boxes, represented cy, cz, ly. with floating numbers un of meters. These values correspond x,y, coordinates of center of bounding box and langth of bounding box alongix, y, zaxis
These coorcinates
3D VQA
You are able to understand the visual content that the user provides, and assist the usar with variety using naturallanguaaa are now performing visual quastian answering task, and your goal generate natural language answers that accurately solve the question. order to generete eccurate answers to questions about visual content, you must be able to understand content of peint cloud, understand the meaning questions, and perform complax reasoning remplaxr
procasses.
Figure 18: System messages for benchmark on point cloud tasks
Instruction: What is the calegory for this image?
LAMM's Response:
Upon close inspection of image, can be concluded that the object inteeeest is toad.
(Ground Truth: "frog"?
Instruction:
What the category for this image?
LAMM's Response:
close inspection of image, be concluded that the object of the image is Land Rover Series II.
(Ground Truth: "automobile"
Instruction: What is category for this image?
LAMM's Response:
close inspection
image, concluded
object of the image is Mirage zoonD fighter aircral..
(Ground Truth "aiplaane")
Instruction: calegury for Uhis imeqe? LAMM's Response:
image object classified police car based on the descriptions provided.
[Ground "automobil")
(8)
(b)
(c)
Figure 19: Results of model trained by our dataset CIFAR10. (a) The images from CIFAR10 test
set. (b) The instruction, response from our model and the ground truth. (c) The reference images.
27

---

What caption this image?
(Ground Truth:
man standing in bar with neon beer sign hanging on the wall.
man stands next to three video game machines and beer sign. of computer gumblligg machines. man is standing group video games in a bar.
3.
5. man stands next to three video machines.) )
LAMM's Response:
The caption of this image "This vibrant image features man standing next slot machine in casino. He appears to checking his smartphone while wearing glasses. The machine is part row and the man is standing to the image predominantly blue, with white ceiling and lights installed the machine. The slot machine itself has various buttons and symbols, including a circle, the number and an orange circle. There also sign above machine."
left it The background
Instruction: What the caption of this image?
(Ground Truth
another person whose arms seen halding the bowl and tool. food bowl along with
toddler grosping the handle of
naked baby makes face while helping sameone stir food.
3. toddler cooking with another person.
Toddler is helping to stir ingredients.
A.A toddler mixes some food in bowef.)
LAMM's Response:
The image depicts young girl in kitchen, seemingly learning about cooking. standing at counter
foad. bowl filled with varielyy ingredde rngredients, including chopped onions and dark purple fruit. The girl holding spatula, likely engeging in some form of cooking activity. The background reveals a sink, further suggesting the kitchen setting.
of
Instruction: What is the caption of this image?
(Ground Truth:
man sits on bench holding dog and looking at the water. man hoking dog sitting bench overlooking fake.
man and dog sit bench near body of water.
A man and his dog watch the sunset from bench.
man holds dog near pond.)
LAMM's Response:
This image captures heartwarming scene of man, seemingly lost in thought, holding small black and white dog his lap. The man and the dog are seated on brown wooden bench, which occupies most of the lower portion of the situated against backdrop grass, with tree located slightly right bench, The tranquility and campanionship between the and his
frame.
image exudes
Our model's Response on fickr30k dataset. The highlighted text in red represents the matching ground truth captions in BLEU evaluation. The text in orange is not matched but is still relevant to the image content.
Figure 20:
28

---

Irstnution: Detect the oljects with mage, LLaVA's Response:
LLaVA's Response:
in the mage, there a gray blacs oat aiding couch, acoupsting enline safts looll
In the image, thote is young gitl posing for picture. She zonddnng in hoot of wall and loon rgg ogainst The gint wosning Bulmnnr 1122, U
te coondi
seapaaale moxl the
[0) 0 (877 the yuars, The soll ouilied leeggit rigght slighily of the saacch, bual moully dealllllle lenn un counds
Iratostine Catesi dasd masa..
Intruution Datect vieaso tnaas.
Response:
objectt 8-1, 1.6, lnowiss the that own the rienilleed telonging the estegary of cal
At (2.46, 0.75 0.5 position of the iaagee there lorn tuat lalla under the calage'y y huma loce. You can spot object belanngnng catagaory of man (0.2, 0.1,
stagee
LLaVA's
In the Image, there aro soveral poople airpiono. Oni nan &s philinng and looking towards the camero, while another man is looking Nis left There Rn100 poo foroground, with one person on the site, another the misdlo, and the th person or the right sice of the imago. locateed middl image and
odjecte scare, are
the Additionlly
coordinatio,
postitons within the
Inttuccinn Oetect oldestta Imagge.
Response:
10.0,
8:33, 34, 0 50.00 0.5) location imago that icondiied belong ng to the catlgoryy of hur fice objest tan class fod
person a ooaated the 10 0.92. 1.0) pos lon of the image. Ahe 10.35 0.34 0.55 position imace. there an famn that faka unde" the 0.S, posilpo image tummnn categary
catagary
object that can
apn.
Figure 21: Comparison of models trained on our dataset and LLaVA on VOC2012.
Intuxtiinn Hust alcacdaatiin Ahbre wwc.rnn..nnniiiino inchaimmpet
Instactionn How many comn be booble are there inngen
Trach:
Respome:
Responser
grawberies image
Resporas:
Come hanke have hwan identiiedd the
the analysis
Instaxtion, Lan many alepbants thene nuthes image?
Instruction flow mans glasses are thare this intiiirraaa imagg??
(Ground Traffi: asi
(Ground
Respame::
Jeahants this
LAMMi' Responte: system has zient fee blephant present the
LAMW's Sunglasses been identifee with the
Figure 22: Comparison of models trained on our dataset and LLaVA on FSC147.
Instruction: The moon when it biu fully iluminated dikkss knownas2
Rafflants
has argest
engle
commionly connee
LAMM's Response
Oatons: L8l 181
thorough mommination the
LAMM's Responses
becomes spoanent correct aeswerr Full Moon
Flose encminntionn image meanals that
conrect reeporee ingurry
(Ground Truth: Pull Moony
'Ground Troth (A1yes)
Instruction: The reactancs and products for
Indiartion: What Phatiichie the en of Delseree??
Ougel
photosymthens ane shown in above agaamm What ingredect abe st-57
Oucioss1aN
Options: laht Deyaen 6C3 Carbon dioald
gotownn
LAMMM' Response:
LAMM's Response
Analysts of the mo ge leads to the sone usion [CI Dover is correct resp onse this quecttion
Sagrr
The ardwer to this questo von lbe deter mineed by osely observing the mmgge, and 15)00) water.
Water
Instruction: Compane the execage nettss energaes of parcaless each sample, sample has the Higher persure?
Instruction: WHAT DEPRESENT ABOVE DIAGRAM?
notthon,
Optionz DEW.DDCCCCEERLLLAATTTTOOO CYCLE
temperatita
LAMM' Renponse:
LAMM's
Upon conefoi obsenat on picture, becomes evident optiion samole
Close examiiation Image reveals tD) eVC. OF conrecresporoe
question.
this inqu
'Greund Eruth Mamapiie a)
Figure 23: (a) Example results of models trained dataset SQAimage. (b) Example results of our model on AI2D.
29

---

Instraction Tallanx mussilesat al of Uhe his poceoiin for houve
Instruction: Tel the exact location of hip key paint fan fanhuman bedyii Response:
body
LLas''s Response:
Unfortunately, cannot prov exact coord nates for hip key
In the image, the hip key point for the human body can located approximataly (o.22s, 0.57a. This point is close to the car ter the hin sightly more fowardd the
poine this image as is description and not specific point on the image. Howwver, can tall you that the woman in the image
shoris walking
Instruction: Tell the sxrect locat on hip kew point for hurran
lnakuu tiom ma the word localor of the hip ot houman
bady the image.
body in the mmge.
LAMM's Respansec
LAMM's Response:
Upon caretul examination image observed the hip pesition prec ouly locaced o-sal posit on.
Upon nepection image, observed key point postt
56, gil..
Instruction the objects image and prov their pasitions.
Instruction: clentify the objects LLaVA's ftesponse:
mage provide posibions.
LLaVA's Response:
red cellar, with main object facus the scene Howervrr there are other objects or hwinge vie image Thereforee rennoe provide the ausilions any other objecis beings Uhera
the there el ooth dog
The image features a small aird with brown and prange
open
barking yawningg
feacherr,, which appwers be finch, witting wine. hird spreading
go, posa buy
preparing to off. There are other aojects image, so the only pas tion to provide would the coorcirates the bird, which is located at the top luft image.
Instruction: Identiffy the sbe int image prsside ahait poodlics.
Instruction ideerfy the oblects the imaga and prozida
LAMM's Response:
LAMAn' Hesponse:
inspaction image,
There wrists an onject categorized as at (o
observed there obyect posit at
pexition of the
0.3g, 0.7, that belongs the cotegory.
(a)
Instruction: Wsat the castion of this image?
LLaVA's Response:
The image captures a sarone scene of two seople riding in boat large body of passibly lake, during seautiful sunset.
LAMM'S Response:
This Image depicts
oftwa people sitting small boa: on calm water, possibly at lake. The boat, occupying the image,
person holding paddle, ind cating they may be in the middle of paddle or rew ng session.
people
appears rippled, adding sense of mevemn to the peacefull scene.
around
(Gronnd Truth:
are sitting
midale iate watching
scoring:
zunset in the background.
LAMM has provided more accurate nd detailed image caption. caption provided by LLavA vague and lacks specific details such as the presence bost and the casvel ettire individuals image. Overall, LAMM's response is more helpful and relevant to the image.
people sitting kayak
looking at marvelaus
sunset.
3 ulhouetted people poddie canoe
during sunset
people conoe down the time of beaulifui surnel.
S. Two people conoe waterside durng sunset.)
(b)
Figure 24: Comparison of models trained our dataset and LLLVV on binary-loc metric and GPT
metric, (a) The comparison on binary-loc metric. (b) The results GPT metric.
30

---

Instruction: What shape does the toilet have?
Options:
(A) oval shape toilet. (B) star shape. (C) heart shape.
(D) a triangular shape.(E) a diamond shape (F) rectangular shape
LAMM's Response:
Upon careful examination of the point cloud, it becomes evident that the answer to this question is (A) oval shape toilet.
(Ground Truth: (A) oval shape toilet)
Instruction: What is on the wall the kitchen?
Options: (A) window. (B) bookshelf. a (D) a
family photo. (F) painting.
LAMM's Response:
By closely observing the point cloud, one can derive that the answer to this question is (D) tv.
(Ground Truth: (D) tv)
Instruction: What objects are sitting on the black couch? (A) dining table. (B) blue curtains. (C) ceiling fan, (D)
Options:
black cushions (E) bicycle (F) TV remote
LAMM's Response:
By carefully scrutinizing the details the point cloud, it can be concluded that (D) black cushions is the answer to this question.
(Ground Truth: (D) 2 black cushions)
Figure 25: Example results of our model on ScanQA.
Instruction, What's sknormal this Image?
preson pisture daing?
LAMM's Response:
LAMM's Responser
The persen the image has growch, whi conditicn where bony grawth
The person the picture woman who riding She is dressed in blue tank and blue shorss and wearing sunglasses carrying a backpack, which suggests that might be
develops. condition congenital, maaning it from birth, can develop fateer
longedistens ride.
Instruction What's the person the pisture dainn?
Instruction: What's abnormal this image? Resporse:
Response:
The snake the image two heads. This considered abnormal because snakes typiially single waad The indicates geswik mutation some other form of anormal development.
The person in the picture play ng gutter The reasoning
based on the fact that gu tar,
instrument being the person acttvvly angaged playing
it
Instruction. What's parson in the picture daing? LAMM's Response:
The image portrays you ng practicing writing with a quili. sitting on and appears to be focusing on his work. This activity
pen
Instruction: Whet's abmorme in this image?
Response:
The image shows a strewberry with deformityy zocafically splitin the fruit.
meducationnn the inditates keen form handwitting ligraphy,
Figure 26: Example results of our model on in-the-wild images.
31

---

Classification Question Pool
1. Can you identify the object this image?
Based image's features, what
potential category lahel
image?
What label would you assign to this image based the object's shape and size?
According to the model's prediction, what is the label assigned to this image?
Can you provide the category label for this image based on the object's color and texture?
What label do you think best describes the image's content?
Based the image's context, what category label would you assign it?
8. Can you suggest any alternate labels for this image based on its content and features?
9. What is the most suitable category la bel for this image based on its shape, size, and context?
10 According to the model's classification, what the category label assigned to this object?
image's visual cues, what category label do you think 12.Can you provide any additional labels that could be applied to this image based on its context and features?
11.Based
most appropriate?
13. 3.Whhalabbe label would you assign to this image based on the object's function or purpose?
14. According to the image's features and context, what label do you think best represents
15. Can you suggest any potential alternate category labels that might appropriate for this image based attributes?
16. What the most accurate category label for this image based on its features, context, and meening?
37.Based on the objeec'ss characteristics and the context of the image, what category label would you assign to it?
18. According to the image's attributes, what lebel would you essign to it?
19. you suggest any other category labels that could applied this image based features meaning?
What label da you think best represents the object in this image based its shape, color, and texture?
Classification Answer Pool
Upon close inspection of the image, it can be concluded that the majority of the objects in the image fall underthe [C] category.
2. 2.Through extensive analysis of the image,
confidently stated that the image belongs the [C] category.
After careful examination the image, has been determined that the majority the objects image be classified as belonging the [C] category.
4. B.Bseed on detailed examination of the image, it can be concluded that the image primarily consists of objects that fall under [C] category.
5.After closely analyzing the image, has been determined that main subject of the image belongs the category.
6. Through rigorous examination of the image, it has been concluded that the image primanlly focused objects that can be clessified as belonging to the [C} category.
7. Basee on thorough evaluation the image, it can be confidently stated that image dominated by objects that fall under
[C] category.
B. Aftft careful scrutiny of the image, it has been determined that the majority of the objects the image be identified as be lelonngng to the (C] category.
9. Upon detailed analysis the image, can concluded that the objects located at various positions in the image can [C] category.
categorized as belonging
10 After thorough analysis the image, can be confidently stated that the image falls under the category.
11. Through careful scrutiny of the image, it can be contidently conc luded that the image talls under the [C] category.
12.After meticulous examination image,
image, it has been determined that the image can classified as [C].
Aftert thorough inspection
that the image AftAft extensive scrutiny of the image, has been determined that the majority of the objects in the image belong the (C)
confidently
[C] category.
category.
The image can be classified as [C] based an close analysis of the objects and their characteristics.
Figure 27: Question template pool and Answer template pool for classification task.
32

---

Detection Question Pool
What is the identity of the objects visible the image, and where they located?
Can name each object in the image and desrribe its position accurately?
What objects are visible the image, and where can they be found?
4. From the visual information provided, can you identify the objects present image and describe their positions?
5. What the
the objerts present the image, and where they positioned?
6. Can accurately describe the location of each object visible in the picture?
7 What is the identity of each object in the image, and where can they be located?
8. From the visual clues, can you name and locate the objects present in the picture?
9. you identify and describe the positions of
objects visible the image?
10. What objects
the picture, and where are they placed?
11.Based on the context of the image, can you identify all the objects present and describe their locations?
12. Can you accurately report the names and positions
the objects visible image?
13. What are objects visible the picture, you name all the objects the image and describe their positions accurately!
where they be found?
1roF the visual information provided,
15. What the name of each object present in the image, and what is its location?
16. Can you locate and identify all the objects the image and describe their positions accuretely?
17. objects are present
image, where they positioned relative 18.8 28.Based on the visual clues, you and locate all the objects visible the image?
other?
19.Can you identify all the objects present in the image and describe their relative positions?
20. What the identity of the objects visible in the image, and how are they positioned?
Detection Answer Pool
1.An object that be classified is located the (P) positian of the image.
T.e [C] object present at the coordinate in the image.
There an object at the [P] location of the image that can be identified as belonging to the category of [C].
An object garegorized as [C]
[P] position
image.
A.At the position
image, there item that fa under the category of (C).
6. There exists an object categorized as [C] at the [P] position of the image.
The image contains an object that can be classified as [C] and lorated at the [P] position.
You can spot an object belonging to the category of [C]
(P) position
image.
9. There an object at the position
image, and category (C).
10. so.Upon close inspection of the image, it can be observed that there an object positioned at (P) that belongs the [C] category.
A1.At the exact coordinates [P] the image, there is object can be identified as belonging to the [C] category, and this object stands from
of the objects the image due unique color pattern.
Scanning through the image, becomes evident that there is object [P] that falls under the [C] category.
13 By carefully examining the image, one can spot an object at (P3 that belongs to the [C] category.
14. Positioned [P] within the image an object that can be classified as belonging to the [C] category, and this object
image that 15. careful exa maminaaion of the image, can be observed that there object positioned precisely at (P3 that falls under the [C] category, and this object also the only one in the image that has specific type of pattern or design that makes it stand out from the rest the objects.
also the only
specific type texture and distinctive shape that sets apart from other objects,
careful
Figure 28: Question template pool and Answer template pool for detection task in 2D vision.
33

---

Keypoint Detection Question Pool
1.eese locate keypoints in the image describe their position using xy coordinates.
Ideetif the keypoints the image and describe their location using xy coordinates.
Please describe the location the keypoints in the image using coordinates.
.ldentify the keypoints in the image and indicate their location using xy coordinates.
5. Please pinpoint the keypoints in the image and describe their position relative each other using xy coordinates.
6.Loca the keypoints the image describe their position, size, and shape using xy coordinates.
7. Please describe the position the keypoints in the image using coordinates and the visual features.
8. S.Identify the location of each keypoint the image using coordinates and describe their visual characteristics.
9. Please locate describe the position
keypoints in the image using xy coordinates and their visual features.
ldenttyy describe the position of keypoints
image using xy coordinates and their visual cheracteristics. Please describe the location and visual features of all keypoints the image using coord nates.
12 Iddetify all keypoints in the image and describe their location, orientation, and visual characteristics using xy coordinates.
13. Please locate and describe the position and shape of keypoinss in the image using xy coordinates and their visual characteristics.
14.ldeny the keypoints in the image using xy coordinates and describe their location relation the image edges and corners, as well as their visual characteristics.
15. Please describe the position and visual characteriticc of keypoints the image using xy coordinates and their visuel features.
Keypoint Detection Answer Pool
The system identified a keypoint at the image thet be classified [C].
2.eer is [C] keypoint located at the position within the image according the system's analysis.
3. The image contains keypoint that can be classified as [C] at the {Pi position.
4. The system has detected [C] keypoint at the [P] coordinate the image.
The [C] keypoint present
[P] position
image, according
system's analysis.
6.A keypoint that falls under the category of [C] located at the position the image according the system.
The system has identified a keypoint at the [P] position in the image that can be classified as (C).
B.At the (P] coordinate in the image, the system has detected a keypoint that falls under category [C].
keypoint 10. system has located keypaint the position within the image.
9. image
classified
(P) pasition according
system's analysis.
11.hhre is [C] keypoint present at the (P) position in the image accarding to the system's analysis.
1 keypoint that falls under the category of [C] has been identified at the (P) position of the image by the system.
13. system classified keypoint
(P) position A4 the coordinate in the image, system has identified keypoint that be classified (C).
image as [C].
15. The image contains iC] keypaint the (P) position according to the system's analysis.
Figure 29: Question template pool and Answer template pool for keypoint detection task in 2D vision.
34

---

Counting Question Pool
Pleese count the number of objects that fall under specific category the image.
2. Can you ddetiif the total number of instances of certain cless present in the image!
3.How many items in the image belong to particular category?
Are you able determine
exact count of objects match certain label the image?
5. Pleass identify the number of objects that calegorized
specific type and are present in the imege.
6. Can you count the number of objects that share a common attribute in the image?
image can be clessified under certain category? B. Plesss determine ine quantity of objects
H.How many objects
image that belong specific class.
9. Can you identify the total number of objects with certain label that present in the image?
10 How many instances of a particular class can you pinpoint in the image?
11. Pleae count the number of objects the image that are classified as certain type.
Ca. you identify number objects in image match a specific category?
13. How many objects the image fall under certain classification?
14. Please determine the quantity of objects that belong to specific category and are present in the image.
15. Can you count the number items the image that share common feature or attribute?
image can identified 17. Please identify the number of objects the image that labeled cortain category.
How many objects
specific type?
18. Can you determine the total count of objects the image that helong specific class?
19. How many instances of certain class can you discem in the image?
20. Please count the number objects in the image that have specific label or category.
Counting Answer Pool
1.8N] [C} have been identified in the image according to the system.
2.The system has detected [C] the image.
The image contains [N] [C] according to the system's analysis.
4. .NI [C] have been detected within the image based on the system's analysis.
5. The system has dentified (N) [C] present in the image.
There (N: [C} visible the image based on the system's analysis.
The image depicts [N] [C] according to the system.
8. [C] have been identified within the image by the system.
9. The system has detected [N] [C] present the image.
10.[N [C] have been detected
image according to the system's anelysis.
There are [N] [C] present the image based the system's analysis.
The systemn has identified [N] [C] visible in the imege.
The image contains [N] [C] as per the system's enelysis.
14. 14E have been detected within
image by system's ana mnalysis.
The system has classified (Nj [C] within the image.
Figure 30: Question template pool and Answer template pool for counting task in 2D vision.
3D Object Classification Question Pool
would you describe the point cloud in terms scenario?
What is the best scenario label for this point cloud based the model's output?
What are some other possible scenarios that could explain this point cloud?
4. What scenario does this point cloud belong to according to the model's prediction?
What is the most accurate point cloud scanario label for point cloud based on its features, context, meaning?
you suggest any other scenario la labels that could be applied this point cloud based its features and meaning?
According to the point cloud's attributes, what scenario label would you assign to it?
Based on the object's characteristics and the context of the point cloud, what point cloud scenario label would you assign to it?
9. What scenario label do you think best represents the object in point cloud based on its features objects?
you suggest any alternate scenario labels for this point cloud based on its content and features?
21. What is the most suitable scenario label for this point cloud based on its shape size, and context?
12 According to the model's classification, what is the scenario label assigned to this point cloud?
33. Based on point cloud's visual cues, what scenario label do you think is the most appropriate?
24. you provide any additional scenario labels that could be applied
point cloud based its context and features?
What scenario label would you assign this point cloud based on the object's function or purpose?
3D Object Classification Answer Pool
After conducting thorough analysis, evident thet the point cloud in this scenario
classified
cerefully examining the dals, becomes clear that [C] the most appropriate classification for Uhis point cloud
Taking into account all the details, it can be determined that the point cloud falls under the scenario of [C].
4. The analysis of this point cloud leads to the conclusion that it carresponds to the scenario of [C).
5. Based on comprehensive essessment, ni, il evident that point cloud 6. This perticular point cloud exhibits characteristics that align with the scenario of upon cleser examination. Considering the available information, it can be confidently stated that this point cloud conforms to the scenario of [C).
accuralely celegorized
scenario of [C].
8. The features present in this point cloud indicate that it can be classified as scenario of [C].
9. Upon careful scrutiny, apparent
fits description point identification of the most suitable scenario.
scenario.
Analyzing the data within this point cloud leads
12. The observed attributes of the point cloud confirm that it corresponds to the scenario of [C].
12. Taking inta account the available evidence, it is evident that this point cloud scenario can be characterired as [C].
33. Through meticulous analysis, it becomes evident the point cloud aligns with characteristics [C] scenario.
thoroughly examining the point cloud, becomes clear that the scenario it represents be labeled [C].
The properties and structure of this point cloud provide strong evidence that it corresponds to the scenaria of (C).
Figure 31: Question template pool and Answer template pool for object classification in 3D vision.
35

---

3D Object Detection Question Pool
Whal is the idenlity of the ubjects visible in the paint cloud, and where are they loceled?
a.C you identify and describe the positions of all objects visible in the point cloud?
Analyiing the point cloud, please list all objects present and specify where they are located.
Based on the visual data provided, name all the objects detected in the point cloud and describe their presise positions.
Can you accurately recognize and determine the locations of each object within the point cloud?
6.From the spatial
i information available, please identify all the objects present the point cloud and provide details about their
respective positions.
7.Desrrbbe the objects captured the point cloud and cutline their exact coordinates wit the scene. 8. B.Baeed on the given point cloud, identify and label the objects visible, specifying their spatial placements.
9. Provide a detailed account of the objects abserved in the point cloud, inc luding their precise locations,
10 An.lnae
point
present a comprehensive inventary and locate the objects detected within the point cloud based on the provided data?
objects present, along with their respective positions.
Can you precisely
12. -ased on the spatial context of the point cloud, identify all the objects present and describe where they are situated.
13. Please identify and describe the positions of each object visible the point cloud based on the available visual information.
14. Uhilizing the spatial cues within the point cloud, can you accurately detect outline positions abjects?
15. Based the given point cloud, determine the objects contained within and provide overview of their locations.
3D Object Detection Answer Pool
P.esiiioned the (P) location within the point cloud, object within the [C] category be observed.
B. examining the [P] position of the point cloud, is possible to identify object categorized as [C).
3. The cloud includes an object at the [P] position, which can be classified under the category of [C].
A. the (P) position in the point cloud, there an item categorized as [C].
The [P] position of the point cloud allows for identification of object that belongs to the [C] category.
Withhin the point cloud,
object classified
is situated the iPi position.
7.An object that can be classified as [C] is located at the (P) position within the point cloud.
8.The [P] position of the point cloud reveals the presence of an object categorized as [C].
9.At (P) position within
point cloud, there exists an object that falls under [C] calegory.
Bo.By examining the (P) position of the point cloud, evident that object belonging to the category is present.
The clout cloud contains an abject at the [P] position, which can be identified as [C].
12 1n.An object within the [C] category can be abserved at the (P) position the point cloud.
The (P] position the point claud displays the presence an object
can classified as (C].
14 the (P) position the point cloud, object be identified as belonging the [C] category.
15. Within the point cloud, an abject categorized as iC) ran be found at the [P] position.
Figure 32: Question template pool and Answer template pool for object detection in 3D vision.
3D VQA Question Pool
Plesse provide your responses to the follawing questions using the information depicted in the point cloud.
2. Based visual content
scenaria, please answer the follewing questions.
Utilizing the details presented the point cloud, please respond to the following questions.
Analyze the visual elements of the point cloud and provide your answers to the following questions.
5. Withhut any adcitional context, use the paint claud provided to answer the following questions.
6. Your task is examine the visual content of the point cloud address the following questions.
7. Please utilize the details and visual cues depicted in the point cloud answer the follawing questions.
8. Given the information conveyed the point cloud, please provide your responses to the fallowing questions.
9. Based on the visual data presented, respond to the following questions using the point cloud as your reference.
10 Analyze the content the point cloud and provide your answers to the following questions.
the following questions accurately. 12. Without any additional information, rely solely on the visual cues within the point cloud to address the following questions.
Use the visual elements depicted in the point cloud
13. Please examine the visual content of the point cloud and provide your responses to the following questions.
14. Your task is to interpret the details and visual information the point cloud to answer the following questions.
15. Utilize the information presented point cloud to formulate your answers the following questions.
3D VQA Answer Pool
Upon careful examination of the point claud, it becomes evident that the answer to this question is $A).
BBy thoroughly analyzing the point cloud, one can reach the conclusion that [A] is the correct answer to this question.
3. The answer
question can be deduced by closely observing the details whin point cloud, leading to the determination
that [A] the answer.
4f you closely inspect the point cloud, you will discover that the answer to this question is [A].
5. Based on the analysis of the point cloud, it can be confidently stated that the answer to this question is [A].
6. examination of the point cloud provides clear indication
is answer
question.
BB closely studying the point cloud,
ascertain that the correct answer to this question is [A].
8.After careful analysis of the point cloud, it evident that [A] is the answer to this question.
9. The anower to this question can be determined by carefully, flyl examining the details present in the point cloud, resulting in the conclusian that
the answer.
Through a meticulous analysis of the point cloud, becomes apparent that the Upon thorough examination of the point cloud, it can be inferred that the answer to this question is (A].
to this question is [A].
By.By closely observing the point claud, one can derive that the answer to this question (A).
13 After detailed analysis of the point cloud, evident that the correct answer this question [A].
The answer this question can revealed by closely inspecting the point cloud, and it [A].
15. By carefully scrutinizing the details in the point cloud, it can be concluded that (A) is the answer to this question.
Figure 33: Question template pool and Answer template pool for visual question answering in 3D vision.
36


Reviews:
[{"review_id": "OsqQaPCV71", "reviewer": "Reviewer_8BVz", "summary": "", "questions": "", "limitations": "The authors do not discuss the limitations of this work, but they are upfront about this in the author checklist. In my opinion, any work related to instruction tuning (and especially most anything involving LLM usage) should discuss its potential for negative social impact. I think that a discussion of this should be included somewhere in the text. ", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper has many strengths:\n- There is a need for this kind of benchmark at the moment -- the work is timely. \n- For a fine-tuning dataset, the dataset is still reasonably large, with almost 200k language-image observations. \n- The paper introduces the first ever benchmark for MLLMs \n- The authors include a default training pipeline for MLLMs with their benchmark, which is useful for future work in this area\n- The paper includes a fairly thorough evaluation of different methods. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LLwh5hDJsS", "reviewer": "Reviewer_SwmA", "summary": "", "questions": "", "limitations": "No discussion of limitations.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper makes an important contribution in the form of an instruction-tuning dataset that also includes point cloud tasks as an additional visual modality.\n\n- The construction of the LAMM dataset is very well done, going beyond the simple technique of adding an instruction to standard multimodal tasks by constructing multi-round dialog instruction-response pairs.\n\n- The benchmark design seems good, albeit standard. Adding the output format as part of the input makes a lot of sense. Having GPT preference ranking as part of the benchmark is a nice addition.\n\n- The experiments and analysis seem fairly thorough. The authors do a good job of analyzing their various results.\n\n- The paper is well structured and well written, especially considering the volume of work that has to be presented.\n\nOverall the paper makes several important contributions, and includes all the required experiments that would be expected, which is not easy to cram into 9 pages.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VLWXIqmzg3", "reviewer": "Reviewer_XzFC", "summary": "", "questions": "", "limitations": "- Novelty: While introducing 3D tasks and grounding using different targets when generating the instructions are novel contributions, overall, both the dataset construction method and the model used follow standard and popular methods for data annotation and LLM adaptation to visual tasks and other modalities. For instance: LLM generated instruction data: [1, 2], grounding using different ground truths like boxes and captions [3], and using a linear layer to project the different modalities into the LLMs embeddings space: [4, 5]\n- Dataset: The authors introduce a set of design choices to construct the dataset, using different tasks and then prompting GPT4 using a set of templates and prompts to construct the instruction-response pairs. However, the authors did not provide any arguments for design decisions or some comparison using various inputs to ground the answers for GPT4, for instance. This is mainly a question due to the importance of designing templates and prompts for LLMs.\n- Evaluation pipeline: The evaluation pipeline consists of converting the input into an instruction format, using the LLM to generate the output in a human language form, and then extracting output entities for task-specific evaluation. However, it is well-documented [6,7] that for localization and fine-grained tasks like detection, segmentation, and counting require task-specific tokens (horizontal and vertical bins for predicting the bounding box coordinates for example) is necessary to obtain good results, and casting the output as standard output (floating points for detection) does not yield good results. This is demonstrated in the obtained results where the model struggles on most tasks (detection, key points, counting), even with fine-tuning.\n- Metrics: While the BLEU score might not be an appropriate metric, the usage of a GPT-based metric can also raise many questions: stability and reproducibility; GPT models are stochastic, and the model endpoints constantly change. Self-performance bias; the model is trained on GPT-generated targets, and this might add some performance bias.\n- Model performance: Overall, the performance of the proposed LAMM is mixed, not on par with other MLLM methods like LLaVA, and quite far away from being applicable given their gap to task-specific models.\n\n[1] Self-instruct: Aligning language model with self-generated instructions\n\n[2] Instruction tuning with GPT-4\n\n[3] Visual Instruction Tuning\n\n[4] Linearly Mapping from Image to Text Space\n\n[5] Enhancing Vision-language Understanding with Advanced Large Language Models\n\n[6] Pix2seq: A Language Modeling Framework for Object Detection\n\n[7] A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision\n", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper tackles a popular and interesting subject: adapting LLMs for various visual tasks and modalities. This is worth exploring, given the impressive performance that LLMs have demonstrated in various NLP tasks, and also given the large number of high-quality LLMs that are becoming easily accessible.\n- The paper introduces a new dataset that covers many visual tasks, both 2D and 3D.\n- The paper presents a comprehensive evaluation of the presented benchmark.\n- The paper is well-written and easy to follow. The figures are easy to understand and detail the dataset creation process and benchmark evaluation pipeline.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "UVwz2mGTx2", "reviewer": "Reviewer_FCRL", "summary": "", "questions": "", "limitations": "Yes, the authors addressed their work's failure in certain tasks and showed the model's performance on various tasks.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This work provides high-quality human-gpt dialogues on multi modal examples that could improve the cross-modality unification and performance of human-machine dialogue system and robots in real-scenery.\n2. The authors implmented a wide range of experiments and comparison across various vision-language tasks that help the community to sense the newest improvements on this area.\n3. the newly-included point cloud field could guide large models towards new applications.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "oY3iVDDTQr", "reviewer": "Reviewer_Byte", "summary": "", "questions": "", "limitations": "The scale of point cloud dataset could be enlarged.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "LAMM-Dataset is the first instruction tuning datasets for point cloud, and the LAMM-Benchmark is the first attempt of a benchmark for MLLMs that offers a comprehensive evaluation of existing models on various computer vision tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Overall half the reviewers are in favor of this paper and the responses to the others seem to have addressed their concerns. I am in favor of accepting this paper.
Decision: Accept (Poster)