SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: CV tasks in Our Benchmark
Task	Output	Metrics
Classification	label name	Acc
Detection	list of object label andbbox	mAP50
VQA	option and answer	Acc
Image Caption	captions	BLEU4
Fine-grained classification	fine-grained label name	Acc
Object counting	number	MAE
OCR	list of words	word Acc
Facial classification	answer	Acc
Keypoints detection	keypoints	PCK
3D Detection	list of object label andbbox	mAP50
3D VQA	option and answer	Acc
3D Visual Grounding	bbox	mAP50
Classification This task involves predicting the most likely category label for an image. For MLLM models, the task involves performing open-vocabulary classification. We selected CIFAR-10 [41] as the test dataset for the evaluation of classification. CIFAR10 contains 10000 test images across 10 common categories. We utilize NLTK to extract noun entities from the models' output text, and expand them to a synonym set for accuracy evaluation calculation.
Object Detection We selected the VOC 2012[42] datasets to evaluate the model's ability to detect objects in images while considering both its visual interpretation and localization capabilities. To evaluate the accuracy of object category predictions, we employ a similar approach to classification tasks. We also use regular expression matching to extract the models' output bounding boxes for mAP50 calculation.
Visual Question Answering We selected the ScienceQA[43] and AI2D[44] datasets to evaluate the MLLM model's ability to answer questions about images. The ScienceQA and AI2D datasets include over 2017 and 5793 multiple-choice questions with images, respectively. We extract the image-containing data from the ScienceQA dataset to create the SQAimage dataset. We then tested MLLM models on the SQAimage dataset to evaluate their multimodal understanding skills. As both ScienceQA and AI2D datasets are presented in a multiple-choice format, we evaluated the model's performance using the accuracy metric. Following LLaVA [15], we prompt the MLLM to output the complex reasoning procession, followed by the final option answer.
Image Caption The image caption task involves generating a textual description of an image. We selected the Flickr30k[45] dataset to evaluate the MLLM model's ability to understand images and generate descriptive captions. Flickr30k contains a variety of objects and scenes with diverse captions, providing a challenging task for the MLLM model. To evaluate the quality of the models' text outputs, we split the generated text into sentences and calculate the BLEU-4 score for each. The highest score is selected as the final result.
Fine-grained classification Similar to the classification task, the fine-grained classification task requires the model to make predictions across a large number of fine-grained categories. We selected UCMered Land Use dataset [46] as the test set. UCMered Land Use contains 21 classes of land-use categories, including airports, forests, and residential areas. Similar to classification, we report Accuracy.
Object counting We selected the FSC147 dataset for object counting evaluation. FSC147[47] is a dataset of 1190 images containing various objects, including animals, vehicles, and household items. The images in this dataset are challenging and contain occlusions and overlapping objects, making it a suitable choice to test the model's object recognition and localization capabilities. We utilize regular expression matching to extract the numeric entity and evaluate the model's performance using the mean absolute error (MAE) metric.
Optical Character Recognition The OCR (Optical Character Recognition) task involves recognizing and transcribing text from images. To evaluate the MLLM model's ability to recognize text from images, we selected SVT dataset [48]. We extract the entities enclosed in quotation marks from the generated text as the predicted word list. Word Accuracy is adopted as the evaluation metric.
Figure 15: Template instructions for VQA inference. "Response a" is the generated reasoning process, which is the output of the first inference. "Response b" is the output answer, which is the output following the prompt "# ANSWER".
Facial Classification Due to the difficulty of performing face recognition tasks using MLLM, we evaluated the model's performance on facial attribute classification tasks. We selected the CelebA[49] dataset, which contains 19962 images for testing with annotations for 40 facial attributes, including hair color and facial expression. Specifically, we evaluated the model's ability to predict whether a person in an image is smiling, named CelebA(Smile) dataset, and the color of their hair, named CelebA(Hair) dataset. We aimed to evaluate the MLLM model's ability to understand facial images. Classification accuracy is used as the evaluation metric.
Keypoints Detection To evaluate the models' ability to perform fine-grained point localization, we utilized the LSP[50] dataset for keypoint detection. To simplify the task difficulty for MLLM models, we employed a grounding approach, where we sequentially asked the model to predict the position of each human body keypoints in the image. The evaluation metric used for this task was PCK (Percentage of Correct Keypoints).
C.2 Inference Details
C.2.1 System messages for image tasks
Figure 17 shows the system messages defined for each image task. The system messages, which include the task definition and the output structure, is a part of the instruction that prompt the MLLM models to generated responses. This is designed to enable the model to better understand the task it is performing, focus on the critical aspects, and output the appropriate structure. Note that some tasks do not require a defined output structure. In such cases, the model can output any text as a response.
C.2.2 Instructions for VQA
Different from other common image tasks, besides the system messages designed in C.2.1, we prompt MLLM to generate the reasoning process additionally, as figure 15 shows. To prompt the model to output its reasoning process, we first use conventional instruction texts to generate "Response a". We then combine the first instructions, the "Response a", and the prompt "# ANSWER" to make the model generate the option as the final answer.
C.2.3 Metrics
Our benchmark includes two evaluation settings. The first is a zero-shot setting, where we selected downstream tasks that have no intersection with the MLLM's training data. We provide the zero-shot results of the current MLLM models on these datasets. The second setting involves fine-tuning on mainstream task datasets, covering tasks such as detection, classification, and VQA.
C.2.4 Binary Locating Metric
The ability to accurately localize objects in an image is a crucial component of MLLM models' visual understanding skills. In addition to using conventional detection tasks to calculate mAP, we attempted a more direct method for evaluating the models' localization ability, namely Binary Locating Metric. Distinct from object detection, which requires the model to output a bounding box, we instructed the model with "output the position of the object" instead of "output the bounding box of the object" to output the approximate position. During the evaluation phase, the model's predicted keypoint was considered correct as long as it was within the object's bounding box. Object locating is evaluated on all datasets involving object localization, including object detection, object counting, and keypoints detection. Compared to the traditional detection evaluation methods, the object locating evaluation
[System Message]  
```java
////Human: [Vision Tokens] [/vision] [Instructions]  
////Assistant: [Response a]  
////ANSWER: [Response b]
method provides a more reasonable and direct approach for evaluating the localization ability of MLLM.
C.2.5 GPT Metric
To evaluate the overall understanding and question-answering abilities of MLLM models, we utilized the GPT Metric. Unlike LLaVA[15] and Vicuna [9], we ranked the answers of multiple models using GPT. Similar to the pipeline approach, we give GPT an instruction, informing it of the task definition, the question, and the answer provided by each model. We then ranked each model's response based on its relevance and accuracy with the answer. Each model received a score based on its ranking, and the average score obtained on all test data served as a metric for measuring the model's overall ability. Our GPT evaluation datasets cover various visual tasks, including captioning and VQA tasks involving image description and answering, as well as a small number of detection and counting tasks related to object localization.
C.3 Benchmark on point cloud tasks
For benchmark on point cloud tasks, we focus on three tasks of scene perception, including 3D object detection, visual grounding, and 3D visual question answering. Figure 18 presents system messages for point cloud tasks.
3D Object Detection. As it's widely used in 3D object detection, we select ScanNetv2 [51] as the dataset to evaluate MLLM's ability to locate objects in a point cloud and identify semantics, whose validation set contains 312 scenes. In this task, MLLM is expected to list all objects along with bounding boxes, and we extract bounding boxes from the response text by entity extraction. Boxes whose IoU with ground truth is larger than  $50\%$  count for positive predictions and we use mean Average Precision (mAP) to evaluate performance.
Visual Grounding. This task aims to locate the object described by a given caption and output the corresponding bounding box. We test on ScanRefer [52] in this task, which provides human-labeled captions towards each object in ScanNet and its test set contains 9508 samples. Similar with object detection, mean average precision (mAP) is reported to evaluate MLLM's capacity.
3D Visual Question Answering. ScanQA [53] is proposed for 3D visual question answering before, and models are required to answer the given questions based on the point cloud. It has been formatted as an attribute classification task in previous work [53]. However, MLLM's output cannot be constrained with several classes consistently and is usually long text to explain details, so the original metrics in ScanQA, Exact Matching & BLEU, cannot be used for test, as long text is different from the style of given ground truth and the BLEU score inevitably decreases for long-text results. Following ScienceQA in 2D VQA task, we transfer this task to be a multiple-choice problem. First, we feed the original question-answer pairs to GPT-API and ask for 5 confusing options. Then MLLM is expected to choose the correct option or output the correct content. Thus, a metric of accuracy is used to evaluate model performance.
Evaluation Settings Similar to evaluation for 2D tasks, our 3D benchmark includes two settings for evaluation. The first one is a zero-shot setting. MLLM is trained on instruction data from 3D portion of our dataset, whose point clouds come from 3RScan or ShapeNet and has no overlap with ones in downstream tasks. Furthermore, we finetune the models trained on our 3D datasets by training a set of downstream tasks and reporting metrics on the corresponding test set.
In our experiments, 2D and 3D models are trained independently, and only the feature projection layer and LoRA parameters are optimized during training while LLM can be shared among tasks.
For all experiments, trainable parameters are optimized by Adam optimizer with a learning rate initialized to be 5e-4, and scheduled using a linear decay scheduler. We For 2D experiments, models are trained for 2 epochs. For 3D experiments, we increase the number of iterations to 10,000 in case of too few samples. We use 4 A100-80GB to conduct experiments. Each GPU process 2 samples every iteration and the effective batch size are set to 64 by gradient accumulation. For reference, 2D
[System Message]
```c
## # Human: <vision> [Vision Tokens] </vision> [Instructions]
Assistant: [Response]
Figure 16: Template for multi-modal data pairs. Bold words stand for corresponding text data and italic words indicate fixed templates.  $<\text{vision}> \& </\text{vision}>$  stand for start & end token for vision contents.
experiments at most last for about 8 hours for 186K samples, while 3D experiments require about 3 hours.
Following Vicuna [9], we format multi-modal training data as Figure 16. [SystemMessage] specifies the corresponding task of sample, [Query] refers to position of texts from human and [Response] refers to contents expected for LLM. The special tokens  $<$  vision  $\rangle \& <$  /vision  $\rangle$  represents start and end positions for vision content. We use  $<$  Img  $\gg <$  /Img  $\gimel$  and  $<$  Pcl  $\gg <$  /Pcl  $\gimel$  in 2D and 3D datasets, respectively. The training objective used is next token prediction loss, and only text tokens of [Response] count for loss computation. As we use CLIP [38] pre-trained ViT-Large-14 as visual encoder, the number of vision tokens are 256 and length of text tokens after vision tokens are limited to 400 in training.
E Demonstrations
E.1 Results on CIFAR10
Figure 19 presents some examples responses from model trained by our dataset on CIFAR10, where the model's answers were judged as incorrect in the evaluation, but in fact, our model provided a more granular classification result. The left column shows the test images from CIFAR10, and the right column displays the images of the objects that the model classified, including toad [61], Land Rover Series II [62], Mirage 2000D fighter aircraft [63] and police car [64]. It is evident that the fine-grained objects classified by our model have very similar features to the input images, demonstrating its ability to perform fine-grained classification.
E.2 More detailed information on image caption
Our model performed poorly on the Flickr30k dataset in terms of BLEU scores. This is because model's responses include additional details that are not captured by the ground truth captions. Figure 20 illustrates this phenomenon, where the highlighted text in red represents the matching ground truth captions, while the text in orange is not matched but is still relevant to the image content. It is evident that our model is capable of providing more detailed descriptions of the image, which is not captured by the traditional BLEU metric.
E.3 Comparison with LLaVA on detection and counting tasks
We compared the performance of model trained by our dataset with that by LLaVA on both object detection and counting tasks. Figure 21 illustrates the comparison results on detection, where the leftmost images represent the ground truth bounding box, and the rightmost images show the visualizations of the responses after entity extraction.
Although LLaVA was able to identify the approximate location of the object, it was unable to provide precise bounding box coordinates. On the other hand, our model demonstrated superior detection capabilities after fine-tuning on detection-related data and was able to provide more accurate bounding box coordinates. Additionally, our model also exhibited better counting performance, as shown in Figure 22. It is worth noting that counting is essentially a task that tests the model's localization ability.
E.4 Results of binary-loc metric and GPT metric
We present the results of our model and LLaVA on the binary locating metric in Figure 24 (a), where our model demonstrates more precise localization abilities. The green points in the image are
the visualization of the predicted key points. In the second row of the figure, our model outputs a bounding box, which we break down into two position coordinates (top-left and bottom-right) during entity extraction.
In Figure 24 (b), we show the evaluation results of the two models' image captioning responses using the GPT metric. The GPT metric considers our model's responses to be more specific and accurate compared to LLaVA, resulting in a higher ranking. These results further demonstrate the effectiveness of the model trained on our dataset in accurately detecting, locating, and describing objects in images.
E.5 More demonstration examples
Figure 23 shows the results of our model on VQA task and Figure 25 shows its example results on 3DVQA task. Figure 26 shows the results on in-the-wild images.
Figure 17: System messages for benchmark on image tasks
3D object Detection
You are now performing an object detection task, and your goal is to locate all instances of objects in a point cloud, such as furniture, transportation, or other objects, and give the corresponding coordinates.
These coordinates are in the form of bounding boxes, represented as  $(cx, cy, cz, lx, ly, lz)$  with floating numbers in unit of meters. These values correspond to the  $x$ ,  $y$ ,  $z$  coordinates of bounding box center and length of bounding box along  $x$ ,  $y$ ,  $z$  axis.
3D Visual Grounding
You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language. You are now performing an visual grounding task, and your goal is to locate the instances of objects I want with the given description in a point cloud, such as furniture, or other objects, and give the corresponding coordinates.
These coordinates are in the form of bounding boxes, represented as  $(\mathbf{cx}, \mathbf{cy}, \mathbf{cz}, \mathbf{lx}, \mathbf{ly}, \mathbf{lz})$  with floating numbers in unit of meters. These values correspond to  $x, y, z$  coordinates of center of bounding box and length of bounding box along  $x, y, z$  axis
3D VQA
You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language. You are now performing an visual question and answering task, and your goal is to generate natural language answers that accurately solve the question. In order to generate accurate answers to questions about visual content, you must be able to understand the content of point cloud, understand the meaning of questions, and perform complex reasoning processes.
Figure 18: System messages for benchmark on point cloud tasks
Upon close inspection of the image, it can be concluded that the object of interest is a toad.
(Ground Truth: "frog")
Upon close inspection of the image, it can be concluded that the object of the image is a Land Rover Series II.
(Ground Truth: "automobile")
Upon close inspection of the image, it can be concluded that the object of the image is a Mirage 2000D fighter aircraft. (Ground Truth: "airplane")
Figure 19: Results of model trained by our dataset on CIFAR10. (a) The images from CIFAR10 test set. (b) The instruction, response from our model and the ground truth. (c) The reference images.
The image contains an object that can be classified as a police car based on the descriptions provided. (Ground Truth: "automobile")
(Ground Truth:
1. A man standing in a bar with a neon beer sign hanging on the wall.
2. A man stands next to three video game machines and a beer sign.
3. A man stands next to a bank of computer gambling machines.
4. A man is standing by a group of video games in a bar.
5. A man stands next to three video machines.
The caption of this image is "This vibrant image features a man standing next to a slot machine in a casino. He appears to be checking his smartphone while wearing glasses. The slot machine is part of a row and the man is standing to the left of it. The background of the image is predominantly blue, with a white ceiling and lights installed on the machine. The slot machine itself has various buttons and symbols, including a circle, the number 7, and an orange circle. There is also a sign above the machine."
1. A toddler grasping the handle of a tool to stir food in a bowl along with another person whose arms are seen holding the bowl and tool.
2. A naked baby makes a face while helping someone stir food.
3. A toddler is cooking with another person.
4. Toddler is helping to stir ingredients.
5. A toddler mixes some food in a bowel.)
The image depicts a young girl in a kitchen, seemingly learning about cooking. She is standing at a counter in front of a bowl of food. The bowl is filled with a variety of ingredients, including chopped onions and a dark purple fruit. The girl is holding a spatula, likely engaging in some form of cooking activity. The background reveals a sink, further suggesting the kitchen setting.
(Ground Truth)
1. A man sits on a bench holding his dog and looking at the water.
2. A man holding a dog sitting on a bench overlooking a lake.
3. A man and a dog sit on a bench near a body of water.
4. A man and his dog watch the sunset from a bench.
5. A man holds his dog near a pond.
This image captures a heartwarming scene of a man, seemingly lost in thought, holding a small black and white dog on his lap. The man and the dog are seated on a brown wooden bench, which occupies most of the lower portion of the frame. The bench is situated against a backdrop of grass, with a tree located slightly to the right of the bench. The image exudes a sense of tranquility and companionship between the man and his dog.
Figure 20: Our model's Response on flickr30k dataset. The highlighted text in red represents the matching ground truth captions in BLEU evaluation. The text in orange is not matched but is still relevant to the image content.
Figure 21: Comparison of models trained on our dataset and LLaVA on VOC2012.
Figure 22: Comparison of models trained on our dataset and LLaVA on FSC147.
(a)
(b)
Figure 23: (a) Example results of models trained on our dataset on SQA image. (b) Example results of our model on AI2D.
Instruction: Tell me the exact location of the hip key point for human body in the image.
Unfortunately, I cannot provide exact coordinates for the hip key point in this image as it is a description and not a specific point on the image. However, I can tell you that the woman in the image is wearing shorts and is walking on a road.
Upon careful examination of the image, it can be observed that the hip position is precisely located at  $[0.42, 0.39]$  position.
Instruction: Identify all the objects in the image and provide their positions.
In the image, there is a small white dog wearing a red collar, with its mouth open, possibly barking or yawning. The dog is the main object of focus in the scene. However, there are no other objects or beings visible in the image. Therefore, I cannot provide the positions of any other objects or beings as there are none in the scene.
There exists an object categorized as dog at the  $[0.58, 0.44]$ ,  $[1.0, 0.75]$  position of the image.
LLaVA's Response: In the image, the hip key point for the human body can be located at approximately [0.224, 0.571]. This point is close to the center of the image, but slightly more towards the top.
Upon close inspection of the image, it can be observed that the hip key point is positioned at  $[0.56, 0.38]$ .
The image features a small bird with brown and orange feathers, which appears to be a finch, sitting on a wire. The bird is spreading its wings, possibly in flight or preparing to take off. There are no other objects in the image, so the only position to provide would be the coordinates of the bird, which is located at the top-left part of the image.
Upon close inspection of the image, it can be
observed that there is an object positioned at [0.26,
0.29, 0.77, 0.96] that belongs to the bird category
The image captures a serene scene of two people riding in a boat on a large body of water, possibly a lake, during a beautiful sunset
This image depicts a serene scene of two people sitting in a small boat on calm water, possibly at a lake. The boat, occupying the center of the image, has two people in it. One person is holding a paddle, indicating they may be in the middle of a paddle or rowing session. The water around the boat appears rippled, adding a sense of movement to the peaceful scene.
1. Two men are sitting in a canoe in the middle of a lake, watching the sunset in the background.
2. Two people sitting on a kayak in calm water looking at a marvelous sunset.
3. Two silhouetted people paddle a canoe on the ocean during sunset.  
4. Two people canoe down a river at the time of a beautiful sunset.
5. Two people in a canoe on the waterside during sunset.)
GPT scoring:
LAMM:8
LLaVA:4