SlowGuess/NeurIPS_2023_OCR huggingface dataset:
You need to generate a n-round Daily Dialogue data between two people. The answers should be in a tone that a visual AI assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers. Include at least one plausible question about the image, and provide the answer in detail. To answer such questions, one should require first understanding the visual content, then based on the background knowledge or reasoning, either explain why the things are happening that way, or provide guides and help to user's request. Make the question challenging by not including the visual content details in the question so that the user needs to reason about that first.
Instead of directly mentioning the bounding box coordinates, the captions and relations, utilize those data to explain the scene using natural language. Include questions asking about the visual content of the image, including the object types, counting the objects, object actions, object locations, relative positions between objects, etc. Always answer as if you are directly looking at the image. }}]
for sample in fewshot_samples: messages.append(['role":"user","content":sample['context']) messages.append(['role":"assistant","content":sample['response']) messages.append(['role":"user","content":"n'.join(query)])
Figure 6: Messages used to construct  $n$ -round Daily Dialogue data for image instruction tuning.
messages = [(You are an AI visual assistant that can analyze a single image. You receive five sentences, each describing the same image you are observing. In addition, specific object locations within the image are given, along with detailed coordinates. These coordinates are in the form of bounding boxes, represented as (x1, y1, x2, y2) with floating numbers ranging from 0 to 1. These values correspond to the top left x, top left y, bottom right x, and bottom right y. It is worth noting that Attributes and Relationships of different objects are also given.
You need to generate a 1-round Detail Description data. You can use the provided caption, attributes, relationships and bounding box information to describe the scene in a detailed manner.
Instead of directly mentioning the bounding box coordinates, utilize this data to explain the scene using natural language. Include details like object counts, position of the objects, relative position between the objects. When using the information from the caption and coordinates, directly explain the scene, and do not mention that the information source is the caption, relationships or the bounding box. Always answer as if you are directly looking at the image. You can include multiple paragraphs if necessary."]}
for sample in fewshot_samples: messages.append(['role":"user","content":sample['context']) messages.append(['role":"assistant", "content":sample['response']) messages.append(['role":"user","content":"\n'.join(query)])
Figure 7: Messages used to construct 1-round Detailed Description data for image instruction tuning.
results are not suitable for large language model dialogues, as they do not allow for natural language interactions. To address this issue, our proposed approach involves converting computer vision tasks, such as image classification, into natural language dialogues to enable large language models to perform these tasks through dialogue interactions. In detail, we first use GPT-API to generate a template pool of questions and answers for each task. Then, we randomly select a pair from the question template pool and answer template pool to combine with a piece of data from the computer vision dataset, creating the C4: 1-round Visual Task Dialogue data. Figure 27-30 provide some examples of the dialogues generated using our proposed approach for converting computer vision tasks into natural language dialogues. This approach allows us to leverage the rich visual information in traditional computer vision datasets and incorporate it into the instruction tuning process, thereby enhancing the ability of MLLMs to understand and respond to natural language instructions related to these tasks.
In summary, the construction of 2D part in our dataset provides a comprehensive and diverse samples of real-world scenarios, incorporating fine-grained and dense information from object relationships and factual knowledge sources. The dataset contains 186K unique language-image instruction-following samples, including 49K in daily dialogues, 49K in detailed descriptions, 42K in factual Knowledge dialogues, and 46K in visual task dialogues. Our experiments showed that the use of GPT-API consistently provides higher-quality instruction-following data, such as spatial reasoning.
Captions: [A clock on a tower with a church steeple and bell. A clock tower has a fenced in bell on top. A very tall clock tower with a large bell sitting above it. The cross, bell and clock on the tower over a church. There is a tower clock on the wall.] Objects: bell: rusty, [0.4, 0.23, 0.57, 0.33]. clock: white, black, [0.18, 0.65, 0.77, 0.89]. building: white, tall, rusty, [0.14, 0.09, 0.9, 0.99]. sky: blue, [0.01, 0.0, 1.0, 0.7].
C1: n-round Daily Dialogue
Instruction: <image> What kind of building has a clock tower in the image?
Response: In the image, there is a church with a clock tower.
Instruction : Is there anything special about the clock tower?
Response: Yes, there is a large, fenced-in bell on top of the clock tower, which makes it unique.
Instruction : What color is the clock on the tower?
Response: The clock on the tower is white with black numerals and hands.
Instruction : Can you describe the appearance of the sky in the image?
Response: The sky in the image is blue.
Instruction : What role might this clock tower and bell play in the local community?
Response : The clock tower and bell might play a significant role in the local community by serving as a landmark and a focal point of the town or neighborhood. The clock tower, with its large, visible clock face, helps the residents keep track of time and maintain their daily schedules. In addition, the bell might be used to signal important events, such as religious services, weddings, or funerals. It could also serve as a reminder of community values and traditions, as well as a symbol of unity and shared history.
C3: 1-round Detailed Description
Instruction: <image> What can you observe in this image?
Response : The image presents a tall, white clock tower that stands out against a clear blue sky. The clock face is prominently visible, featuring white background and black hands. Above the clock, a fenced-in, rusty bell can be seen, adding an air of history to the tower. Furthermore, the tower also displays a cross at the top, signifying its connection to a church. The combination of the cross, bell, and clock creates a striking visual that showcases the tower's architectural and historical significance.
Figure 8: Example of generated  $n$ -round daily dialogue and 1-round detailed description data.
messages = ["You are an AI visual assistant that can analyze a single image. You receive a category label for an image and its associated English description describing the same image you are looking at. The category information is the category label of the image you are seeing. In addition, The English description is a detailed semantic information for the given category.
You need to generate a  $n$ -round Factual Knowledge Dialogue data based on the received category table and its English description. Please ask questions around the given category information and its corresponding description information, and don't ask questions that cannot be reasoned based on the given information. Questions and answers should be concise and effective, with reasoning logic. Answer all questions as you are seeing the image.
The answers should be in a tone that a visual AI assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers. Include questions asking about the visual content of the image. Only include questions that have definite answers: (1) one can see the content in the image that the question asks about and can answer confidently; (2) one can determine confidently from the image that it is not in the image. Do not ask any question that cannot be answered confidently. Also include complex questions that are relevant to the content in the image, for example, asking about background knowledge of the objects in the image, asking to discuss about events happening in the image, etc. Again, do not ask about uncertain details. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. You can include multiple paragraphs if necessary }}]
Figure 9: Messages used to construct  $n$ -round Factual Knowledge Dialogue data for image instruction tuning.
These features make our dataset a valuable resource for researchers and practitioners working in the computer vision and natural language processing fields.
B.2 Point Cloud Instruction Tuning Dataset
The construction pipeline of point cloud instruction tuning data is similar to that of image instruction tuning data. However, due to the limited availability of 3D data, point cloud instruction tuning dataset only consists of three major components: n-round plain conversation and 1-round detailed description data from GPT-API and 1-round visual dialogue data converted from datasets for existing 3D vision tasks.
C1: n-round Daily Dialogue & C3: 1-round Detailed Description. To construct the C1: n-round Daily Dialogue and C3: 1-round Detailed Description data, we choose point clouds from 3RScan [59]
Category: Women in Belarus.
Description: The modern-day characteristics of women in Belarus evolved from the events that happened in the history of Belarus, particularly when the "concept of equal rights for women was first developed and substantiated in the late 16th century". The so-called Grand Duchy Charter of 1588 - one of the most important legal documents in Belarusian history - protected the dignity of Belarusian women under the law. Women in Belarus and their contribution to Belarusian society is celebrated annually.
C2: n-round Factual Knowledge Dialogue
Instruction : <image> What significant historical event marked the development of equal rights for women in Belarus?
Response : The development of equal rights for women in Belarus was significantly marked by the "Grand Duchy Charter of 1588", one of the most important legal documents in Belarusian history, which protected the dignity of Belarusian women under the law.
Instruction : When is the contribution of Belarusian women to their society celebrated annually?
Response : The contribution of Belarusian women to their society is celebrated annually on the 8th of March, during International Women's Day.
Instruction : How have the modern-day characteristics of women in Belarus been influenced?
Response : The modern-day characteristics of women in Belarus have been influenced by the events that have occurred in the history of Belarus, particularly those related to the concept of equal rights for women.
Instruction : Is there a specific legal document that stands out in the history of women's rights in Belarus ?
Response: Yes, the Grand Duchy Charter of 1588 stands out as a significant legal document in the history of women's rights in Belarus. This charter protected the dignity of Belarusian women under the law.
Figure 10: The example for constructing  $n$ -round Factual Knowledge Dialogue data. The description is from Wikipedia page.
Figure 11: Message to transfer visual question answering annotations from CLEVR3D [58] to declarative sentences for 3D data.
messages = ["You are an expert in linguistics. please help me to turn these questions and answer sentences into declarative sentences, there are many question-and-answer sentences, each will be separated by ***, these is a
Are there the same number of sofas and wide sinks? no
\*\*\*
How many objects are tall beds or tall square beds? 1
There is a blue ottoman that is on the left side of the tv stand; is its shape the same as the brown tv stand? yes
There is a couch that is close by the armchair that is lying on the low white cushion; what is its color? brown
the results of the two question-answer sentences I hope you return is :
There are not the same number of sofas and wide sinks.
There is 1 object is tall bed or tall square bed.
There is a blue ottoman that is on the left side of the tv stand and its shape is the same as the brown tv stand.
There is a couch that is close by the armchair that is lying on the low white cushion and its color is brown.
Please do not add 'yes or no' word at the first of the response sentence.}]
as data source and use its original 3D bounding box annotations. Since there is no caption annotation for 3RScan, we input visual question answering (VQA) annotations from CLEVR3D [58] to GPT-API and ask it to convert the Q&A data into declarative sentences, which serves as point cloud captions in further steps. Figure 11 shows the corresponding prompts. Object attributes and relationships are extracted from scene graph annotation in 3DSSG [60]. Figure 12 and 13 show the prompts to let GPT-API generate daily dialogue and detailed description data for point clouds. Since full annotation of a scene point cloud may easily exceed input token limits of GPT-API, we randomly selected 10 captions and keep bounding box and relationships of corresponding objects as input contexts. For GPT-generated data, We limit the number of turns in each dialogue data to no more than 10, and any data exceeding this limit will be split into different samples. Figure 14 shows an example of GPT-generated data.
C4: 1-round Visual Task Dialogue. On the other hand, we also leverage annotations for existing 3D vision tasks, such as point cloud classification, 3D object detection, and CLEVR3D for 3D VQA. Similar to 2D datasets, we designed 15 templates for instruction and response by sending definitions of the corresponding tasks to GPT-API. Then instruction data are formulated by replacing keywords with corresponding annotations. Templates of 3 tasks involved are presented in Figure 31, 32 and 33, respectively.
messages = ["You are an AI visual assistant that can analyze a single point cloud. You receive five sentences, each describing the same point cloud you are observing. In addition, specific object locations within the point cloud are given, along with detailed coordinates. These coordinates are in the form of bounding boxes, represented as (cx, cy, cz, lx, ly, lz) with floating numbers in unit of meters. These values correspond to the top x, y, z coordinates of bounding box ceter and length of bounding box along x, y, z axis. It is worth noting that Attributes and Relationships of different objects are also given.   
Generate some questions and answers of the scene in a detailed manner. Instead of directly mentioning the bounding box coordinates or captions given, you should utilize this data to explain the question using natural language. Include details like the scenario, object counts, position of the objects, relative position between the objects. When using the information from the caption and coordinates, directly explain the scene, and do not mention that the information source is the caption, relationships or the bounding box. Answer questions or descriptions as if you really saw the whole scene, using the tone of seeing the scene to ask questions or answer. Please ask questions around the given category information and its corresponding description information, and don't ask questions that cannot be reasoned based on the given information. Questions and answers should be concise and effective, with reasoning logic. Answer all questions as you are seeing the point cloud. The answers should be in a tone that a visual AI assistant is seeing the point cloud and answering the question..   
Ask diverse questions and give corresponding answers. Include questions asking about the visual content of the point cloud. Only include questions that have definite answers: one can see the content in the point cloud that the question asks about and can answer confidently. All descriptions are attributes or relationships with other objects, which does not mean that there are people in the scene."]}   
for sample in fewshot_samples: messages.append {"role": "user", "content": sample['context']}) messages.append {"role": "assistant", "content": sample['response']}) messages.append {"role": "user", "content": n' join(query)}
Figure 12: Message to generate  $n$ -round Daily Conversation Dialogue data in 3D portion of our dataset.
messages = ["You are an AI visual assistant that can analyze a single point cloud. You receive five sentences, each describing the same point cloud you are observing. In addition, specific object locations within the point cloud are given, along with detailed coordinates. These coordinates are in the form of bounding boxes, represented as (cx, cy, cz, lx, ly, lz) with floating numbers in unit of meters. These values correspond to the top x, y, z coordinates of bounding box ceter and length of bounding box along x, y, z axis. It is worth noting that Attributes and Relationships of different objects are also given. You need use the provided caption, relationships and bounding box information to describe the scene in a detailed manner. Instead of directly mentioning the bounding box coordinates or captions given, you should utilize this data to explain the scene using natural language. Include details like the scenario, object counts, position of the objects, relative position between the objects. When using the information from the caption and coordinates, directly explain the scene, and do not mention that the information source is the caption, relationships or the bounding box. Answer questions or descriptions as if you really saw the whole scene, using the tone of seeing the scene to ask questions or answer. Always answer as if you are directly looking at the point cloud. You can include multiple paragraphs if necessary. Every question and answer must be related.'] for sample in fewshot_samples: messages.append(['role":"user", "content":sample['context']) messages.append(['role":"assistant", "content":sample['response']) messages.append(['role":"user", "content":"n'.join(query)])
Figure 13: Message to generate 1-round Detailed Description data in 3D portion of our dataset.
Finally, 3D portion of our dataset contains 10K samples in total, and the number of ShapeNet, 3RScan detection, CLEVR3D, and GPT-generated dialogue are 2K, 1.3K, 2K, and 4.9K, respectively.
B.3 Quality Check
In order to ensure the quality of the generated instruction tuning data, we implemented several measures. Firstly, we generate a small amount of data as a cold start and conduct manual check on the generated data. This involved carefully assessing the quality and making necessary adjustments to the message information provided as input to GPT-API. The iterative process aimed to eliminate ethical concerns and establish a strong correlation between the generated data and the corresponding inputs. We repeated this process until the desired level of quality was achieved. Once satisfied, we proceeded to generate a large volume of data. Furthermore, to verify the quality of the generated dataset, we randomly select a subset of  $10\%$  data for manual checks. This step allowed us to evaluate the generated data against our specific requirements and quality standards. During this evaluation, any formatting issues or incorrect answers generated by GPT-4 were filtered out to ensure the usability and reliability of the data. By combining manual checks during the iterative generation process and
Captions: [There is a tall box and there are 6 boxes lower than it. The table that is made of the same material as the tall box is narrow. There are 9 boxes that are the same material as the door. The chair that is the same color as the glass window is tall. There are 7 objects that are either wide boxes higher than the low box or big shelves.]
Objects: floor: brown, tiled, placing items on, same shape lab_name, [-2.25, 2.83, -1.59, 0.26, 7.81, 0.4]. shelf: placing items on, hanging, big, right box, same object type shelf, [-0.17, -2.12, -1.48, 0.16, 2.4, 1.33]. boxes: moving, placing items in, placing items on, [-1.21, 0.23, -1.46, 0.85, 0.52, 0.93] ....
Instruction: : <point cloud> What is the material of the tall box in the point cloud?
Response: The tall box in the point cloud is made of the same material as the table, but the captions do not give information on the exact material.
Instruction : "How many boxes are in the point cloud?
Response : There are a total of 16 boxes in the point cloud, with 9 of them made of the same material as the door and 6 of them lower than the tall box.
Instruction: Are there more desks or windows in the point cloud?
Response : The captions mention that there aren't more desks being supported by the table than windows in the point cloud.
Figure 14: Example of GPT-generated n-round daily dialogue and 1-round detailed description data in 3D portion of our dataset.
Instruction: <point cloud> Can you describe the scene in this point cloud?
Response : The point cloud consists of a room with tiled flooring. There are four walls with two windows supported by the leftmost wall at different heights. Besides the walls, the point cloud shows six shelves on four walls, varying in size and position, with boxes placed on and behind them. There is also a narrow table made of the same material as the tall shelf with boxes placed on it. A single door hangs on the wall, and no other objects share the identical shape. There is no matching number of doors and shelves. Nine boxes are made of the same material as the door.
subsequent random manual checks on the final dataset, we strive to ensure that the generated data meets our rigorous quality standards and aligns with the specific needs of our dataset.
B.4 Social Impact
Our dataset is a compilation of publicly available datasets that have been licensed under the Creative Commons license (CC-BY). We have taken great care to follow all necessary legal protocols to use this data in our research, and believe that transparency in data licensing is crucial for ensuring proper attribution and appropriate use of the data. Besides, the dataset includes images sourced from publicly available datasets and language data generated using the GPT-API. While we have taken steps to ensure appropriate content, we acknowledge that problematic content may exist. If you encounter any such content, please notify us immediately, and we will make necessary modifications to maintain a high-quality dataset that is free of inappropriate content. To protect the privacy of individuals and vehicles captured in the images, we plan to obfuscate sensitive information, such as faces and license plates, before publishing the dataset. We are committed to maintaining a dataset that is both high-quality and ethically responsible and pledge to uphold principles of privacy and transparency in our work.
C Benchmark
C.1 Benchmark on image tasks
We selected a set of nine commonly used CV tasks to evaluate the performance of MLLM models in our benchmark on image tasks. Our task selection criteria were based on widely studied tasks in the CV field that can showcase the MLLM model's abilities in visual interpretation, localization, and question-answering. Table 5 provides a summary of the tasks and the corresponding common evaluation metrics, which are based on the output that the MLLM models are required to generate for each task. We utilized a prompt-based approach to instruct the MLLM models to understand the task definition and generate the desired output. The ability of the models to understand and interpret the given instruction was also evaluated as part of the assessment criteria. As the models' outputs are text, we use different text-processing techniques for each task to extract entities as the final answers for evaluation. For each task, We selected datasets that are distinct from the training datasets, as our benchmark evaluation is conducted in an out-of-distribution zero-shot setting.