SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mengxue  $\mathbf{Q}\mathbf{u}^{1,3*}$  Yu Wu  $^4$  Wu Liu  $^5$  Xiaodan Liang  $^{6,7}$
Jingkuan Song  $^{8}$  Yao Zhao $^{1,2,3\dagger}$  Yunchao Wei $^{1,2,3\dagger}$
$^{1}$ Institute of Information Science, Beijing Jiaotong University  $^{2}$ Peng Cheng Laboratory
<sup>3</sup>Beijing Key Laboratory of Advanced Information Science and Network Technology
$^{4}$ Wuhan University  $^{5}$ JD Explore Academy  $^{6}$ Sun Yat-sen University  $^{7}$ MBZUAI
<sup>8</sup>University of Electronic Science and Technology of China
qumengxue@bjtu.edu.cn, wuyucs@whu.edu.cn, yunchao.wei@bjtu.edu.cn
Intention-oriented object detection aims to detect desired objects based on specific intentions or requirements. For instance, when we desire to "lie down and rest", we instinctively seek out a suitable option such as a "bed" or a "sofa" that can fulfill our needs. Previous work in this area is limited either by the number of intention descriptions or by the affordance vocabulary available for intention objects. These limitations make it challenging to handle intentions in open environments effectively. To facilitate this research, we construct a comprehensive dataset called Reasoning Intention-Oriented Objects (RIO). In particular, RIO is specifically designed to incorporate diverse real-world scenarios and a wide range of object categories. It offers the following key features: 1) intention descriptions in RIO are represented as natural sentences rather than a mere word or verb phrase, making them more practical and meaningful; 2) the intention descriptions are contextually relevant to the scene, enabling a broader range of potential functionalities associated with the objects; 3) the dataset comprises a total of 40,214 images and 130,585 intention-object pairs. With the proposed RIO, we evaluate the ability of some existing models to reason intention-oriented objects in open environments.
Category-based object detection has been extensively investigated in recent years [1, 35, 5, 44], where common object categories are typically defined based on object names, such as "cat", "dog", "car", and so forth. These name-based categories facilitate the recognition and classification of objects with similar visual characteristics by computer systems. However, variations in category definitions among individuals pose challenges in predefining categories for all objects worldwide. In reality, individuals tend to search for objects based on their immediate needs rather than predefined category names. This challenges the traditional approach of category-based object detection, as the focus shifts towards objects that fulfill specific functional requirements.
Previous research [6, 32, 30, 47, 28] has explored the concept of "Affordance" to establish a connection between functional requirements and objects. "Affordance" refers to the function or potential action that an object offers for utilization. For instance, when seeking to rest, one may look for objects that afford the action of "lying down", such as a "bed", "sofa", or "bench". However, this approach can lead to ambiguity as the same verb may apply to multiple objects. For example, the verb "play" could refer to playing frisbee or playing the piano. Previous works use a limited number
*Work done during an internship at JD Explore Academy.
†Co-corresponding author.
Aff.Det.
Affordance class: ride
Closed set
Affordance classes:
Play/wear/sit/cut/ride/brush.
TDOD
Task6: Get lemon out of tea
Task1: step on something
Task 14: pound carpet
REC
Open set
Figure 1: Comparison between tasks: Affordance Detection (Aff. Det.), Task-Driven Object Detection (TDOD), Referring Expression Comprehension (REC), and Intention-Oriented Object Detection (IOOD). Closed set: Aff. Det and TDOD are set to detect the object that matches the class or task of the limited closed set from the given image. There are limited affordance classes in Aff. Det. and 14 predefined tasks in TDOD. Open set: REC is set to detect the object that matches the referring expression mainly containing category, attribute, and relationship to other objects while there is no explicit category in IOOD. Only the intention description is given in IOOD.
Sentence: The second yellow and blue bus from the left.
Category/Attribute/Relation
IOOD
Sentence: Something to lounge on and relax while reading a book.
Intention Description
of verb phrases to represent an intention, which fails to capture the diverse range of human intentions necessary to recognize the desired object accurately.
To address the issue of intention-oriented object reasoning in open environments, we introduce the RIO dataset in this paper, which is based on the images and annotated objects of the COCO dataset [20]. Compared to phrases, we consider sentences to be more descriptive and contain more relevant information. Our dataset contains multiple objects labeled with intention descriptions related to the scene. These descriptions are generated by conversing with ChatGPT and are comprehensive intention sentences rather than simple verb phrases, e.g., the description for a frisbee is "Something you can use to play games with your dog.", while for a piano, the description is "Something you can use to create music." The RIO dataset comprises 40,214 images and 130,585 intention-object pairs, where each intention-region pair refers to an intention description and its corresponding object in the image. It is worth noting that in our dataset similar intention descriptions may have different selected objects depending on the scene. For instance, in a park when the intention is to "lie down and rest", the objects like a "bench" or "grass" may fulfill the need rather than a "bed" or "sofa". This indicates that intention reasoning is challenging, which requires considering the environment and diverse intention descriptions rather than relying solely on category mapping and limited predefined intentions. In addition to this, we constructed an uncommon test set especially for some long-tail intentions, which are used for testing the model's few-shot capability.
We construct solutions for intention-oriented object reasoning in open environments based on existing methods and evaluate their capability with our proposed dataset. Our work contributes to the advancement of object recognition and reasoning by highlighting the significance of intention-based approaches in open environments. The RIO dataset provides a valuable resource for further research and development in intention-oriented computer vision tasks. Future work can explore the potential of incorporating context and environment reasoning into intention-based object recognition systems, leading to more robust and context-aware computer vision models.
2.1 Affordance Detection
Affordance Detection is a critical component in intelligent systems as it empowers machines with the vital ability to understand and adapt to their environment and has important implications for human-robot interaction and machine decision-making. Affordance detection, on the one hand, is studied in physical machine systems and can facilitate robot-environment interactions. Grabner et al. [9] used 3D models of the human skeleton to learn how to sit on a chair and infer whether an object can provide the affordance of "sitting". On the other hand, affordance detection has also been studied in the field of vision only. Early work perceived affordances by establishing associations between object surface features and affordances, and Myers et al. [32] proposed a framework to
Table 1: Comparison with previous affordance detection datasets and task-driven object detection datasets. Our descriptions are natural sentences while others are words or phrases, and our descriptions are scene-related. The number of our affordance vocabulary is much higher than other datasets.
Dataset	ADE-Aff	PAD	PADv2	PAD-L	COCO-Tasks	Ours
# images	10,000	4,002	30,000	4,002	39,724	40,214
# categories	150	72	103	72	49	69
intentions	verb words	verb words	verb words	phrases	phrases	sentences
# affordance	7	31	39	31	14	>100
scene-related	×	×	×	×	×	✓
jointly localize and identify the affordances of objects and presented the first pixel-level labeled affordance dataset.
However, the affordance of an object does not only correspond to surface features, it is also closely related to human interaction. Therefore, to address the practical problem of affordance inference, Chuang et al. [6] considered the physical world and social norms and constructed the ADE-Affordance dataset [6] based on ADE20k [49]. In reality, affordance is related to the purpose of human behavior, and considering the existence of multiple potential complementarities between animals and their environment as defined by Gibson, which leads to multiple possibilities for specific affordances, Luo et al. [30] constructed a purpose-driven affordance dataset PAD [30] and PADv2 [47] that considers the relationship between human purpose and affordance, covers more complex scenarios, extends the number of affordance categories, and is more applicable to practical robotic applications.
The aforementioned work focuses on computer vision. Further, Lu et al. attempted to investigate affordance detection involving natural language, i.e., inputting a set of phrases and images describing affordances with the expectation of being able to segment out the corresponding objects. This is consistent with the reality that robots receive information from multiple sources. Thus the PAD-L dataset [28] is constructed based on the PAD dataset, in which the phrase collection is based on a limited affordance dictionary paraphrase. However, this form of expression is still not natural enough to match the natural human daily language description habits. In addition, people not only consider affordances when selecting objects but for objects with similar affordances, depending on the specific task, people will select targets that better match the task. Therefore, Sawatzky et al. [37] proposed task-oriented object detection and constructed the COCO-Task dataset based on the COCO dataset. In the COCO-Task dataset, people select the best class of objects in the images for different tasks. However, this dataset only predefined 14 tasks and used phrase representation, which lacked diversity.
Therefore, in this paper, we tried to reason about the corresponding objects using a more natural form of expression than affordance phrases, i.e., sentences of intention description. We argue that predefined categories of affordances limit more possibilities for human-object interaction, while open-world natural sentences can cover more affordance situations.
2.2 Referring Expression Comprehension
Benefiting from the development of large-scale visual language pre-training models, many transformer-based models have been proposed, such as VL-BERT [38], ViLBERT [27], UNITER[4], 12in1 [26], and MDETR [14]. In addition to REC, there has been a lot of research [26] related to the Referring Expression Segmentation (RES) task, which is to generate a binary mask based on the referring expression. Recently, a simple and universal network term SeqTR [50] is presented, which regards both REC and RES as a point prediction problem. SeqTR greatly reduces the difficulty and complexity of both architecture design and optimization. Based on SeqTR, Polyformer extends the sequence prediction of the mask when there are multiple contour polygons. Furthermore, by randomly downsampling dense contour points during training, the model can predict better contours with adaptive fitting in evaluation.
Previous work has more fully investigated the noun-based REC model [10, 11, 21, 39, 40, 41, 45, 48, 51, 3, 19, 36, 43, 42] and the RES model [29, 7, 8, 2, 12, 13, 17, 50, 7, 34], and great progress has been achieved in terms of performance. However, in real-world applications, such as intelligent service robots, system input is typically in the form of affordances (i.e., the ability to support an action or speak a verb phrase). Whether modern visual language models are designed to effectively understand verb references remains under-explored. Li et al. [18] proposed TOIST to distill the noun
Step1: Image Selection
Step2: Intention Generation
Step3: Manual Selection
Figure 2: Step-by-step diagram of data collection. We first select those images with more than 3 categories of instances, then we chat with GPT to generate the draft of the intention description. For the draft description, we manually determine whether it is the best match and finally perform sentence conversion.
Step4: Sent. Modification
knowledge learned from REC models into verb comprehension models to solve Task-oriented Object detection and segmentation tasks.
In our work, with diverse descriptions of intentions, we validate the ability of TOIST models to reason about intention-oriented objects in the open environment. Also, we build baselines based on several recent REC and RES models to test their capabilities on our proposed RIO dataset.
3 RIO Dataset
In this section, we present the process of collecting the RIO dataset in Sec. 3.1, and then we provide statistics on the dataset distribution in Sec. 3.2. In Sec. 3.3, we propose evaluation metrics for the Intention-Oriented Object Reasoning task.
3.1 Dataset Collection
Our annotations are based on images and instance annotations from the COCO2014 dataset. The process of data collection is illustrated in Fig. 2. We collected two types of intention description annotations, called common and uncommon.
3.1.1 Collection of Common Intentions
Step1: Image Selection We select the categories in the COCO dataset that are suitable for Intention-Oriented Object Reasoning, i.e., categories with more affordances. We remove the category "People" because "People" is the one that generates Intention. At the same time, since the affordance of "food" is usually only "eat" and as an ingredient, we also remove the class related to "food". We want to interact in a complex open environment to collect more diverse descriptions of human intent, so we filter the image scenes from the COCO image set to include instances of more than 3 classes.
Step2: Intention Generation We generate intention descriptions for each category of instances in each image. We do not consider which object in the same category is more consistent with the intention description, but only which category of objects in the image is more consistent with the intention description. For example, as shown in Fig. 2, although there are multiple tables and sofas in the image, we only distinguish between categories when generating intention descriptions in Step2.
Due to the development of LLMs, ChatGPT is able to simulate human intentions and interactions to a certain extent, so we collect a draft description of intentions by talking to ChatGPT. In order to make ChatGPT "see" the image content, we first provide a Caption describing the content of the image, and then we ask ChatGPT "what interaction it can have with a certain category of objects". Using the gpt3.5 model, we ask ChatGPT questions in the following format:
Question: "Suppose you are in a place: [Caption], What can you do with the [Obj]?
Step3: Manual Selection We present the annotated images, instances, and intention description drafts from the previous steps to human annotators, who are asked to check the clarity of the intention statements and remove those intention descriptions that are irrational and unconventional. In addition, they are also asked to determine whether the intention and the object are the best match and to remove
(a) Something you can use to cut the cake into slices.
(b) Something you can use to scoop the grilled fixings onto the buns.
(c) Something you can use to cut the strawberries and chocolate into small pieces.
Figure 3: Uncommon Intentions. We additionally collected some uncommon intents. for example, "cut" is an uncommon intention for spoons and forks.
(d) Something you can use to cut the pizza into smaller pieces.
Knife--Cut
Spoon -- Scoop
Spoon -- Cut
Fork--Cut
those data pairs that are not the best match. Since we ask category-by-category during the collection process, there may be two categories of objects with similar intention descriptions, although the sentences may not be identical. E.g., for the intention description "You can sit comfortably in a chair and read a book", there is a "sofa" in the diagram to sit more comfortably, so the original intention description of the chair should be deleted. Since human priority judgments are also more subjective, we only do careful filtering for the val and test sets. For the train set, the per-class intention description is retained, but we observe that there are also certain statistical properties in the train set, for which the more matching intention-instance pair appears more often.
Step4: Sentence Modification In practice, we should not use "You can use the [obj] to ..." to express our intention directly, but "I need something to ..." or "Something I can use to ..." . Therefore, we automatically modify the sentence to remove the category name contained in the sentence.
3.1.2 Collection of Uncommon Intentions
We find that for some categories, there may be some less-used affordances. They may be used as a substitute for some other objects to perform some functions commonly used by other objects. As shown in Fig. 3, knife usually corresponds to cut, and spoon often corresponds to scoop. Spoons and forks are not generally used to cut things, but they do have the ability to cut soft things.
First, we analyze the intention descriptions of various categories to identify potential uncommon category-intention pairs. Subsequently, we create category mapping charts that suggest possible replacements for each other. We asked ChatGPT questions as follows:
Question: "Suppose you are in a place where: [caption]. Can the [original class] be used as a [substitute class]? Please answer yes or no. If yes, what can you do with the [original class]?
For the spoon, the question is like "Suppose you are in a place where: [caption]. Can the spoon be used as a knife? Please answer yes or no. If yes, what can you do with the spoon?", and the answer generated by ChatGPT is "You can use the spoon as a knife to cut the strawberries and chocolate into small pieces." The final intention description is "Something you can use to cut the strawberries and chocolate into small pieces".
Although we pre-define the category maps for mutual substitution, by asking ChatGPT in this format, we still get open-intention descriptions with diverse language expressions.
Our dataset consists of 40,214 images and 130,585 object-intention description pairs. Our training set images are from the COCO train and account for  $33\%$  of the original COCO [20] train set, and the test set images are from COCO val and account for  $30\%$  percent of the original dataset. There is no overlap between COCO training images and our test images, so models pre-trained on COCO can be used and evaluated fairly. The test set is divided into two parts according to common and uncommon, and the images they use are overlapping, differing in the annotation of intentions. The final dataset contains 96,415 data pairs (27,696 images) for training, 29,344 data pairs (12,392 images)
Common Intentions
Uncommon Intentions
Figure 4: Word clouds of partial categories from the common set.
for common tests, and 4,248 data pairs (4,826 images) for uncommon tests. Common intentions are more consistent with the distribution of the train set, and uncommon intentions are an expansion of the common long-tail data, to test whether the model can few-shot after training with various intentions. Fig. 4 shows some word clouds and Fig. 5 illustrates the statistics of the relationship between categories and verbs. For each category, an average of 158 verb words appear within their descriptive sentences. There are about 300 (20%) verbs that appear in the intention descriptions of more than 10 categories, and thus cannot be detected by constructing a verb-noun mapping table, but must instead reason the target by intention, which makes the ROI task more challenging.
The task of intention-oriented reasoning is to locate objects that satisfy the intent given an intention description sentence. In order to facilitate the study of downstream application scenarios, we divide the task into two subtasks, intention-oriented object detection (IOOD) and intention-oriented instance segmentation (IOIS). We define different metrics for IOOD and IOIS in order to comprehensively evaluate the model performance.
Intention-oriented object detection We use Top1-Accuracy and AP@0.5 as evaluation metrics. Top1-Accuracy indicates that the model only needs to predict one correctly to understand the intention. We calculate the IoU (Intersection over Union) between each prediction bounding box and the ground truth bounding box and set the IoU threshold  $>0.5$  for correct prediction. However, we also want the model to have the ability to recall all objects that satisfy the intention, so we use the AP@0.5 object detection evaluation metric of COCO detection.
Intention-oriented instance segmentation We use Top1-IoU and mIoU as evaluation metrics. Similar to IOOD, Top1-IoU only calculates the IoU between the highest probability predicted object and the ground truth object masks. While mIoU (mean Intersection over Union) is the average of the IoU calculated for all preferred categories of objects.
4.1 Construction of Detector-Reasoner Baseline
A straightforward baseline approach is to conceptualize the Intention-Oriented Object Detection (IOOD) task as a combination of Object Detection and Question Answering (QA). This can be realized by architecting a hybrid model comprising an object detector paired with an LLM (Language Language Model) reasoner. In the training of IOOD, the goal is to predict the object in an open environment based solely on the open-set intention description, without predicting predefined categories. To ensure the fairness of the comparison, we use the open-set object detection SOTA model, GroundingDINO [23], which is also class-agnostic, as the object detector. We directly utilize ChatGPT as our Language Reasoner.
GroundingDINO is trained without being confined to a specific category range, enabling it to detect objects from any language input. For the IOOD task, we utilize the COCO category list as input, facilitating zero-shot inference in GroundingDINO. Then we set a score threshold to filter the box predictions. This results in a list of candidate object bounding boxes represented as  $[box_1, box_2, \dots, box_n]$ , and their associated category labels, denoted  $[label_1, label_2, \dots, label_n]$ . With this information, we craft a prompt for ChatGPT structured as "In categories  $[label_1, label_2, \dots, label_n]$ , which category can you use to...?" ChatGPT subsequently selects the appropriate category. The final output from the hybrid system is the bounding box of the selected category.
(a) Categories to verbs
(b) Verbs to categories
Figure 5: Dataset Statistics. (a) shows the statistics of the verb diversity in intention descriptions for each category, and (b) shows the frequency of occurrence of a verb in different categories of intention description. The horizontal coordinates are only examples of some categories and verbs.
4.2 Construction of End-to-End Baselines
In REC and RES tasks, there are many models and paradigms that primarily focus on noun-oriented reasoning and lack exploration of the understanding of verb-oriented intention. Since our task is similar to REC and RES, we construct baselines based on several state-of-the-art methods, i.e., MDETR [14], SeqTR [50] and Polyformer [22] in REC and RES to evaluate their capability in verb-oriented intention reasoning. Additionally, a recent work TOIST [18] investigates task-oriented object detection and segmentation, and we also build a baseline based on it. In this section, we briefly review four related models and provide a detailed description of the baselines built upon them.