SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MDETR [14] is a Vision-Language (V-L) pre-training model which can achieve good performance after finetuning on VQA, REC, and other downstream V-L tasks. In pre-training, the training goal is set as follows: input a sentence and an image, make the model detect the objects corresponding to the nouns involved in the sentence, and draw the bounding boxes in the image. The object list corresponds one-to-one with the noun list. While in the task of intention-oriented reasoning, the form of the sentence is like "something you can use to ...", we construct a correspondence between the objects and the word "something" only. The original model and the loss function remain unchanged, in this way the model can be trained to recognize objects referred to by the intention description.
TOIST [18] is an improved model based on MDETR. It is designed to solve task-oriented object detection and segmentation tasks. Considering that the trained MDETR has the noun recognition ability that may be useful for verb recognition, an attempt is made to retain this ability by noun-pronoun distillation in TOIST. Therefore, following TOIST, we first train the model with sentences with category nouns, e.g., "You can use the knife to cut cakes.", then distillate to reason "You can use something to cut cakes."
SeqTR [50] is the first work to unify both REC and RES as sequence prediction. In previous works, both MDETR and TOIST train REC and RES by detection head and segmentation head respectively, but in SeqTR, REC is defined as predicting the coordinates  $(\mathrm{x},\mathrm{y})$  of the top-left and bottom-right points of the bounding box, and RES is defined as predicting the sequence of contour points of the binary-mask. It is well worthwhile to test how this latest sequence prediction paradigm performs on IOOD tasks. Since this model can only predict a single object, in order to be able to roughly evaluate the ability of the sequence prediction model, we change the training target to predict the largest object that can satisfy the intention, and we only measure Top1-Accuracy and Top1-IoU when testing.
Polyformer [22] improves on the base of SeqTR. Considering that many objects are divided into multiple parts due to occlusion or their own shapes, Polyformer, a model that can be used for multiple polygon contour point prediction, is constructed. At the same time, this ability to predict multiple
Table 2: Comparison of baselines. "D-R" denotes the Detector-Reasoner hybrid model. Both D-R and D-R* employ GroundingDINO [23] as their object detector. D-R uses parameters from the GroundingDINO model that have not been fine-tuned on the COCO dataset, while D-R* uses parameters that have been fine-tuned on the COCO dataset.
Method	Object Detection	Instance Segmentation
common AP@0.5 Top1-Acc	uncommon AP@0.5 Top1-Acc	common mIoU Top1-IoU	uncommon mIoU Top1-IoU
D-R	38.30	58.21	10.67	33.07	-	-	-	-
D-R*	45.15	64.77	17.25	41.98	-	-	-	-
MDETR [14]	48.61	65.05	24.20	39.60	44.14	54.55	22.03	34.35
TOIST [18]	49.05	66.72	21.96	39.28	45.07	55.85	19.41	34.00
SeqTR [50]	-	67.44	-	23.24	-	47.26	-	29.91
Polyformer [22]	-	-	-	-	48.75	-	26.77	-
polygons can be used exactly for inference when the intention description corresponds to multiple instances. We train and test Polyformer to reason intention-oriented objects on the RIO dataset while keeping the model and loss function unchanged.
To make the most of the pre-trained noun recognition, we train all four models based on pre-trained parameters, which are trained on the merged region descriptions from Visual Genome (VG) [16] dataset, annotations from RefCOCO [46], RefCOCO+ [46], RefCOCOg [31], ReferItGame [15], and Flickr entities [33]. The total pretraining data is approximately 6.1M distinct language expressions and 174k images in the train set. The learning rate and backbone used for each model are basically the same as the original model, with some slight adjustments that are detailed in the supplementary material. All of our experiments are completed on four RTX 3090s.
4.4 Comparison of Baselines
As shown in Table 2, we report the results of four models on intention-oriented object detection and instance segmentation, including common set and uncommon set. Since SeqTR is a prediction model for a single object, we calculate Top1-Accuracy and Top1-IoU according to the only prediction result which is regarded as the highest probability. As for the multiple polygon prediction model Polyformer, we only report the mIoU results because there is no probabilistic prediction for multiple polygons and the output order of each object is random.
In terms of the results, end-to-end methods are superior to the Detector-Reasoner hybrid approach. This superiority can be attributed to inherent performance limitations associated with object detectors. For end-to-end methods, we can observe that the sequence prediction model SeqTR has a higher Top1-Acc than both MDETR and TOIST in the IOOD, but it can only recall one object. The noun-pronoun distillation introduced by TOIST makes some performance improvements compared to MDETR in the common set, but the performance degrades in the uncommon set. In the intention-oriented instance segmentation, the multi-polygon sequence prediction model Polyformer has a higher mIoU than MDETR and TOIST, and SeqTR has a lower Top1-IoU than MDETR and TOIST.
The same phenomenon in both tasks is that the performance is acceptable on the common set, but the performance drops very much on the uncommon set. This indicates that the model's ability to few-shot is relatively weak, and future work should mainly explore how to use common data training to improve the inference ability on uncommon data.
In this section, we first analyze the advantages of end-to-end methods over hybrid methods. In addition, we perform ablation studies on MDETR and TOIST model structures and investigate the effect of sentence forms and distillation on the model.
Table 4: Ablation studies in MDETR [14] and TOIST [18] model structure. "Sent. Form" is Sentence Form.
Model	Sent. Form	Pretrain	Confidence	AP@0.5
(a) MDETR	"You can use something to ..."	×	'something'	2.57
(b) MDETR	"You can use something to ..."	✓	sentence	27.6
(c) MDETR	"You can use something to ..."	✓	'something'	48.16
(d) MDETR	"You can use [obj] to ..."	✓	'[obj]'	32.64
(e) TOIST	"You can use [obj]->something to ..."	✓	'something'	49.00
4.5.1 Advantage Analysis of End-to-End Approach over Hybrid Approach
Performance Advantages In terms of results, visual-text alignment is superior to the Detector-Reasoner hybrid approach. The final performance of the hybrid method can be affected by both bounding box accuracy and category accuracy. Misclassification of categories can lead to incorrect judgments when integrating with LLM.
Generalization Advantages One perennial challenge in open-set object detection is the inconsistent labeling of the same object by different annotators. This inconsistency arises because various annotators might use different category names for the same object, making standardization a complex task. To illustrate that, we conducted a simple ablation study. By replacing original COCO category terms with commonly used synonyms, the performance of GroundingDINO on the
Category Name	AP	AP50
Original	48.4	64.6
Synonymous	35.6	47.5
Table 3: Simple ablation of categories with synonym replacement.
COCO2014 validation set exhibited a noticeable decline, as shown in Table 3. This observation emphasizes the motivation behind our IOOD task - our goal is to avoid intricate name definitions. Instead, the IOOD task focuses on detecting objects based on their functions and human intent, which encourages the model to understand the essence of the object, not just its categorical label.
Structural Advantages Compared with end-to-end models, hybrid models are limited by the capabilities of their individual components and often struggle to utilize the combined feature information from vision and language. Furthermore, in hybrid models, there's an increased risk of error propagation. In contrast, end-to-end systems streamline these processes, optimizing the synergy between vision and language while reducing cascading errors.
4.5.2 The Effect of Model Structure and Sentence Forms
As indicated in Table 4, comparing (a) and (c), the performance is significantly degraded when the model is not pre-trained. This shows that the ability of noun object detection learned during pre-training is helpful.
As we know in MDETR, the target of pre-training is to predict the object corresponding to each noun contained in the sentence. When MDETR is finetuned on the REC task, it shares the confidence weight equally for each word in the sentence because the whole sentence refers to an object. However, this approach may not be suitable in RIO because our sentences are much longer compared to the sentences in REC. Therefore, we set the confidence weight to be applied only to the word "something" instead of sharing it equally throughout the sentence. Comparing (b) with (c), we found that using the confidence of only the word "something" instead of the whole sentence, the performance improves. In (d), we introduce the object noun [obj] into the sentence, and the detection accuracy is indeed enhanced compared to the equivalent condition (b). (e) indicates the performance gain of using noun-pronoun distillation, i.e. distillation from "[obj]" to "something".
4.6 Qualitative Result
We visualize some of Polyformer's prediction results as examples in Fig. 6. (a)-(c) are the results of the common set, and (d) is the results of the uncommon set. The bounding box in the figure is of all instances in the Polyformer model, not the per-instance bounding boxes in the intention-oriented object detection task. The model can generally predict objects that match the intention description, and can also identify the uncommon intention.
(a) You can store and chill food and drinks in the thing.
(b) You can sit in the things to cook and eat at the table.
(c) You can use the thing to view documents, browse the internet, and use various programs and software.
Figure 6: Visualization of Polyformer in IOIS. (a)-(c) are the results of the common set, and (d) is the results of the uncommon set. The first line is the ground truth, and the second line is the prediction polygons. The bounding box in the figure is of all instances in the Polyformer model, not the per-instance bounding boxes in the intention-oriented object detection task.
(d) You can use the thing to fold out the bottom cushion and lay down a mattress or other sleeping surface.
In conclusion, this paper addresses the limitations of category-based object detection and introduces the RIO dataset, which focuses on intention-oriented object reasoning in open environments. The dataset consists of images from the COCO dataset and includes multiple objects labeled with comprehensive intention descriptions. Unlike simple verb phrases, these descriptions capture diverse human intentions related to the scene. The dataset contains a large number of intention-object pairs, allowing for robust evaluation of intention reasoning algorithms. By leveraging existing methods, we constructed solutions for intention-oriented object reasoning and evaluated their performance using the RIO dataset. The results demonstrate the importance of considering diverse intention descriptions and reasoning about the environment when recognizing the desired object accurately. This research contributes to the advancement of object recognition and reasoning by highlighting the significance of intention-based approaches in open environments. The RIO dataset provides a valuable resource for further research and development in intention-oriented computer vision tasks. Future work can explore the potential of incorporating context and environment reasoning into intention-based object recognition systems, leading to more robust and context-aware computer vision models.
This work was supported by the Fundamental Research Funds for the Central Universities (2023JBZD003), the National NSF of China (No.U1936212, No.62120106009), the Fundamental Research Funds for the Central Universities (No. K22RC00010), and Beijing Nova Program (NO. 20220484063).
A.1 General Discussion
Limitation Since we use ChatGPT to generate intention descriptions for each class of instances, the diversity and reasonableness of the intention descriptions depend on the performance of ChatGPT, and the manual filtering stage only filters inappropriate intention descriptions without the modification of intention descriptions. Meanwhile, the subjectivity of object usage for tasks can lead to ambiguity in determining task fulfillment criteria. Although we have provided clear guidelines to annotators and conducted multiple quality checks to ensure consistency, there still remains some subjectivity inconsistency.
Ethical Concerns Our dataset is constructed based on the existing publicly available dataset MSCOCO [20], so there are fewer privacy concerns. For the object instances in the images that are already labeled, our RIO dataset only supplements the available intention description of the instances. Our process of collecting the available intention of instances is a combination of automated and manual, where the construction of most of the matching sentence is automatic and only requires manual prioritization based on human preferences. Overall our approach is relatively environmentally friendly and less costly. Since the intention description is generated from conversations with ChatGPT, it may be affected by the bias of the ChatGPT training data, though it can be filtered out in the manual confirmation stage.
License MDETR [14], SeqTR [50], and PolyFormer [22] follow Apache License 2.0, TOIST [18] follow MIT License follow Apache License 2.0. We used author-released code for baseline construction. MSCOCO [20] annotation data is licensed under a Creative Commons Attribution 4.0 International License. In addition, ChatGPT is an AI language model developed by OpenAI, which is based on GPT-3.5 architecture. Specific licensing information for the GPT-3.5 model is a trade secret of OpenAI and therefore precise licensing details cannot be provided.
A.2 Comprehensive Discussion about the Use of ChatGPT
Quality ChatGPT has been trained on a vast amount of text data, enabling it to generate high-quality text outputs. We randomly sampled 500 original answers generated by ChatGPT and assessed their suitability for dataset construction based on accuracy, completeness, and fluency. The usability rate of the data was  $96.2\%$ .
Diversity When using ChatGPT, the value of temperature can be adjusted to control the randomness of the output. The higher the value (closer to 1), the more varied and creative the output; the lower the value (closer to 0), the more deterministic but also potentially too repetitive the output. We set temperature=0.7 to ensure the diversity of the model output. In addition, WordCloud in Figure 4 and the dataset statistics in Figure 5 also illustrate the diversity of responses generated by ChatGPT.
Human-likeness ChatGPT is already been implemented as a product and, from a language expression perspective, it can provide relatively accurate descriptions of human intentions. From the perspective of linguistic naturalness, the 500 answers randomly sampled from original answers generated by ChatGPT all meet the standards of human likeness.
To make the most of the pre-trained noun recognition, we train all four models based on pre-trained parameters, which are trained on the merged region descriptions from Visual Genome (VG) [16] dataset, annotations from RefCOCO [46], RefCOCO+ [46], RefCOCOg [31], ReferItGame [15], and Flickr entities [33]. The total pretraining data is approximately 6.1M distinct language expressions and 174k images in the train set. The learning rate and backbone used for each model are basically the same as the original model, with some slight adjustments that will be detailed in the supplementary material. All of our experiments were completed within four RTX 3090s but with different learning rates and training epochs. The details are as follows.
(a)Something you can use to fill with water and hold flowers or other decorates.
(b)Something you can use to flip it into a sofa.
(c)Something you can use to provide a stable surface for writing or eating.
(d)Something you can use to cut the pizza into slices.
(e)Something you can use to serve snacks and treats to the boys and animal friends.
(f)Something you can use to turn it into a makeshift bed by adding pillows.
Figure 7: The failure cases predicted by the TOIST model on the RIO dataset uncommon set, the blue box is the ground truth, and the red box is the prediction result. When no object in the model prediction result reaches the probability threshold, the output is the empty set.
(g)Something you can use to hold it above your head to keep you dry when raining.
(h) Something you can use to cut the sandwich and other items on the plate.
MDETR Following MDETR [14], all parameters in the network are optimized using AdamW [25] with the learning rate warm-up strategy. The model is trained with a batch size of 80. We set the learning rate of the language backbone RoBERTa [24] to be  $1 \times 10^{-5}$ , and all the rest parameters to be  $5 \times 10^{-5}$ . In initial training, the model is trained for 30 epochs. We randomly resize and crop the image, and set the maximum side length of the input image as 640 while keeping the original aspect ratio. Images in the same batch are padded with zeros until acquiring the largest size of that batch. Similarly, sentences in one batch will be adjusted to the same length as well.
TOIST Following TOIST [18], both the student and teacher TOIST models are initialized with the model pre-trained by MDETR [14] and use the AdamW [25] optimizer. The batch size is set as 80. We fine-tune the teacher model on our RIO dataset for 30 epochs. Then we use the fine-tuned teacher model to distill knowledge to the student model for 15 epochs. The initial learning rates are set to  $1 \times 10^{-5}$ ,  $1 \times 10^{-5}$ ,  $5 \times 10^{-5}$  for the language encoder, visual encoder, and other parts of the model.
SeqTR Following SeqTR [50], all parameters in the network are optimized with AdamW [25], and the batch size is 128. We load the pre-trained parameters and finetune the model with 30 epochs. The learning rate is set as  $5 \times 10^{-4}$ . Images are resized to  $640 \times 640$ , and the length of language expression is trimmed at 20. We set the number of sampled contour points as 18 points.
PolyFormer Following PolyFormer [22], we use the AdamW optimizer [25]. The initial learning rate is  $5 \times 10^{-5}$  with polynomial learning rate decay. We load the pre-trained parameters on combined large-scale visual grounding data and finetune the model on the RIO dataset for 100 epochs with a batch size of 32. Images are resized to  $512 \times 512$ , and polygon augmentation is applied with a probability of  $50\%$ . The 2D coordinate embedding codebook is constructed with  $64 \times 64$  bins.
C.1 Failure Cases Study
As shown in Figure 7, we visualize some prediction failures of the TOIST model on the RIO dataset uncommon set, the blue boxes are the ground truth and the red boxes are the prediction results. When no object in the model prediction result reaches the probability threshold, the output is the empty set, i.e., no prediction result, and no red box. We find that among the uncommon set prediction failure samples, (a), (c), and (h) are the failure samples where the prediction is partially correct, but not the complete prediction. Among them, (a) is a priority judgment error. In (a), both bottle and cup can be used as alternatives to the vase to mount flowers in special cases, but the bottle is more suitable than the cup and only the bottle should be predicted. (c) and (h) may be an incomplete prediction due to masking. (b), (c), (e), (f), and (g) are all samples with no correct predictions, and the model does not seem to be able to reason about these uncommon functions.
(1) Something you can use to sleep on by placing a sleeping bag on top of it.
(2) Something you can use to cut the strawberries and chocolate into small pieces.
(3)Something you can use to fill it with water and add some fresh flowers.
(4) Something you can use to open it up and use it as a cover from the rain.
(5) Something you can use to write, study, or for other activities.
(6) Something you can use to cut up the kiwi and peach pieces on the plate.
Figure 8: The case study of the uncommon set predicted by the TOIST model, the blue box is the ground truth, and the red box is the prediction result. When no object in the model prediction result reaches the probability threshold, the output is the empty set.
(7) Something you can use to fill it with flowers or other decorative items.
(8) Something you can use to hold over your head to keep out the rain.
C.2 Cases Study of Uncommon Set
In Fig. 8, we visualize more results in the uncommon set for observation. The blue boxes represent the ground truth, and the red boxes denote predictions.
It is discernible that for analogous tasks if the shape of the target object is similar to that of commonly observed objects, it is more readily detected. For instance, in (1), the shape of the bench is reminiscent of a bed, leading to a more accurate identification. Conversely, in (5), objects typically associated with actions like "write" and "study" are desks; given the stark contrast in appearance between benches and desks, the former is not detected.
For objects like cups, which frequently appear alongside water, detection can be influenced by context. Even though both scenarios in (3) and (7) aim to use a cup as a surrogate for a vase's function, the presence of "water" in the sentence in (3) versus its absence in (7) leads to different detection outcomes. For items with more variable appearances, such as "backpacks", the detection outcome can vary even if the intention description remains relatively consistent. For example, a horizontally placed backpack more closely resembles the shape of an umbrella, as seen in (4).
According to the detection results of the current models, it is evident that they primarily rely on appearance for judgments. Incorporating reasoning with external knowledge from large language models in the future should offer a more effective solution to challenges related to uncommon intentions.
D Details of Data annotation
D.1 Data Annotation Platform
We build a simple annotation platform as shown in Fig. 9. On the left is the image and the object bounding boxes, and on the right is the intention description of the object. The annotation goal is to check whether the intention description is appropriate for the object boxed out in the figure and whether there are other objects in the figure that better match the description.
The checking scores are divided into 3 grades:
Fit-and-Best The description is reasonable and feasible (Fit) and the object class is the best choice that will satisfy the intention (Best).
Not-the-Best The description is reasonable and feasible (Fit) but there are other object classes in the image that are better suited to this intention.
Unfit Unreasonable description or unsuitable for intention-oriented object detection.
bowl: You could use the bowl to mix ingredients for the pastries.
Figure 9: Screenshot of the data annotation platform.
(1) You can relax on the chair and watch TV or use it as a sleeping space.
(2) You could use the bottle to pluck the strings of the guitar.
(3) You can board and ride the car to your destination.
(4) You can use the bowl to scoop up and spread the cheese onto the pizza.
(5)You can use the refrigerator to store and cool food and beverage.
(6) You can mix the carrots and broccoli together and enjoy a healthy snack.
(7) You can scoop up the banana split with the spoon and enjoy its deliciousness.
(8) You cannot do anything with the car, as it has nothing to do with the football game.
Figure 10: Typical examples of dropped data from original data.
The human annotation statistic numbers are tabulated in Table 5.
Annotator Num.	Annotation Num.	Annotation Time	Annotation Consistency
10	~5000 per person	~10 days	93.7%