# Open-Vocabulary Part-Level Detection and Segmentation for Human-Robot Interaction
## 1 Introduction  
With the development of deep learning, robots play an increasingly important role in human daily life owing to their more user-friendly and intelligent properties [1]. However, there are still two critical challenges before the initiation of the next generation containing advanced agents. One challenge involves the ability of robots to understand natural language, which affects the efficiency of robots responding to human commands to achieve seamless interactions. Another challenge involves the environmental perception of robots, including both object detection and part recognition in the interaction process. Specifically, the intelligent agent receives a command about \"the cap of the bottle\" in a complex environment, which implies that it comprehends the location of the part that the intelligent agent can twist, such as when people grasp a bottle cap to open it. In this paper, the datasets, including images of fine granularity and short commands, are reconstructed, and their corresponding part-level segmentation masks and detection boxes are selected as the ground truths to evaluate the capability of intelligent agents to solve those challenges.  
The multi-model has made significant progress in language and visual knowledge. Nevertheless, most existing studies based on the contrastive language-image pre-training (CLIP) model [2; 3] mainly focus on object-level recognition, and part-level recognition is rarely reported. For object recognition, current foundational vision language models (VLMs) can localize objects based on their labels. However, more fine-grained recognition is required for the intelligent agent to execute subsequent manipulations, such as twisting the cap of a bottle. Hence, it is necessary to recognize the fine-grained parts for the development of the intelligent agent. On account of the scarcity of part datasets, especiallypart-level segmentation datasets, the focus of studies on part-level recognition is placed on open-world or open-vocabulary part-level segmentation [4; 5]. In this paper, VLMs are explored by reconstructing a dataset containing both language and fine-grained annotations. Additionally, VLMs are constructed to segment fine granularity from language. To this end, the datasets are reconstructed based on Pascal Part [6] and PartImageNet [7]. Pascal Part includes 10,103 images, 6 object-level categories, and 40 part-level categories; PartImageNet contains 24,000 images, 11 super-categories, and 93 part-level categories. The evaluation of the VLMs on the reconstructed dataset reveals that the text embedding, to a certain extent, improves the accurate detection and segmentation of different parts.  
In summary, an integrated framework is presented in this paper for fine-grained segmentation and detection by exploring vision language models in the interaction between robots and humans. Different from conventional methods, this method highlights the part-level understanding, which maximizes the chance of the robots controlling the objects. Additionally, the proposed method is related to the relationship between parts and their objects via the connection with text features. Part-level segmentation has been investigated in previous studies, and it involves inputting an image and outputting the names of parts that do not contain objects. However, we cannot know what objects these parts belong to through the output, as it neglects the relation between the parts and objects. Finally, some experiments are performed based on two datasets to verify the performance of the proposed method.