# Open-Vocabulary Part-Level Detection and Segmentation for Human-Robot Interaction
## 3 Methods
### Image Encoder  
The operation of the image encoder is consistent with the backbone extraction of part-level features in the images. The FPN can be employed to provide multi-scale features, and the RPN can be utilized to generate the RoI. In the image encoder, the backbone and FPN are modified, except for the RPN, which do not undergo any improvement. Backbones can be broadly divided into convolution-based networks and transformer-based networks, such as Resnet and Swin Transformer. Compared with Resnet, Swin Transformer is employed to extract image features by using the self-attention mechanism based on the shifted window, enabling the model to provide a more compact feature representation by capturing global and local context information. Furthermore, the sizes of the pooling layer and convolutional kernels can affect the convergence and computational efficiency of the model. Therefore, the backbone in the image encoder adopts Swin Transformer to extract visual features. The feature map obtained by the backbone in the image encoder  
Figure 1: The architecture of the proposed method for part-level segmentation and detection is composed of a text encoder, an image encoder, a detection decoder, and a mask decoder. has low resolution; thus, it lacks information about multi-scale features, especially small parts. Therefore, an FPN should be developed for the image encoder to process features from the backbone to generate multi-scale features. There are many schemes for FPNs due to different design and improvement strategies, and the structures mainly include the original FPN, FPN++, PANet, BiFPN, and NAS-FPN. As shown in Figure 2, an FPN is split into two paths: the bottom-up path and the top-down path. The bottom-up path is composed of transformer structures, which are responsible for acquiring features from the image. The top-down path fuses the extracted features from high-level semantic features with low-level detailed information. Swin Transformer can enhance the ability of robots to extract fine-grained information, thus improving computational efficiency. Therefore, Swin Transformer is introduced into the backbone to facilitate fine-grained detection and segmentation in robots performing subtasks. The Swin Transformer architecture was proposed in [35], as shown in Figure 3A. The Swin Transformer block consists of two successive Swin Transformer blocks, as illustrated in Figure 3B. In Figure 2, Swin0, Swin1, Swin2, and Swin3 are features with different scales generated by the backbone. The operation of the Swin Transformer block is divided into five stages. Suppose that the shape of the input image is \((\text{N}\times 3\times\text{W}\times\text{H})\), where \(N\) represents the batch size, 3 represents the number of channels of RGB images, \(W\) represents the width of the input, and \(H\) represents the height of the input. In these stages, the space of the feature maps is compressed, and simultaneously, the receptive fields are expanded. Firstly, the output of stage 1, \((\frac{H}{4}\times\frac{W}{4}\times 48)\), is obtained via the initial image from robot acquisition divided into non-overlapping patches. Secondly, the output from the previous stage is projected via a linear embedding layer with dimension \((\text{C})\), and it is then processed by several Swin Transformer blocks to acquire Swin0 \((\frac{H}{4}\times\frac{W}{4}\times\text{C})\). Thirdly, Swin1 \((\frac{H}{4}\times\frac{W}{4}\times 2\text{C})\) is constructed similarly to the generation process of Swin0. However, it includes an additional patch merging layer for \(2\times\) down-sampling on the space of features and \(2\times\) on the dimension of the features to generate hierarchical representations. Fourthly, Swin2 \((\frac{H}{16}\times\frac{W}{16}\times 4\text{C})\) is developed by using a patch merging layer and Swin Transformer blocks, similarly to Swin1. Finally, Swin3 \((\frac{H}{32}\times\frac{W}{32}\times\text{C})\) is the final stage of Swin Transformer and its generation follows the same pattern as Swin1 and Swin2.