# Open-Vocabulary Part-Level Detection and Segmentation for Human-Robot Interaction
## 3 Methods
### Image Encoder
The output of Swin3 represents high-level semantic features extracted from the input image acquired by robots. Figure 2: The architecture of an FPN, whose inputs are different-scale features from Swin Transformer denoted as Swin0, Swin1, Swin2, and Swin3, and the outputs are P2, P3, P4, and P5. Following the aforementioned processing procedures, with an increase in neural network layers, the ability of the algorithm for feature expression is enhanced [36]. These processes contain multiple Swin Transformer blocks, which mainly consist of Window Multi-head Self-attention (W-MSA), Shift Window Multi-head Self-attention (SW-MSA), LayerNorm (LN) layers, and Multi-Layer Perceptron (MLP) layers. The two improved MSA layers are located between the two LN layers, and the MLP layer connects to the LN layer, which occurs after the MSA layer is improved. Different from MSA, W-MSA employed to compute the self-attention in non-overlapping local windows. Through W-MSA, the computational complexity of the model is effectively reduced compared with MSA. SW-MSA is modified based on W-MSA by introducing a shifted window partitioning approach to construct cross-window connections and maintain efficient computation of the non-overlapping windows, thus solving the problem of lacking connections across windows. The shifted window partitioning approach utilizes a cyclic shift to keep the number of windows the same as that of the regular window partitioning to maintain efficient computations. Suppose that each window includes \(M\times M\) patches, and a window based on an image of \(h\times w\) patches. For a fair comparison of computational complexity, MSA and W-MSA are used to compute patches of the same shape. Compared with W-MSA, SW-MSA does not increase the computation complexity, which only introduces a shift window operation. The computational complexity of MSA and W-MSA can be expressed as follows:  
\[\Omega(\text{MSA})=4 hwC^{2}+2(hw)^{2}C \tag{1}\]  
\[\Omega(\text{W}-\text{MSA})=4 hwC^{2}+2M^{2}hwC \tag{2}\]  
This highlights a significant difference in the computational requirements from these functions between MSA and W-MSA. MSA entails substantial computation, especially for high-resolution images, while W-MSA demands comparatively minimal computational resources. This observation underscores the efficiency and scalability advantages of window-based approaches, particularly when processing large-scale visual data. Moreover, Swin Transformer blocks ensure linear computational complexity relative to the image size, rather than square complexity, thereby mitigating the computational burden of training models on higher-resolution images. This facilitates training on larger image datasets with higher resolution without excessively increasing computational demands. In the FPN, the new feature maps P2, P3, P4, and P5 are obtained via the up-sampling and fusing of high-level feature maps to correspond with the feature maps in the bottom-up branch, so that the semantic information is better represented in the decoder. More specifically, P5 is obtained from the processing of Swin3 by the lateral connection and the smooth layer process. P4 is obtained by adding the pixel-wise operation between Swin3 after a later connection and up-sampling with Swin2 after a lateral connection, followed by a smooth layer. The generation process of P3 and P2 is the same as that of P4. The lateral connection usually involves a \(1\times 1\) convolution operation, and its function is to connect the feature maps of different levels in the backbone to achieve cross-level information transmission and fusion. The smooth layer consists of a \(3\times 3\) convolutional layer and  
Figure 3: (**A**) The structure of Swin Transformer in the image encoder and (**B**) the structure of the Swin Transformer block. an activation function, which can reduce high-frequency noise, enhance the semantic information of features, and adjust the scale and number of the channels of features.