# Multiscale Feature Fusion with Self-Attention for Efficient 6D Pose Estimation
## 3 Materials and Methods
### Overall Structure of the Network
#### 3.2.5 Feature Fusion and Pose Estimation  
During feature fusion, the pixel-level image features obtained from the two feature extraction modules and the point-level features containing local information need to be fully integrated. By leveraging the natural complementarity between the two, the features are concatenated to construct global features. These features are then input into the pose estimation network to estimate the object's pose. Feature fusion: Considering that the point cloud features are derived from the depth map and that there is a direct correspondence between the depth map and the RGB image, feature fusion is performed by concatenating features at the pixel level based on this positional correspondence. This forms point cloud features that not only include geometric and color information but also contain neighborhood information. These features can then be directly used to extract global variables, which are encoded through two multilayer shared perceptrons. Afterward, an average pooling layer is used to obtain representative global variables. The pixel-level fused features and global features are concatenated and input into the pose estimation network as the final features. The process is expressed as follows:  
\[\left\{ \begin{aligned} F&=x+emb\\ F^{fin}&=F+ap(h^{\prime}(h(F)))\end{aligned} \right. \tag{5}\]  
where \(x\) represents the point cloud features, \(emb\) represents the image features, \(F\) represents the pixel-level fused features containing color and local information, \(ap()\) is the average pooling operation, \(h^{\prime}()\) and \(h()\) are nonlinear functions (shared perceptrons), and \(F^{fin}\) is the final feature. Note that the \(+\) operation in Equation (5) represents feature concatenation. To clearly illustrate the feature extraction pipeline from the input images to the final fused features, we have presented the entire fusion process in a diagram. As shown in Figure 7, the flow proceeds from left to right, showing how RGB and depth images are processed. A mask is first applied to extract the target object region (choose), which is then  
Figure 6: Inverse residual module. used to index both the RGB image and depth map--leveraging the one-to-one pixel-level correspondence between them--for precise feature alignment. RGB features are extracted using a CNN, producing a feature map of shape \(C_{1}\times H\times W\). Using choose, we obtain an image feature tensor of shape \(C_{1}\times N\). Simultaneously, the depth map is back-projected using built-in camera functions to generate 3D coordinates and corresponding geometric features of size \(C_{2}\times N\). These features are concatenated to form the fused feature \(F\in\mathbb{R}^{(C_{1}+C_{2})\times N}\), where \(N\) is the number of sampled points and \(C_{1}\) and \(C_{2}\) are the channel dimensions of the corresponding features. To capture the global context, \(F\) is passed through shared MLPs and an average pooling operation is used to extract global features. These are then concatenated back to the point-wise fused features, resulting in the final feature \(F^{fin}\). This process is represented in Equation (5) and offers an intuitive view of pixel-aligned fusion between image and point cloud features. Pose estimation: When the features are input into the neural network for pose estimation, the network directly regresses the rotation, translation, and confidence score. The pose estimation network in this paper is similar to that of DenseFusion and consists of three similar small network layers that each consist of four 1D convolutional layers. The estimation accuracy is determined by calculating the average deviation distance between the sampled points of the predicted and ground truth point clouds. For asymmetric objects, the loss is defined as follows:  
\[L_{i}^{p}=\frac{1}{G}\sum_{j}^{G}\parallel(Rx_{j}+t)-(\overline{R}_{i}x_{j}+ \overline{t}_{i})\parallel \tag{6}\]  
This metric is only suitable for asymmetric objects with a unique correct pose.