# RSCS6D: Keypoint Extraction-Based 6D Pose Estimation
## 3 Proposed Method
### Obtaining Target Point Cloud  
In practical scenarios involving viewpoint changes, varying lighting conditions, and background interference, the accuracy of point cloud extraction using only 2D bounding boxes from object detection algorithms is significantly reduced. To address this issue, pixel-level object classification through semantic segmentation algorithms is essential. In this work, we employ the PIDNet semantic segmentation algorithm, which not only effectively classifies 2D pixels but also performs pixel-level segmentation of 3D point clouds. This approach ensures robust and accurate point cloud extraction even under challenging conditions.  
Inspired by the proportional-integral-derivative (PID) controller in dynamic systems, the PIDNet algorithm employs a tri-branch network design, consisting of proportional (P), integral (I), and derivative (D) branches. These branches, respectively, capture detailed information, contextual information, and boundary information from RGB images.  
The integral branch gathers rich contextual information, which is fully integrated with boundary information. Additionally, the Pag module selectively combines this information with detailed information, ensuring that significant semantic features are not overshadowed. Boundary attention guides the Bag module to organically integrate these three types of information, forming fused features. Finally, multi-scale feature extraction from these fused features, by combining feature maps of different sizes, enhances the capture of spatial semantic information across scales, thereby improving the effectiveness of semantic segmentation.  
The PIDNet-S version was chosen for its impressive performance on the Cityscapes dataset, achieving a speed of 93.2 FPS and an accuracy of 78.6% mIOU. This balance of speed and accuracy makes it suitable for practical applications. To further enhance the accuracy of 2D segmentation and optimize the final 6D pose estimation results, several methods can be employed. These include utilizing keypoint cloud to improve the accuracy of 3D point clouds and pruning the network structure to eliminate redundant computations in feature extraction. These approaches aim to enhance the overall efficiency and effectiveness of the segmentation and pose estimation process.  
The mask image obtained from semantic segmentation provides 2D point-wise labels, indicating the location information of the target object. By combining this mask with depth information from the same pixels, we can derive the 3D point cloud coordinates of the target object. This integration of 2D labels and depth data allows for the conversion of 2 Dimage information into 3D spatial coordinates, effectively reconstructing the target object's shape and position in 3D space.  
\[\left\{\begin{array}{l}depth=depth\_img[u,\,v]\\ x=(u-cx)\,*\,z\,/\,fx\\ y=(v-cy)\,*\,z\,/\,fy\\ z=depth/factor\end{array}\right., \tag{1}\]  
The conversion from image coordinates to camera coordinates can be expressed in matrix form as follows:  
\[\begin{bmatrix}x\\ y\\ z\end{bmatrix}=D\begin{bmatrix}1/f&0&0\\ 0&1/f&0\\ 0&0&1\end{bmatrix}\begin{bmatrix}u\\ v\\ 1\end{bmatrix}, \tag{2}\]  
Here, \((x,y,z)\) represents the coordinates of a point in the camera coordinate system, \((u,v)\) are the coordinates in the image plane, \(D\) is the depth information at pixel \((u,v)\), and \(f\) is the focal length of the camera. This matrix formulation allows for the transformation of 2D image points into 3D camera coordinates using depth information.