## 3 System and Methodology
### Object 6D Pose Estimation  
We estimate the object 6D poses through the transformation matrix between the object coordinate system and the camera coordinate system, which comprises the rotation matrix \(R\) and translation vector \(t\). This concept is illustrated in Figure 7.  
To solve the rotation matrix \(R\) and translation vector \(t\) requires sufficient geometric information; we converted the region of the target in the depth map to a point cloud according to the following formula:  
\[z = d/s\] \[x = (x^{\prime}-c_{x})*z/f_{x}\] \[y = (y^{\prime}-c_{y})*z/f_{y}\]  
where \(d\) and \(s\) denote depth and scale factors, respectively. \(x^{\prime}\) and \(y^{\prime}\) represent the coordinates of pixels in the depth map, and \(c_{x}\), \(c_{y}\), \(f_{x}\), \(f_{y}\) represent the intrinsic parameters of the camera, and (\(x\), \(y\), \(z\)) are the corresponding point coordinates calculated. By converting each pixel in the depth map, as described above, the data of the point cloud can be obtained. The trimmed RGB image and its corresponding point cloud data are then fed together into the 6D Pose Estimation network.  
The overall framework of our network is shown in Figure 8. We use a feature extracting backbone based on ResNet18 to map pixels to the color feature embeddings. Our backbone has made some improvements to ResNet18; for details, the output of layer2, layer3, and layer4 and the output of PPM go through an RRB module to strengthen the features. The features from different scales are then fused, which is beneficial for pose estimation. Meanwhile, a geometric feature extraction backbone based on PointNet is also constructed to map 3D points to geometric feature embeddings. With the pixel-wise fused color and geometric feature embeddings, the convolution layers will regress the 6D pose parameters \(T_{i}(i=1,2,\ldots,N)\) and confidence scores \(c_{i}(i=1,2,\ldots,N)\) for each pixel. The pose with the maximum confidence score is regarded as prediction \(T_{pre}\). To improve the pose-estimation accuracy, we use the iterative refinement network proposed by denseFusion [41] to obtain the refined pose.  
Figure 7: Illustration of object 6D pose estimation. The pose transformation matrix between the object coordinate system and the camera coordinate system is composed of the rotation matrix \(R\) and the translation vector \(t\).  
We designed two loss functions in the training stage to deal with symmetric and asymmetric targets. For asymmetric ones, each pose corresponds to the unique shape or texture of the object; therefore, we define the loss as the average distance between the points sampled on the object model transformed by the ground truth pose and corresponding points transformed by the predicted pose. The loss function is defined as follows:  
\[L_{i}=\frac{1}{M}\sum_{j}\
orm(R_{gt}x_{j}+t_{gt})-(R_{pred}^{i}x_{j}+t_{gt}^{i})\
orm\]  
where \(x_{j}\) represents the homogeneous form of the \(j\)-th point of \(M\) randomly selected points from the 3D model of the object. \(p_{gt}=[R_{gt}|t_{gt}]\) is the ground truth pose, and \(R_{gt}\) and \(t_{gt}\) are the rotation matrix and translation vector, respectively. In addition, \(p_{pred}i=[R_{pred}|t_{pred}i]\) is the current predicted pose of the \(i\)-th pixel.  
For targets with a symmetric structure, one texture or shape can theoretically be associated with multiple or infinite poses. Therefore, to avoid ambiguous learning objectives, loss is defined as the average distance between the points transformed by the predicted pose and their closest points on the object model transformed by the ground truth pose. The loss function is defined as follows:  
\[L_{i}=\frac{1}{M}\sum_{j}\min_{0<k<M}\
orm(R_{gt}x_{j}+t_{gt})-(R_{pred}^{i}x_{ k}+t_{pred}^{i})\
orm\]