# A Novel Metric-Learning-Based Method for Multi-Instance Textureless Objects' 6D Pose Estimation
## 2 Methodology
### Overview  
In this section, we introduce the framework of the proposed multi-instance pose estimation method in detail. The framework consists of four modules, namely, the object detection module, mesh feature encoding module, image feature encoding module, and pose estimation module. The flowchart of the framework is shown in Figure 1.  
The input of the model is an RGB image taken by an industrial camera. The image is first fed into the object detection module to find out the bounding boxes for each object in the image. We chose to use EfficientDet [30] in this module due to its light weight and high performance among current commonly used object detection modules. The EfficientDet offers seven versions with different model sizes to fit the need for variant applications. The bi-FPN used in the EfficientDet can effectively extract useful features for different kinds of objects. Figure 1: Flowchart of our proposed two-stage pose estimation framework. After each bounding box of the objects in the image was correctly obtained, the objects were cropped out of the image through the bounding box. We expanded the bounding box by \(\delta_{w}\) and \(\delta_{h}\) in width and height, respectively, to ensure the object is inside the bounding boxes. The bilinear interpolation method was used to resize all the cropped images into a fixed size \((H_{crop},W_{crop})\). Then, the resized images were parallely fed into the image feature encoding module to achieve the deep representation for each pixel. The image feature encoding module utilized the U-Net structure as the backbone for feature extraction. The cropped images with the same size were fed into the U-Net to extract deep features. Two multiple-fully-connected layers were designed to predict the semantic segmentation and pixelwise deep representation, respectively. The function of the U-Net can be represented as \(F_{I}=\Lambda_{\theta_{super}}(I)\), where I denotes the input cropped image of an object. \(\theta_{unet}\) is the parameter of the U-Net model. \(F_{I}=(F_{I}^{seg},F_{I}^{feat})\in\mathbb{R}^{(C+1+D)\times H\times W}\) indicates the output tensor of the U-Net. The \(F_{I}^{seg}\in\mathbb{R}^{(C+1)\times H\times W}\) part of the tensor is responsible for the semantic segmentation for \(C\) classes objects while the \(F_{I}^{feat}\in\mathbb{R}^{D\times H\times W}\) part of the tensor represents the pixelwise \(D\) dimensional deep features of the object in the image. In the mesh feature encoding module, a 4-layer SplineCNN \(F_{M}=\Phi_{\theta_{spline}}(M)\in\mathbb{R}^{D\times L}\) is used to extract nodewise deep features from the 3D mesh model. \(M\) is the 3D mesh model of the target object. \(\theta_{spline}\) denotes the parameters of the SplineCNN. \(F_{M}\) represents the node features calculated through the SplineCNN, where \(L\) is the number of the nodes from the 3D mesh model. The affinity submodule explicitly provides an affine transformation between the pixelwise deep features and nodewise deep features through Equation (1). \[s_{i,j}^{k}=F_{M_{x_{i}}}^{k}A_{k}F_{I_{y_{j}}} \tag{1}\]  
where \(A_{k}\in\mathbb{R}^{D\times D}\) are the learnable parameters of the affinity submodule for the \(k\)-th object. \(F_{M}^{x_{i}}\in\mathbb{R}^{D}\) and \(F_{I}^{y_{j}}\in\mathbb{R}^{D}\) are the \(x_{i}\) pixel and \(y_{j}\) node in the image and 3D mesh model, respectively. This submodule provides the ability for the network to learning affine-invariance features that can match with each other through feature similarity \(s_{i,j}\in\mathbb{R}\). In the pose estimation module, the deep features encoded from the image feature encoding module and the mesh feature encoding module are multiplied through dot product to calculate the similarity of the correspondences. The features with the maximum similarities were chosen as the 2D-3D correspondences. As there was one correspondence from 3D model for each pixel in the RGB image, dense correspondences were directly obtained for the RANSAC-based PnP method to calculate relative pose from the camera to the object.