# Relative Pose Estimation between Image Object and ShapeNet CAD Model for Automatic 4-DoF Annotation
## 2 Literature Review
### Retrieval of CAD Models and Pose Estimation  
[PERSON] et al. [7] proposed a method most similar to the framework proposed in this paper for estimating the 3D pose of an object. Their study also used RGB images as the input and showed the process of estimating the pose of the CAD model with respect to the camera coordinate system and projecting it onto the image as the final result. In their study, the ResNet-50-FPN [8; 9] was used to detect objects, and Mask-RCNN [10] was used to segment the region of the object. Furthermore, to utilize the 3D distance information of an object, the depth information was obtained from a single input image using MonoDepth deep learning convolutional neural network (CNN) based on multiscale feature fusion [11]. The normal vector image of the surface of the object was recalculated using the depth and segmentation information. In addition, the 3D shape and normal vector information of the CAD model and the 2D color image and 3D normal vector image of the object were used as the input to a deep learning network, and the pose of the CAD model with the best match was estimated using the deep learning network. Moreover, the ScanNet25K [12] image data  
Figure 2: The proposed 4-DoF pose estimation pipeline.  
and CAD models provided by Scan2 CAD [13] were used in their experiment. The error measurement of the method proposed in their paper utilized the 3D annotation information of Scan2 CAD, and the accuracy of translation, rotation, and scale ratio was 27.4%.  
In the previously described paper by [PERSON] et al. [7], 3D CAD models of Scan2 CAD [13] were used. In Scan2 CAD, a deep learning network was proposed for detecting an object in an RGB-D input image and aligning a CAD model matching the detected object with the RGB-D image. In addition, 3D CAD models, 3D scan data obtained with a depth camera, and 3D pose annotation data of each model between the CAD model and scan data were provided to train the network. For the 3D scan data, 1506 data points imported from ScanNet [12] were used, and the annotation information of keypoints between 14,225 ShapeNet CAD models and scan data were provided. In their study, a 3D CNN-based deep learning network was proposed to align the 3D pose of the object detected in an RGB-D image in 9-DoF.  
In Mask2 CAD [14], introduced by [PERSON] et al., a deep learning network was proposed for detecting an object from an RGB input image and estimating the 3D pose of the object in 5-DoF. The Mask2 CAD network segments the region of the object and compares the deep learning features of the segmented object with the features of the CAD models in the embedding space to retrieve the most similar CAD model and estimate the pose of the object. Moreover, Pix3D [15] and Scan2 CAD datasets were used in their experiment. In Pix3D, 10,069 images and the annotations of the 3D pose information of objects in the images were used to measure accuracy. Moreover, the performance of the network was tested using 5436 RGB images of Scan2 CAD. The results of the experiment for Pix3D showed an intersection over union (IoU) of approximately 0.613, and the average precision (AP\({}^{\text{mesh}}\)) was 8.4 for AP50-AP95 [16] of Scan2 CAD.  
In Pix3D, presented by [PERSON] et al. [15], a deep learning network was proposed for detecting an object in a single RGB image and estimating the 3D pose of the object in 9-DoF. In addition, a dataset was constructed using RGB images and CAD models for training the network and the annotation information of the 3D pose of objects in the images, and the dataset was published. In particular, in their study, all the objects in the images consisted of only the products sold by IKEA, and the IKEA benchmark dataset was used for the 3D CAD models of the products [17]. The images of IKEA products were crawled from the Internet, and the authors of the paper constructed the dataset using only the images that almost perfectly matched the IKEA products. Hence, the results of the object pose estimation experiment showed an accuracy of approximately 70%.