# Relative Pose Estimation between Image Object and ShapeNet CAD Model for Automatic 4-DoF Annotation
## 2 Literature Review
### RGB Image-Based Pose Estimation  
Studies on technologies for estimating the 3D pose of objects in RGB or RGB-D images captured with a camera have been ongoing for a long time [1; 2]. As 3D spatial information is required to estimate the pose of an object, performance is generally better when 3D RGB-D images are used than when only 2D RGB images are used. Therefore, further studies have been conducted using RGB-D images as the input--either a learning-based or non-learning-based method was used. However, a separate sensor, such as a depth camera, is required to obtain RGB-D images. Hence, applying this technique to all image-processing fields is difficult. Therefore, multiple learning-based studies have recently been conducted using only RGB images.  
[PERSON] et al. [28] proposed a deep learning network that extracts and matches features from an RGB image and a CAD model. If feature point matching between the 2D RGB image and the 3D CAD model is accurate, the 3D pose of the object can be obtained through the RANSAC and PnP algorithms. In addition, VGG was used as the backbone of the network, and a 2048-dimensional descriptor vector was generated for each grid of the image using the keypoint proposal network. Moreover, [PERSON] et al. introduced a more improved deep learning network in [29].  
[PERSON] et al. [30] also introduced a technique for extracting and matching the feature points of an RGB image and a CAD model using a method similar to [28]. The most difficult problem in matching features between an RGB image and a CAD model is the domain gap. RGB images have natural colors like photographs because they display information captured with a real camera. However, CAD models contain information generated in a virtual graphic environment, and they do not have sufficient texture information. Hence, the domain gap problem occurs. To solve this problem, in their study, the region of the object was segmented from the RGB image using Mask-RCNN, and the VGG [31] network was used to extract features from the RGB image and the rendered image of the CAD model. To mitigate the domain gap problem in the feature extraction process, the feature vectors were mapped to the joint embedding space to minimize the difference between the vectors of the matching features. In their study, the Pix3D dataset was used for training. Moreover, the proposed pose estimation improves the APmesh[32] to 37.8 on seen objects and to 17.1 on unseen objects.  
[PERSON] et al. [33] compared the similarity between 2D images and 3D CAD models using a Multi-view CNN [34]. Their study does not focus on a technique for estimating the pose of an object in an RGB image. Instead, it focuses on retrieving a CAD model that best matches the object in the RGB image among multiple CAD models. On the other hand, [PERSON] et al. [35] introduced a technique for retrieving a CAD model that best matches the object in the input RGB image and estimating the pose using the same multi-view CNN network. To solve the domain gap problem, they trained the network to improve the retrieval performance by projecting the feature vector of the RGB image and the rendered image of the CAD model onto a common vector space.