# Relative Pose Estimation between Image Object and ShapeNet CAD Model for Automatic 4-DoF Annotation
## 3 Proposed Pose Estimation Pipeline
### Retrieval of a CAD Model and Similarity Measurement
Therefore, a model that best matches the object detected in the image must be retrieved from the ten candidate models selected by us for each category. This operation can be considered one of the 2D-3D model retrieval methods. As we determined the approximate 3D rotation information of the object detected in the image using PoseContrast, we intend to solve the 2D-3D retrieval problem as a 2D-2D matching problem. The rotation information (Azim, Elev, Inpr) of the object obtained using PoseContrast in Section 3.3 and Blender Python API [38] are used to save the shape of a candidate CAD model rotated into the camera coordinate system as a 2D image. If the PoseContrast result can determine the approximate rotation information of the object, the candidate model can be rotated to the viewpoint of the camera, and a 2D image of an appropriate size can be generated. The image generation result can be regarded as the image of a candidate model with the same rotation information as the actual object. Moreover, we determine the final model by measuring the similarity between the BB image that only includes the region of the object in the input image and the 2D image of the candidate model generated using Blender Python API. Measuring the similarity between the actual image and the rendered image of the CAD model is a difficult problem owing to the domain gap. The domain gap is the most difficult problem in current learning-based object detection or retrieval methods [39]. This study does not deal with the domain gap problem directly. Instead, the final CAD model is retrieved using the 2D-2D similarity measurement. In addition, the local features, rather than global features, are compared between the object and the model to minimize the domain gap problem in the similarity measurement process. As the category of the object has already been determined in the object detection phase, a method that can distinguish different local features among 10 candidate models with the same category is more appropriate than to compare the global features among the candidate models again. The local feature information is used to measure the similarity between the RGB image of the actual object (within the BB region) and the 2D rendered image of the candidate CAD model. In this study, the DELF [40] features are used to extract the local features of images. DELF uses a learning-based CNN with the descriptor of image features required for large-scale image retrieval. In a previous study [40], a two-phase process was performed for large-scale image retrieval. We modified the image retrieval pipeline, which is the second phase in DELF, and used it to measure the 2D-2D similarity. In the image retrieval pipeline of DELF, the nearest neighbor is determined among the feature descriptors between the  
Figure 6: Ten ShapeNetCore categories and candidate CAD models used in the experiment. From the top, _Bad_, _Motobike_, _Table_, _Oven_, _Bag_, _Clock_, _Bottle_, _Chair_, _Car_, and _Bench_ (The italic type means the test object’s name). query image and all the images in the database. Subsequently, the matching points with each database image are collected separately. Then, geometric verification is performed using RANSAC [41] to determine the best-matching database image. In this study, DELF was used to obtain and match all the features between the RGB query image and the rendered images of ten candidate models. As the 3D pose of the object was already estimated approximately from the query and rendered images--which are used to measure the similarity--the pose of the object viewed in the image is almost similar. Therefore, it is assumed that there is only small difference in rotation or scale between the query and rendered images. Consequently, the 2D affine transformation model was used in this study for the geometric verification of image feature matching. The final matched candidate model was determined using the affine transformation model in the order of the most inliers.