# Relative Pose Estimation between Image Object and ShapeNet CAD Model for Automatic 4-DoF Annotation
## 3 Proposed Pose Estimation Pipeline  
### Object Detection in RGB Images  
The 3D pose estimation of an object proposed in this paper includes the rotation and translation transformation relationships between the coordinate system of the CAD models of ShapeNetCore and the coordinate system of the camera that captured RGB images. It also includes the projection transformation from the camera coordinate system to the image coordinate system. The rotation and translation transformations between the 3D coordinate systems have 6-DoF. In addition, in order for the CAD model that has been transformed into the camera coordinate system to match the size of the object in the RGB image when the CAD model is projected onto the camera image space, the size of the CAD model needs to be determined in each of the x, y, and z directions. Hence, additional 3-DoF pose information is required. Consequently, 9-DoF is required. However, this study aims to calculate the pose information between the object in an RGB image and the CAD model using only 4-DoF by minimizing the complexity of pose estimation. Here, 3-DoF is required for rotation transformation, and the translation of the camera and the focal length related to the size of the projected object are calculated in 1-DoF by considering them as one parameter.  
First, the rotation and translation transformations, which are external transformation parameters, are relationships between two 3D coordinate systems. Hence, the candidate CAD models need to be determined after determining the category of the object. Therefore, in the first step, the proposed framework detects objects in RGB images. Several studies have been conducted on non-learning- and learning-based object detection methods. Recently, learning-based methods have shown relatively superior performances, and hence, they are commonly used. In this study, DetectoRS [36] was used to detect objects and segment the regions of objects. DetectoRS is a detection and instance segmentation network that uses a feature pyramid network (FPN) [8] as its backbone. It utilizes the FPN repeatedly to extract the features of images more precisely. In this study, the weights pretrained with the Microsoft COCO 2017 dataset were used, and val2017 was used for testing. For the baseline model of the network, ResNet-50 was used as the feature extractor, and a hybrid task cascade was used as the detector. For the learning environment, the epoch was set to 12, and the NVIDIA Titan RTX graphics card was used as the graphics processing unit (GPU). The performance of the trained network was 49.1 for the box AP and 42.6 for the mask AP.  
The CAD candidate models were determined from the object category of ShapeNetCore that matches the class of the object detected by DetectoRS in the RGB input image. Ten candidate CAD models were selected for one category. Section 3.3 describes the candidate CAD models in detail. In addition, a 2D BB was determined with the segmentation information of the region of the detected object, and the image of the object in the BB region of the input RGB image was used for pose estimation.