# Relative Pose Estimation between Image Object and ShapeNet CAD Model for Automatic 4-DoF Annotation  
Soon-Yong Park  
1 School of Electronics Engineering, Kyungpook National University, Daegu 41566, Republic of Korea 11 School of Electronics Engineering, Kyungpook National University,  
image is estimated, the estimated rotation and translation information is used to transform the 3D CAD model of the bench into the camera coordinate system and project it onto the image to align it perfectly with the real image. XR techniques can be applied to remove the bench from the image or to change the shape of the bench bench using this alignment result. The 3D pose of the bench needs to be precisely estimated to align the chair in the image with the CAD model, as shown in Figure 1. In this case, the CAD model is generally represented with an unknown scale. Thus, 3D pose between the CAD model and the real bench object must be represented in 9-DoF parameters. Various approaches have been proposed for estimating the 3D pose of an object using images. Most previous studies on 3D pose estimation employed two types of techniques. The first technique is to utilize the 3D depth information of the object in an image using a depth camera. The second technique is to perform 3D-to-3D matching between the CAD model and the 3D shape information of the object extracted with a stereo matching technique using two or more images [1; 2; 3]. 3D depth or shape of the real object is very useful information for 3D pose estimation. However, acquiring the 3D information of an object from the image is difficult if only a single RGB image is used. In this study, we address a 3D pose estimation using only a single RGB image. In this paper, we propose a new framework to estimate the relative 3D pose between an object in a single RGB image and its CAD correspondence in an Internet dataset. The 3D pose information is represented by only 4-DoF parameters between the object and the model. This also provides the 4-DoF annotation to the CAD model to align the projection of the CAD model to the object region in the RGB image. The proposed framework utilizes a method that retrieves from the CAD dataset a 3D mesh model with the same class as the object detected in the image. Figure 2 shows the proposed pose estimation framework, and the overall process is as follows. First, the object is detected from an RGB input image using a deep learning network to determine its class and bounding box (BB). Second, the 3-DoF rotational pose of the object--azimuth, elevation, and inplane rotation--is determined using only the BB region of the input image. This process uses a pretrained deep learning network for pose estimation. Third, candidate 3D CAD models with the same class are retrieved from the CAD database using the 3D rotational pose and class information of the object. Here, the estimated rotational pose is used to rotate the CAD model, and two-dimensional (2D) rendered images of the CAD models are generated through rendering. Fourth, the similarity between the rendered images of the candidate CAD models and the actual object image is measured. Among the candidate CAD models, the model that best matches the actual object is selected. Finally, the 3D mesh model of the selected CAD model is placed in a fixed distance on the \(z\)-axis from the camera and projected onto the actual image by changing the focal length. This process is repeated, and the 1-DoF focal length is determined by selecting the value that produces the best alignment with the object region of the actual image. The contributions of this study are summarized as follows:  
1. The 4-DoF pose estimation pipeline from a single RGB image is proposed. Figure 1: 3D pose annotation of a real object can be used to project and align the matching CAD model to the object region in an image. 2. The 4-DoF pose annotation database can be generated to align the CAD model of a real object in the image plane. 3. Three image similarity criteria are proposed to match deep features between rendered CAD and real object images.