# Relative Pose Estimation between Image Object and ShapeNet CAD Model for Automatic 4-DoF Annotation
## 3 Proposed Pose Estimation Pipeline
### Retrieval of a CAD Model and Similarity Measurement  
The CAD model whose shape best matches the object detected in the RGB input image was retrieved from ShapeNetCore. ShapeNetCore consists of 51,300 CAD models classified into 55 categories. Each category is further divided into several subcategories. For example, the category \"chair\" has 23 subcategories and consists of 6778 models. Therefore, the class of the object detected in the RGB image can be regarded as one type of category in ShapeNetCore. If the class of the object detected in the image is \"chair,\" the model with the best matching shape among 6778 chairs in the \"chair\" category needs to be retrieved  
Figure 5: An example of the graphical representation of a CAD model. (**a**) Mesh, (**b**) mesh vertices: 17,815, and (**c**) resampled points: 60,000.  
from ShapeNetCore. However, such a retrieval task is difficult, mainly because of the domain gap problem. The CAD models of ShapeNetCore are 3D mesh data generated with a graphic technology. The texture of the CAD models is also generated with the graphic technology. Hence, there is a significant difference between the CAD model and the actual appearance. Even if a 2D rendered image of a CAD model is generated for a comparison with the image features of the object detected in the input image, the 2D rendered image also has noticeable graphic texture characteristics. Hence, there is a significant difference between the 2D rendered image and the actual image. In retrieving CAD models, the number of candidate CAD models is another problem, along with the domain gap problem. As mentioned in the earlier example, the \"chair\" category, which is the most common object among indoor objects, has 6778 CAD models. In the case of the \"table\" category, there are 26 subcategories and 8436 models. Hence, retrieving a CAD model identical to the object in the actual image is a difficult task. Hence, previous studies using the ShapeNet dataset did not use all the CAD models but used only a small number of models. As another example, only 90 CAD models were used out of 759 CAD models in the IKEA dataset [17] in Pix3D [15]. As Pix3D uses nine categories, ten CAD models were used for each category on average. In this study, we also selected ten categories in ShapeNetCore. For each category, ten CAD models with commonly encountered shapes were selected. In other words, ten categories and one hundred CAD models were used in the experiment, as shown in Figure 6. As shown in the figure, CAD models with different shapes were selected within each category for easier classification. However, as evident in the example of the all categories, the shapes of the candidate CAD models were still similar even though we manually selected the candidate models. Hence, retrieving a model that best matches the object in the input image among the candidate models is a difficult task. As explained earlier, the category of the object detected in the RGB input image was determined, and the CAD model with the same category was retrieved from ShapeNetCore. In general, to learn a category called \"car\" with the current learning-based deep learning technology, a dataset is constructed with a large volume of car image data with various shapes, and this dataset is used to train the network. For example, even in training datasets, such as ImageNet [25], Microsoft COCO dataset [16], and YOLOv3, categories related to \"car\" are only classified into categories with a broad meaning, such as \"vehicle\" and \"truck.\" However, the CAD models required in this study must have almost the same shape as the object in real images. The technique developed by us is utilized in the XR field. Hence, when the 3D CAD model is projected onto the 2D image space with the pose information of the actual object, the projected model must align with the region of the object accurately. This indicates that the shape of the detected object must match the shape of the model of ShapeNetCore almost exactly.