## 3 Proposed Methods
### Similarity Measurement Between RGB Image and CAD Rendering  
Using the modified PoseContrast network, the rotational pose of a real object in a single RGB image has been predicted. Using the predicted rotational pose, we can transform a CAD model from the reference to the camera coordinate system. However, to apply the proposed 6-DoF pose estimation technique to AR or XR applications, it is necessary to retrieve and transform the CAD model whose category and shape match exactly with the RGB object. However, because the PoseContrast network is a class-agnostic pose estimator, we need to find the exact subcategory of the RGB object before projecting the CAD model to the image space. Until the previous section, we have known that the upper category of an RGB object from the DetectRS network. However, there are various shapes of CAD models in the same upper category from the dataset. Examples of the _bench_ category in the ShapeNetCore are shown in Figure 7. On the left side of the figure is a _bench_ RGB image and the right side shows _bench_ CAD models. On the right side, 10 different shapes of the _bench_ category are shown, and in this study, we call each different shape a subcategory. This section describes how to determine the best matching subcategory from 10 candidate models. To find a matching subcategory, we employ a deep feature matching network, DELF (Large-Scale Image Retrieval with Attentive Deep Local Features) [19]. As shown in Figure 7, the rotational pose estimation result is used to transform candidate CAD models to the camera coordinate system and to render them into the 2D image plane. Therefore, using DELF, we propose to measure 2D image similarity between the RGB object and the rendered CAD images and to determine the best similar rendering as the final matching model. Details of similarity measurement are as follows. First, the background of the RGB object is removed using the DetectRS segmentation result. Second, we obtain 10 rendered images of subcategories. Third, between each pair of the RGB object and rendered image, we derive a 2D affine transformation matrix using DELF. From the DELF feature detection and RANSAC outlier removal, we obtain the affine transformation matrix. The motivation for obtaining a 2D affine matrix is that the ideal image space transformation between the RGB object and the rendered CAD model is only unknown scaling. The ideal rotational pose of the matching CAD model results in only scale difference with unknown \(z\)-axis translation and focal length. The 2D affine transformation matrix \(A\) in Equation (16) consists of six elements representing 2D rotation, scaling, and skewing transformations, as well as translation. In this equation, two translation elements are not needed because only rotational pose is applied to the CAD rendering. Thus, only the front \(2\times 2\) matrix in \(A\) is considered, and it is defined as matrix \(A_{sim}\), as in Equation (17). The four elements of \(A_{sim}\) consist of multiplications of scale (S), skew (\(h_{x}\),\(h_{x}\)) and 2D rotation (\(\theta\)). \[A= \begin{bmatrix}a_{1}&a_{2}&t_{x}\\ a_{3}&a_{4}&t_{y}\end{bmatrix}  \tag{16}\]  
Figure 7: Rendering of 10 CAD subcategories using the pose estimation result of a _bench_ object. \[A_{sim}= \begin{bmatrix}a_{1}&a_{2}\\ a_{3}&a_{4}\end{bmatrix} = \begin{bmatrix}S_{x}\cos\theta&Sh_{x}\sin\theta\\ -Sh_{y}\sin\theta&S_{y}\cos\theta\end{bmatrix}  \tag{17}\]  
The first image similarity measurement is based on the 2D affine transformation, called SM-AT. The size of the 2D rendering image is fixed at 600 \(\times\) 600 pixels, whereas the size of RGB object images varies significantly in the collected dataset. Therefore, even if the rotational poses of the RGB object and CAD rendering are identical, there is still an image scale difference. Thus, practically, the 2D rotation transformation between RGB and CAD rendering should not be an identity matrix but rather multiplied by two scale factors, as shown in Equation (18).