# Relative Pose Estimation between Image Object and ShapeNet CAD Model for Automatic 4-DoF Annotation
## 3 Proposed Pose Estimation Pipeline
### 4-DoF Pose Estimation  
The pose of an object refers to the position and rotation information of the object in 3D space. In Euclidean space, which we generally use, the pose of an object can be expressed by 3D rotation and translation. The rotation can be expressed as a 3 \(\times\) 3 matrix, and the translation can be expressed as a 3 \(\times\) 1 vector. Here, both the rotation and translation matrices have 3-DoF; hence, the 3D pose information has 6-DoF. In addition, as 3D rotation and translation are transformation relationships between two coordinate systems, the coordinate systems must be defined. In this study, the coordinate system of the CAD models of ShapeNetCore is defined as the world coordinate system, and the coordinate system of the camera that captures RGB images is defined as the camera coordinate system, as shown in Figure 3. Rotation from the world coordinate system to the camera coordinate system can be expressed using three rotation angles, as shown in Figure 3. These rotation angles are denoted by azimuth (Azim), elevation (Elev), and inplane rotation (Inpr).  
Let \(\mathbf{P_{W}}\) be a point in the 3D CAD model space, and let \(\mathbf{P_{C}}\) be this point in reference to the camera coordinate system. Moreover, let \(\mathbf{p_{B}}\) be the projection point in the 2D camera image coordinate system of \(\mathbf{P_{C}}\). Here, \(\mathbf{p_{B}}\) is a 2D coordinate when the BB region of the object in the original RGB input image is considered a new 2D image space. Therefore, 2D translation \(\mathbf{t_{B}}\) exists between the image space of the BB region and the coordinate system of the original image space, as shown in Figure 4. Therefore, points \(\mathbf{P_{W}}\) and \(\mathbf{p_{B}}\) can be expressed using the following perspective projection relation:  
\[\mathbf{P_{C}}=[\mathbf{R}|\mathbf{T}]\mathbf{P_{W}}, \tag{1}\]  
\[\mathbf{sp_{B}}=\mathbf{K}\mathbf{P_{C}}=\begin{bmatrix}f&0&O_{x}\\ 0&f&O_{y}\\ 0&0&1\end{bmatrix}\mathbf{P_{C}}. \tag{2}\]  
Figure 3: Rotation pose representation between the ShapeNet and the RGB camera coordinates.  
of the rotated CAD model--in Equation (8) are all unknown scale values. Here, \(t_{Z}\) can be regarded as a constant. Hence, if \(t_{Z}\) = 2.0 m, Equation (8) can be simplified as follows:  
\[\mathbf{p_{B}}=\begin{bmatrix}p_{B_{x}}\\ p_{B_{x}}\\ 1\end{bmatrix}=\begin{bmatrix}f\frac{P_{W_{x}}^{\prime}}{P_{W_{x}}^{\prime}+2 \cdot 0}+O_{x}\\ f\frac{P_{W_{y}}^{\prime}}{P_{W_{y}}^{\prime}+2\cdot 0}+O_{y}\\ 1\end{bmatrix}. \tag{9}\]  
According to Equation (9), if \(t_{Z}\) is regarded as an arbitrary constant, the optimal f can be determined by changing the focal length f of the BB image camera and determining how much the image of the CAD model projected onto the BB space overlaps with the region of the actual object. In this study, the object in an RGB image is detected, and the region of the object is segmented from the image. Then, the RGB image of the BB region is used as the input to estimate the pose. Hence, the 3D pose information of the object can be obtained by calculating both the 3-DoF rotation and the focal length f that makes the projected region of the CAD model overlap with the object region the most. Thus, the complete pose information is determined in 4-DoF. Previous studies had to solve highly complex problems of 6-DoF, including the rotation and translation, or 9-DoF if including the scale of the CAD model. In comparison, the proposed technique has the advantage of minimizing the complexity of the pose estimation problem to 4-DoF.