## 3 Materials and Methods
### Projection to Recover Symmetry  
Since the camera's optical axis direction \(n_{C}\) is often not directly along the object in real situations, we need to transform the pixels inside the object bounding box by an approximate projective transformation. Thus, the problem can be converted to an ideal symmetric description problem, which has been discussed in Section 3.1. Specifically, we assume that the object is placed on the ground, so that the positive direction of the object \(n_{O}\) must be parallel to the horizontal, which can then be expressed in terms of the angle \(\theta\) between the x-axis of the object's coordinate system and the x-axis of a fixed world coordinate system. Next, we try to simulate the observation of the camera rotating around its center, vertically to the z-axis of the world coordinate system in \([-45,45]\) degree, and  
Figure 2: On an ideal 2D symmetric image, the hue, saturation, gray-scale value and the distance to the nearest edge of two symmetric pixels should be equal.  
evaluate the symmetry of the object in the 2D image plane at that simulated viewpoint to determine the direction of the symmetry plane. When the recovered image best satisfies the symmetry in two dimensions, the simulated \(n_{C}\) should, theoretically, be most aligned with \(n_{O}\) at this point.  
There are two reasons why a pure rotation of the camera around its center is chosen. One is that the three-dimensional position of the object in the world is unknown at the time of obtaining the camera image, so only the rotational distortion can be corrected. The other is that the transformation between pixels on the image before and after the conjugate rotations \(R_{c}^{n}\) has a simple homographic form, i.e.,  
\[v^{n}=(KR_{c}^{n}K^{-1})v^{c} \tag{4}\]  
where \(K\) denotes the camera intrinsics matrix, which describes the relationship between object points and image points.  
It is worth noting that the camera does not rotate around its own z-axis, but around the z-axis of the world coordinate system. Let \(Rot(Z)\) be the rotation matrix corresponding to the rotation \(\theta\) of any coordinate system along its own z-axis, then the rotation transformation matrix of the camera is an adjoint transformationwith the form \(R_{c}^{n}=R_{w}^{c}\cdot RotZ(\theta)\cdot(R_{w}^{c})^{-1}\). Let  
\[\mathbb{P}(\theta)=KR_{w}^{c}\cdot RotZ(\theta)\cdot(R_{w}^{c})^{-1}K^{-1} \tag{5}\]  
then we have \(v^{n}=\mathbb{P}v^{c}\). If the camera is rotated to face exactly the front side of the object, the corrected image should satisfy the 2D direct symmetry, as shown in Figure 3.  
We show the source image projectively warped by homography \(\mathbb{P}(\theta)\) only to better illustrate the result of the original image after the projective transformation. In practice, since we need to traverse all \(\theta\), our comparison is actually performed on the original image (without actually computing the projectively warped result of the original image). The method to obtain the corresponding symmetry point on the original image is shown in the middle of Figure 3, i.e., for a point \(p^{c}\), first obtain the position of the image point \(p^{n}=\mathbb{P}p^{c}\) corresponding to the simulated rotation, then obtain its symmetry point \(p^{n}_{sym}=\mathbb{S}\cdot p^{n}\), and,  
Figure 3: A demonstration of our algorithm. In the room, both the sofa and the wall painting are symmetrical in the 3D world. However, they are not symmetrical in the 2D image because the camera does not directly look at them. We simulate the case when the camera is directly facing their front side by applying two projective transformations to each image, respectively, and then we evaluate the 2D symmetry.  
finally, back-project the point to the original image \(p_{sym}^{c}=\mathbb{P}^{-1}p_{sym}^{n}=(\mathbb{P}^{-1}\mathbb{S}\mathbb{P})p^{c}\). So, we are actually solving  
\[\underset{\theta\in(-45^{\circ},\mathcal{A}5^{\circ})}{\arg\min}\sum_{i}||Sym(p _{i})-Sym((\mathbb{P}^{-1}\mathbb{S}\mathbb{P})p_{i})|| \tag{6}\]