# SMILE: Segmentation-Based Centroid Matching for Image Rectification via Aligning Epipolar Lines
## 3 Methodology
### Image Rotation and Adjustment  
In the proposed method, the rotation to align the y-values is performed based on the object pairs selected through object segmentation. The object pair used for this rotation is chosen such that the difference in the x-coordinates and y-coordinates between the centroids is minimized, and the pair exists in both images. The centroid coordinates for this object pair are derived from the masks obtained through object segmentation rather than from the bounding boxes typically provided by object detection. This approach yields more accurate centroid coordinates compared to those obtained from bounding boxes. Using these centroid coordinates, the image can be rotated. To determine the values used for image rotation and adjustment, the rotation angle \(\theta\) between two points \((x_{1},y_{1})\) and \((x_{2},y_{2})\) is calculated as follows:  
\[\theta=\tan^{-1}\bigg{(}\frac{y_{2}-y_{1}}{x_{2}-x_{1}}\bigg{)} \tag{7}\]  
Given that the image center is \((cx,cy)\), the rotation matrix \(R\) is computed as  
\[R= \begin{bmatrix}\cos\theta&-\sin\theta&(1-\cos\theta)\cdot cx+\sin\theta\cdot cy \\ \sin\theta&\cos\theta&(1-\cos\theta)\cdot cy-\sin\theta\cdot cx\end{bmatrix}  \tag{8}\]  
The coordinates \((x^{\prime},y^{\prime})\) of the rotated image are then calculated using  
\[ \begin{bmatrix}x^{\prime}\\ y^{\prime}\end{bmatrix} =R\cdot \begin{bmatrix}x-cx\\ y-cy\end{bmatrix} + \begin{bmatrix}cx\\ cy\end{bmatrix}  \tag{9}\]  
Both the rotation angle and rotation matrix are necessary because the rotation angle defines the direction and magnitude of rotation, while the rotation matrix is used to actually perform the rotation. Through this process, the y-coordinates of the centroids of object pairs within each image are aligned to an arbitrary y-coordinate. However, in this case, the y-coordinates are only matched within one image, so a translation must be performed to ensure that the object pairs in both images have the same y-coordinates. Let \((cx_{i},cy_{i})\) be the centroid coordinates of image \(i\) and \(cy_{\text{target}}\) be the target y-coordinate for both images. The translation vector \(T_{i}\) for image \(i\) is then calculated as  
\[T_{i}= \begin{bmatrix}0\\ cy_{\text{target}}-cy_{i}\end{bmatrix}  \tag{10}\]  
This vector is used to translate all coordinates in the image. After translation, the new coordinates \((x^{\prime\prime},y^{\prime\prime})\) are computed as  
\[\  
\[W_{\text{final}}=W_{i}+P_{\text{left}}+P_{\text{right}} \tag{17}\]  
\(H_{\text{final}}\) and \(W_{\text{final}}\) represent the final computed height and width after padding, which are used to match the sizes of the two images. The process of rotating and translating the image is shown in Algorithm 2. ```
Images \(I_{1}\), \(I_{2}\), and best matching object pairs Output: Rotated and padded images \(I_{1}^{\prime}\), \(I_{2}^{\prime}\) if\(\text{\it best\_pairs}\
eq None\)then for each image \(I_{i}\) and pair \((p,q)\) in \(\text{\it best\_pairs}\)do  Calculate rotation angle \(\theta_{i}\) and center  Construct rotation matrix \(R_{i}\)  Rotate image: \(I_{i}^{\prime}=\text{\it apply\_rotation}(I_{i},R_{i},\text{\it center})\)  Update centroids \(\text{C}_{i}^{p^{\prime}},\text{C}_{i}^{q^{\prime}}\) endfor  Calculate target y-coordinate \(y_{\text{target}}\) for each rotated image \(I_{i}^{\prime}\)do  Apply translation to align y-coordinates  Update centroids endfor  Calculate target dimensions \(H_{\text{target}}\), \(W_{\text{target}}\) for each translated image \(I_{i}^{\prime}\)do  Calculate and apply padding  Update centroids endfor endif return Rotated and padded images \(I_{1}^{\prime}\), \(I_{2}^{\prime}\)
```  
**Algorithm 2** Image rotation and alignment  
Figure 2 visually represents the process according to the method proposed in this paper. Figure 2a shows the images used as input; Figure 2b illustrates the results after recognizing the objects within the images, performing segmentation, and calculating their centroids.