# SMILE: Segmentation-Based Centroid Matching for Image Rectification via Aligning Epipolar Lines
## 3 Methodology
### Selecting Object Pairs  
To calculate epipolar geometry, camera calibration is essential. Typically, this calibration process involves using reference points such as checkerboards or patterned objects to obtain the necessary parameters. In this paper, we propose using objects that can be easily identified within images as reference points instead of relying on specific markers like checkerboards. After selecting the object pairs from each image, we mathematically calculate the centroid coordinates of the segmentation masks. The object selection method in this paper is as follows:  
* The class order of the selected object pair (A, B) in Image 1 must match the class order of the selected object pair (A\({}^{\prime}\), B\({}^{\prime}\)) in Image 2. (The classes should match exactly: A = A\({}^{\prime}\), B = B\({}^{\prime}\).) * The selected object pairs in both images must have the same class and be located in roughly similar positions. * Among the object pairs that meet the above conditions, the pair that requires the smallest rotation angle to align the centroid coordinates is selected. When selecting arbitrary object pairs, the selected objects in Image 1 and Image 2 may differ. This would contradict our objective of achieving a result similar to image rectification by rotating the images based on identical reference points. Therefore, the object pairs selected in Image 1 and Image 2 must be chosen with strict criteria to ensure they are the same. To achieve this, class matching is applied first. Given that there could be many objects in each image, simply selecting object pairs without strict criteria could lead to mismatches in the class of the selected pairs between Image 1 and Image 2. This would indicate that different object pairs were selected. However, even when the object classes match, it does not necessarily mean the same objects were selected as pairs in both images. Without specific reference points, relying solely on object segmentation to set the reference points can be problematic, especially if the input images are everyday images with multiple instances of the same class. Therefore, the selected objects must not only share the same class but also be in similar positions in both images. Finally, among the object pairs that meet these criteria, the pair with the smallest rotation angle difference, based on the differences in their x- and y-coordinates, is selected. In this paper, the centroids of objects are calculated using the moments derived from the segmented masks of object pairs. The moment \(M_{pq}\) of a 2D discrete image is defined as follows:  
\[M_{pq}=\sum_{x}\sum_{y}x^{p}y^{q}l(x,y) \tag{1}\]  
Here, \(x\) and \(y\) represent the x-axis and y-axis in the image, respectively, and \(p\) and \(q\) represent the order of moments for the x-axis and y-axis, respectively. \(l(x,y)\) is the pixel value at coordinates \((x,y)\). These moments exist in various orders, which are redefined as zeroth- and first-order moments. The 0 th-order moment \(M_{00}\) represents the sum of all pixel values occupied by the object, corresponding to the area of the object, and is calculated as  
\[M_{00}=\sum_{x}\sum_{y}l(x,y) \tag{2}\]  
The 1 st-order moment \(M_{10}\) with respect to the x-axis represents the moment about the x-coordinate, which is the sum of the products of the x-coordinates and the pixel values of all pixels belonging to the object:  
\[M_{10}=\sum_{x}\sum_{y}x\cdot l(x,y) \tag{3}\]Similarly, the 1 st-order moment \(M_{01}\) with respect to the y-axis is expressed as  
\[M_{01}=\sum_{x}\sum_{y}y\cdot I(x,y) \tag{4}\]  
Using the moments calculated from Equations (1) to (4), the centroid coordinates are determined. The centroid here refers to the center of mass of the object as determined by the masked object from object segmentation. The centroid is calculated by dividing the 1 st-order moments by the 0 th-order moment for both the x- and y-axes.