# SMILE: Segmentation-Based Centroid Matching for Image Rectification via Aligning Epipolar Lines
## 1 Introduction  
Stereo image matching, which involves pairing left and right images, has traditionally been an important topic in computer vision [1]. Typically, the acquired stereo images need to undergo an image rectification process to be used meaningfully. This image rectification is performed based on precise parameters obtained through camera calibration [2, 3, 4]. Before delving into this process, it is essential to understand the concept of epipolar geometry [5, 6], which is also a crucial concept in computer vision, especially in stereo vision. The epipole refers to the point where the optical center of one camera is projected onto the image plane of the other camera. In stereo imaging, since the positions of the two cameras differ, the optical center of one camera appears as a point on the image plane of the other camera. Each camera has its corresponding epipole, and the epipole from one camera lies on a specific line in the other camera's image, which is called the epipolar line. The condition for efficient stereo matching is that the corresponding points between the two images must lie on the epipolar lines. By exploiting this condition, the stereo matching problem can be solved more efficiently. To perform stereo image matching, it is necessary to compute the epipolar geometry and determine how the epipolar lines appear. This computation requires obtaining precise camera parameters through camera calibration, which are then used to calculate the epipolar geometry. The epipolar lines in the two images are generally neither parallel nor similarly aligned. To make them parallel, image rectification is applied to align theepipolar lines horizontally. This adjustment ensures that the corresponding points in the images lie on the same row, simplifying the matching process and increasing its accuracy. Consequently, rectified stereo images enable the generation of disparity maps and depth maps based on the disparity between the left and right images. However, the aforementioned process depends on prior camera calibration, which requires specific calibration patterns and multiple images to ensure accuracy. Moreover, image rectification based on calibration parameters involves complex calculations and constraints, often requiring 3D rotations of the images. In this paper, we propose a method that does not require the calibration patterns used in image calibration. Instead, we utilize object detection [7, 8, 9, 10] to segment [11, 12] the object pairs in each image and calculate their centroids. By assuming the line connecting the centroids as an epipolar line, we perform a 2D rotation and padding on each image. This adjustment aligns the centroids horizontally and induces epipolar alignment. Subsequently, we extract a disparity map from the stereo images to demonstrate that the proposed method effectively rectifies images without the need for camera calibration or traditional image rectification. In summary, the key contributions of this paper are as follows:  
* Stereo image alignment without a checkerboard: we propose a novel stereo image alignment method that does not require calibration patterns such as checkerboards, enabling more flexible and practical image alignment. * Object-based epipolar alignment: without prior camera calibration, we utilize object detection and segmentation to identify object pairs and effectively align epipolar lines horizontally based on the centroids of the detected objects. * Performance evaluation in unstructured capture environments: The image dataset used in this study differs from typical stereo capture environments, as it consists of images taken manually, resulting in irregular Y-coordinate variations. The effectiveness of our approach is verified through various evaluation metrics, including Fundamental Matrix analysis, Y-Disparity, X-Disparity Entropy, SSIM, Feature Matching Accuracy, and Area-based Correlation.