OmniStitch: Depth-Aware Stitching Framework for Omnidirectional Vision with Multiple Cameras

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 08:24:41

Abstract: Omnidirectional vision systems provide a 360-degree panoramic view, enabling full environmental awareness in various fields, such as Advanced Driver Assistance Systems (ADAS) and Virtual Reality (VR). Existing omnidirectional stitching methods rely on a single specialized 360-degree camera. However, due to hardware limitations such as high mounting heights and blind spots, adapting these methods into vehicles of varying sizes and geometries is challenging. These challenges include limited generalizability due to the reliance on predefined stitching regions for fixed camera arrays, performance degradation from distance parallax leading to large depth differences, and the absence of suitable datasets with ground truth for multi-camera omnidirectional systems. To overcome these challenges, we propose a novel omnidirectional stitching framework and publicly available dataset tailored for varying distance scenarios with multiple cameras. The framework, referred to as OmniStitch, consists of a Stitching Region Maximisation (SRM) module for automatic adaptation to different vehicles with multiple cameras and a Depth-Aware Stitching (DAS) module to handle depth differences caused by distance parallax between cameras. In addition, we create and release an omnidirectional stitching dataset, called GV360, which provides ground truth images that maintain the perspective of the 360-degree FOV, specifically designed for vehicle-agnostic systems. Extensive evaluations of this dataset demonstrate that our framework outperforms state-of-the-art stitching models, especially in handling varying distance parallax. The proposed dataset and code are publicly available in URL.

Main paper: 
 introduction: 
 Omnidirectional vision systems are becoming increasingly important in a wide range of applications, from advanced driver-assistant systems (ADAS), Virtual Reality (VR) to Unmanned Robotics and Mars exploration [8, 43, 44, 46] , providing essential 360-degree environmental awareness. Traditional image stitching methods for omnidirectional vision [17, 30, 32, 51, 55] that rely on specialized 360°cameras (e.g., Samsung Gear 360 1 and Insta360 2 ) mainly focus on single camera settings with multiple wide-angle lenses and synthesize the images from difference views overcoming large angular parallax. However, the single-camera setting is not suitable for general vehicle applications such as autonomous driving [14, 16, 47, 48, 58] due to high installation requirements [21, 26] and blind spots [15, 45] , which limit their effectiveness across different vehicle sizes and designs. To provide all-around 360°coverage for vehicle-agnostic omnidirectional vision system (VA-OVS), image stitching techniques with multiple camera settings should be adopted as illustrated in Figure 1 .Addressing the technical challenges of omnidirectional image stitching in VA-OVS scenarios presents several technical difficulties. First, the existing image stitching methods are limited to inputs from 360°cameras [17, 24, 35, 51] or specific camera arrays designed for predefined [23, 56] , manually crafted stitching regions. When adapting to different camera settings, manually aligning the images, transforming the viewpoint, and selecting a well-matched stitching region require redundant human efforts. Second, these methods struggle with distance parallax between different camera inputs. Our observation indicates that increased distance parallax leads to diminished stitching performance, as illustrated in Figure 1 . This degradation is mainly due to the extreme depth differences across images, which prior methods overlook, resulting in errors such as ghosting and misalignment artifacts. Third, there are no data sets with ground truths for omnidirectional image stitching with dynamic distance parallax that can be applied to VA-OVS scenarios.To address these challenges, we propose a novel omnidirectional stitching framework, called OmniStitch for dynamic distance consideration with multiple cameras. OmniStitch has two main modules: Stitching Region Maximization (SRM) and Depth-Aware Stitching (DAS). The SRM module solves the stitching region decision difficulty by simplifying the image alignment and viewpoint transformation process by statically matching half of the image for each stitching input pair. By maximizing the candidate size of image stitching, the framework can learn dynamic distance parallax between cameras and eliminate the need for pre-processing and postprocessing. After the SRM process, the DAS module transforms image pairs by considering large depth differences between image pixels and extracted context features to generate a seamless 360-degree image. It enhances traditional flow estimation by incorporating additional depth information computed from 3D feature matching, and warps the images with learnable weights at the pixel level to consider large depth differences between the images.For the challenge of dataset scarcity in the field of omnidirectional vision, we release the General-purpose Vehicle's 360-degree (GV360) dataset, leveraging the CARLA [10] simulator to generate ground-truth images in a virtual environment. This dataset is designed with data captured from four-way multiple cameras, incorporating a wide range of dynamic distance parallax inputs and preserving a 360°FOV perspective. The use of virtual environments overcomes the limitations associated with real-world data collection, broadening the scope for future research in vehicle-agnostic omnidirectional vision systems (VA-OVS). Experimental results using the GV360 demonstrate that our framework outperforms the state-of-the-art stitching model by an average of 18.5% of PSNR and 18.0% of LPIPS for all tested distance parallax scenarios. Furthermore, our framework demonstrates superior performance on unseen datasets in qualitative results.The main contributions are summarized as follows:• We propose a novel end-to-end omnidirectional image stitching framework for dynamic distance parallax in multiple camera settings. It enhances generalization of stitching framework to multi-camera Vehicle-Agnostic Omnidirectional Vision System (VA-OVS) scenarios by effectively eliminating the need for a hand-crafted stitching region decision process. • Our framework outperforms the state-of-the-art stitching methods, regardless of variations in distance parallax between cameras. The SRM module simplifies the manual process of identifying stitching regions for multiple cameras, and the DAS module effectively handles the artifacts caused by distance parallax between cameras. • To tackle the dataset sparsity problem in omnidirectional stitching fields, we release a dataset for omnidirectional vision stitching, referred to as GV360. This dataset includes ground-truth images that capture 360-degree perspectives and simulate varying distance parallax scenarios across different vehicle types. 
related work 2.1 unidirectional stitching: 
 Most image stitching models are designed for unidirectional vision applications, which generate a panoramic image from the viewpoint of a reference image. Unidirectional stitching methods merge two images by warping one towards the other to seamlessly stitch boundary regions [4, 6, 11, 13, 18, 25, 27, 28, 31, 36, 40, 54, 59? , 60] . Various techniques were proposed to divide the image into subregions and estimate homography for each region, using ground and distance planes [13] , grid units [28, 59, 60] , triangles [27, 36] , and superpixels [25] . Other enhancements have included maintaining image structure through local or global similarity transformations [4, 6, 11, 18, 31] and leveraging deep learning-based stitching models [40] for improved geometric integrity. However, these methods still suffer from projective distortion and misalignment artifacts in the case of wide parallax, due to the homography transformation process, which distorts the target image to fit the viewpoint of a reference image. Although VSLA [23] proposes to use bi-directional flow estimation to synthesize an intermediate view for wide parallax between cameras, the need for specific camera arrays and extensive pre-processing reveals the limited adaptability to various vehicles in VA-OVS scenarios.Omnidirectional stitching combines images from all directions to create a 360-degree view, providing a perspective view of spherical maps such as the equirectangular projection (ERP) format. It is commonly used with 360-degree cameras [17, 51] or circular rig cameras [2, 35] , which can produce 360-degree images by synthesizing each viewpoint. Traditional stitching methods for omnidirectional vision rely on mathematical approaches, focusing on geometric calibration through feature points [7, 17, 32, 55] , calibration boxes [57] , and depth map integration for seamless multi-camera setting [2] , as well as advanced optical flow methods [35] . On the other hand, deeplearning approaches can significantly enhance the performance of omnidirectional stitching, such as attention mechanisms [12, 30] for matching SR to human visual perceptions and weakly-supervised learning [51] for real-world ground truth image training. However, despite their efforts for wide parallax, they only support fixed camera formations (e.g., dual 195°FOV lenses or quad 190°FOV lenses) with a single 360°camera, and do not consider dynamic distance parallax in scenarios requiring flexible camera positioning.Datasets are crucial for the development of image stitching methods, particularly those that require ground truth (GT) or deal with parallax. View-free [37] , Stitched MS-COCO [41] , and DIR-D [39] provide GT for unidirectional stitching but lack parallax consideration. In contrast, datasets for large parallax such as VSLA [23] , UDIS-D [38] and PDIS [22] suffer from restricted access or lack of GT, limiting their applicability. For omnidirectional stitching, CROSS [29] and WSSN [51] provide accessible GT but are confined to single camera systems, restricting the versatility of camera configurations. The GV360 dataset fills these gaps by supporting multiple camera settings for VA-OVS using virtual environments to ensure adaptability across different scenarios, thus broadening the scope for stitching algorithm development. 
depth-aware omnidirectional stitching framework 3.1 overview: 
 Our goal is to integrate images from multiple cameras into a single omnidirectional image, accommodating variations in distance parallax that arise due to the diverse sizes and geometries of vehicles. In this work, we identify two requirements for VA-OVS scenarios:1) the implementation of a multi-camera setting and 2) adaptability to dynamic distance between cameras depending on diverse vehicle types. This approach assumes a configuration with four surrounding cameras mounted on the vehicle's body. This is the minimum number of cameras in a typical scenario with multiple cameras [15, 47, 48, 58] , arranged at 90°angles between them, providing an omnidirectional configuration ideal for vehicles of various shapes and sizes. We consider a dynamic distance parallax scenario where the distance between two cameras can vary from 0.01m to 1.4m, which is suitable for various vehicles.The overall process of the omnidirectional stitching framework is illustrated in Figure 2 . Given the input {𝐼𝑀𝐺 𝑖 } 𝑁 𝑖=1 , where 𝑁 is the number of image inputs (i.e., 𝑁 = 4), the general framework first identifies the optimal stitching regions for each pair of adjacent images. These regions are denoted as {𝐼𝑀𝐺 (𝑖, 𝑟 ) , 𝐼𝑀𝐺 (𝑖+1, 𝑙 ) } 𝑁 𝑖=1 , where 𝑟 and 𝑙 indicate the identified right and left parts of the images, respectively. These paired images undergo bidirectional flow estimation and pixel-level warping during the stitching phase, resulting in the output {𝑂 𝑖 } 𝑁 𝑖=1 , where each output𝑂 𝑖 = 𝑆𝑡𝑖𝑡𝑐ℎ𝑖𝑛𝑔(𝐼𝑀𝐺 (𝑖, 𝑟 ) , 𝐼𝑀𝐺 (𝑖+1, 𝑙 ) )is a segment of the panoramic image created by the stitching process. We simplify the notation of stitching region pairs from {𝐼𝑀𝐺 (𝑖, 𝑟 ) , 𝐼𝑀𝐺 (𝑖+1, 𝑙 ) } to {𝐼 𝐿 , 𝐼 𝑅 }.To ensure automatic adaptation of stitching region pairs {𝐼 𝐿 , 𝐼 𝑅 } to input images with dynamic distance parallax caused by multiple camera settings, we suggest using the Stitching Region Maximization (SRM) module instead of the manual process before stitching. Furthermore, the Depth-Aware Stitching (DAS) module is proposed to address artifacts of synthesized outputs 𝑂 𝑖 caused by depth differences by utilizing additional depth information and an advanced warping network.In previous approaches, the SR decision for each input pair typically involves manually transforming the viewpoint and aligning the images or using automatic algorithms such as PTGui 3 . In this case, the region near the seamline of the aligned images is defined as the SR pairs. Only this region will be masked and stitched, followed by a post-processing step such as blending with other unmasked regions.Conversely, we implement half-cropping and full-region matching for each pair 𝐼 𝐿 , 𝐼 𝑅 , thereby maximizing the stitching region. This technique simplifies the SR decision process by eliminating the need for manual pre-processing (e.g., image alignment, viewpoint transformation, and stitching region decision) and post-processing (e.g., blending with unmasked regions), facilitating adaptation to various camera formations with dynamic distance.Although the SRM allows for adaptation to various vehicles with multiple camera settings, its performance may decrease when adjusting to situations with large distance parallax. This is due to the increased candidate size for flow estimation, which can confuse the pixel-wise warping process. To handle the increased number of candidates resulting from SRM, we use a cascaded refinement strategy. It is based on the up-sampled features, flow, and synthesis results, which are fed into the next level of an image pyramid structure inspired by the frame interpolation network [19, 49, 61] . This structure organizes the stitching process into multiple levels, with each level 𝑘 of the total 𝐾 levels refining the input by incorporating up-sampled output from the previous level. This iterative refinement process ensures a more accurate and robust output.While addressing the challenge of stitching region decisions to handle variations in distance parallax, it was observed that stitching performance decreases in scenarios with large distance parallax, as shown in Figure 1 . Taking inspiration from the task of Video 3 https://ptgui.com Frame Interpolation (VFI) [9] tasks, which manage large motions and depth differences -challenges similar to those in large distance parallax stitching -we develop the Depth-Aware Stitching (DAS) module. The DAS module incorporates a methodological enhancement inspired by video frame interpolation models [19, 42, 52] , which are tolerant of large motion. It modifies each step of the traditional image stitching process to address such challenges related to dynamic distance parallax. The module consists of three main steps: Refined Flow Estimation (Step 1), Learnable Forward Warping (Step 2), and Synthesis Network (Step 3). Each step integrates elements of the UPR-Net [19] structure adapted to the specific requirements of VA-OVS scenarios, with further details provided in Figure 3 .Step 1: Refined Flow Estimation. Traditionally, pixel-level flow features are extracted directly from the input pair's stitching region {𝐼 𝐿 , 𝐼 𝑅 } using a basic flow estimator F (•) composed of stacked convolution layers. At each pyramid level 𝑘, traditional methods utilize outputs from the fifth and sixth layers of a stacked convolution neural network (CNN) to define flow features F𝑘 and flow 𝐹 𝑘 respectively as:F𝑘 = 𝐶𝑁 𝑁 5 (𝐼 𝑘 ), 𝐹 𝑘 = 𝐶𝑁 𝑁 6 (𝐼 𝑘 )with bi-directional flow estimation.To enhance flow estimation in the context of large distance parallax, our framework inspired by frame interpolation methods [19, 20] , leveraging multi-scale feature extraction, up-sampled flow from previous pyramid level, and 3D depth costs. This approach mitigates flow estimation errors caused by large candidate sizes from stitching region maximization using up-sampled flow and brightness discrepancies caused by large parallax. Incorporating high-level contextual information via multi-scale features effectively reduces these impacts, providing additional spherical depth information and considering both explicit and implicit structural information.Specifically, the feature extractor consists of three multi-scale stages, with each stage comprising 3, 4, and 4 convolution layers, respectively, which process both left and right images to produce context features {𝐶 𝑘 (𝐿, 0) , 𝐶 𝑘 (𝑅, 0) }, {𝐶 𝑘 (𝐿, 1) , 𝐶 𝑘 (𝑅, 1) }, and {𝐶 𝑘 (𝐿, 2) , 𝐶 𝑘 (𝑅, 2) } at increasing channel depths of 24, 48, and 96. These features are hen used to compute the 3D depth cost, accurately match features between aligned images, and adjust for spherical depth differences.The refined flow encoder integrates these depth costs with upsampled flow, formulated as:𝐹 𝑘 = F (𝐹 𝑘 -1 ⊕ F𝑘-1 ⊕ 𝐶𝑜𝑠𝑡 (𝐶 𝑘 2 ) ⊕ 𝐶 𝑘 2 )with 3D depth cost 𝐶𝑜𝑠𝑡 (𝐶 𝑘 2 ) and warped feature 𝐶 𝑘 2 related to the current feature level.Step 2: Learnable Forward Warping. Given the input image pair 𝐼 𝑘 𝐿 , 𝐼 𝑘 𝑅 and optical flow 𝐹 𝑘 𝐿 , 𝐹 𝑘 𝑅 , the traditional flow-based warping methods typically use mathematical backward warping to produce the warped image pair:{ 𝐼 𝑘 𝐿 , 𝐼 𝑘 𝑅 } = W (𝐼 𝑘 𝐿 , 𝐼 𝑘 𝑅 , 𝐹 𝑘 𝐿 , 𝐹 𝑘 𝑅 )However, these methods result in issues such as object occlusion or duplication in scenarios with large depth differences. To address these challenges effectively, we employ a learnable forward warping technique inspired by the softmax splatting method [42] , which enhances the handling of depth differences and reduces artifacts in the warped images. Specifically, given the input image pair {𝐼 𝑘 𝐿 , 𝐼 𝑘 𝑅 } and refined flow {𝐹 𝑘 𝐿 , 𝐹 𝑘 𝑅 }, the warped left image is defined by the equation: 𝐼 𝑘 𝐿 = - → 𝑤 (𝐼 𝑘 𝐿 , 𝐹 𝑘 -1 𝐿 /2,to find intermediate pixel positions between the left and right images, while a depth metric 𝑍 aids in accurate pixel mapping. Adopting the softmax splatting approach, the depth metric 𝑍 is used to assign appropriate weights to pixels within 𝐼 𝑘 𝐿 for their projection onto corresponding target pixels in 𝐼 𝑘 𝑅 , expressed as:𝑍 = 𝜆 • 𝑣 (𝐶 𝑘 𝐿 , (∥ 𝐶 𝑘 𝐿 -← - 𝑤 (𝐶 𝑘 𝑅 , 𝐹 𝑅 /2) ∥ 1 )where the weighting procedure is derived from backward warping ← -𝑤 and a scaling map 𝜆, with 𝑣 representing a U-Net for training.The core of our learnable forward warping technique lies in its capacity to adaptively determine the source of pixels, especially in scenarios where multiple pixels may target the exact location of the image. This process is made differentiable, allowing for effective learning and optimization. This warping process is also used for feature warping 𝐶 𝑘 during the flow estimation process as outlined in Step 1.Step 3: Synthesis Process. Finally, recursively warped images 𝐼 𝑘 𝐿 and 𝐼 𝑘 𝑅 are combined using an encoder-decoder structure based on the U-Net framework [19, 20, 61] to generate an intermediate view between two images. Given synthesis network inputs, which include the current stitching sources 𝐼 𝑘 , 𝐼 𝑘 , 𝐹 𝑘 , and the recursive output 𝑂 𝑘 -1 from the prior pyramid level, the stitched image for each pyramid level 𝑘 is formulated as:𝑂 𝐾 𝑖 = 𝑀 𝑘 𝐿 ⊙ 𝐼 𝑘 𝐿 + 𝑀 𝑘 𝑅 ⊙ 𝐼 𝑘 𝑅 𝑀 𝑘 𝐿 + 𝑀 𝑘 𝑅 + Δ 𝑘where 𝑀 𝑘 𝐿 and 𝑀 𝑘 𝑅 are the masking maps generated by the synthesis network, located in the fourth and fifth channels, respectively, and Δ 𝑘 represents a residual map contained within the first three channels. The final omnidirectional image 𝑂 is obtained by smoothly concatenating the stitched segments {𝑂 𝑖 } 𝑁 𝑖=1 .OmniStitch framework aims to optimize the parameters 𝜃 across the feature encoder, flow encoder F , warping network W, and stitching network S, jointly minimizing the three distinct losses during the training phase. These include the Charbonnier [5] and VGG loss [50] for pixel-level accuracy and assessing perceptual quality, respectively. In addition, the Census loss L 𝑐𝑒𝑛 , known for its effectiveness in handling object occlusion [34] , is used to consider large depth differences between two images. The overall training loss is expressed as:L 𝑡𝑟𝑎𝑖𝑛 = L 𝐶ℎ𝑎𝑟 + 𝛼 L 𝑣𝑔𝑔 + 𝛽L 𝑐𝑒𝑛where 𝛼 and 𝛽 are calibrating each loss term's scale. 
gv360 datasets: 
 Our goal is to facilitate supervised learning for omnidirectional image stitching to produce precise 360°Equirectangular Projection (ERP) images, addressing the need for accurate ground-truth (GT) data and adaptability to multi-camera configurations in typical vehicles that closely mirror real-world conditions. As shown in Table 1 , although several datasets exist for wide parallax image stitching [7, 23] , there are still limitations in terms of public availability and provision of accurate GT images required for training supervised image stitching models. Certain omnidirectional stitching datasets [29, 51] provide GT images captured with specialized cameras, but these do not align with the requirements of the VA-OVS scenarios. To address these issues, we introduce a new method for generating datasets that can be used in multi-camera settings and are not specific to a particular vehicle. We present the General-Purpose Vehicle's 360-degree (GV360) dataset generated by this method. This GV360 dataset is created using the CARLA simulator, an open-source platform that can simulate various traffic and weather conditions and camera configurations. We render images using fisheye lenses with a wide 185-degree field of view. To satisfy vehicle-agnostic quadratic multi-camera settings [15, 47, 48, 58] , we attach a total of four cameras to the upper side (U), downside (D), left (L), and right (R) of each vehicle as illustrated in Figure 1 . The distance between cameras varied from 0.01m to 1.4m, adjusting to the various shapes and sizes of the vehicles. The four ground-truth images are aligned for different input pairings: front-right (RU), right-back (RD), back-left (LD), and left-front (LU). These GT images are derived by collecting the center pixels from camera inputs and continuously placing them along an ellipse. This approach minimizes distortion and closely replicates a real-world panoramic view for each side. We place 29 cameras on each of the four intermediate sides to ensure comprehensive coverage. We then un-warp the camera inputs, crop the pixels closest to the center, and align and blend the cropped pixels to reduce camera distortion, creating what we consider to be the ground truth image. The camera arrangement in this setup is inspired by the traditional rigged camera systems (e.g., Google Jump [1] ), where cameras are placed at 3°intervals from the center to produce a high-fidelity intermediate image. The dataset consists of 12,724 training samples and 1,284 test samples. The proposed dataset and codes are made publicly available via a provided URL. 
experiments 5.1 implementation details: 
 The weights in the training loss (3.4 ) are set to be 𝛼 = 1 and 𝛽 = 0.01. The training process spans 300,000 iterations with a batch size of 10, utilizing the AdamW optimizer [33] with a weight decay of 10 -4 . The learning rate is gradually reduced through cosine annealing from 10 -4 to 10 -5 . The implementation is based on Pytorch 2.1.0 with CUDA 12.1 on Ubuntu 18.04, and the training is performed on two GPUs (NVIDIA RTX 4090 and RTX A6000). To augment the training data, we randomly crop 480 × 480 patches from the original data and apply random rotations, vertical flips, and channel inversion. For the image pyramid architecture, we decided to use a pyramid level of 4 and skip the flow estimation in the last level of the pyramid, instead using the cascaded flow empirically.To validate the performance of models in omnidirectional image stitching for general vehicle scenarios, we examine two categories of state-of-the-art models. First, we focus on undirectional stitching methods, which predominantly handle pair-wise or linear arrays of image inputs.• APAP [59] is a widely adopted stitching method, recognized for an As-Projective-As-Possible approach. It blends nonprojective deviations into a globally projective framework.• LPC [18] emerges as a novel solution aimed to overcome the wide parallax issue. It leverages co-planar region matching alongside global colinear structures, ensuring the preservation of geometrical integrity on both local and global scales. • UDIS++ [40] is a parallax-tolerant, unsupervised deep-learning model for image stitching that handles large parallax challenges and improves generalization across different datasets and resolutions. • VSLA-like [23] is a representative stitching model that utilizes partial stitching regions. Since the original code is not open source, we built our own version based on the descriptions in the paper. We trained it with GV360 dataset and set the stitching regions to the same proportions as in the original model (40% of the input image).Second, we explore omnidirectional stitching methods designed to produce 360°panoramic images.• Samsung Gear 360 [17] is a commercial stitching solution using dual fisheye lenses. Due to the limited reproducibility of many stitching methods, it is a benchmark for generating omnidirectional images. • PTGui is a widely used commercial software for automatic stitching that employs feature points for image alignment, similar to the Autostitch [3] method. It supports various camera inputs, but its automatic stitching often failed in the GV360 test set, particularly in instances with large distance parallax.The effectiveness of our framework is evaluated by comparing it with popular image stitching methods, as outlined in 5.2. This comparison uses standard metrics such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [53] , and Learned Perceptual Image Patch Similarity (LPIPS) [62] . The comprehensive results are presented in Table 2 . Our experiments span various distance parallax scenarios -0.01 m, 0.8 m, and 1.4 m -to evaluate the adaptability of the stitching methods for general vehicle applications. Our framework, equipped with the SRM module, shows superior performance over the VSLA-like method, achieving an average 18.5% improvement of PSNR, 11.7% improvement of SSIM, and 18.0% decrease of LPIPS. It verifies the benefits of applying full region adaptation with depth-aware flow refinement and a learnable warping network in stitching multiple images, thereby enhancing model adaptability for general vehicles and improving performance in large parallax situations. Notably, our framework shows a 14.3% decrease in LPIPS over the VSLA method in the large distance parallax scenario. This significant performance gap highlights the importance of considering depth differences and reducing flow estimation errors through additional depth information and a fine-grained warping network configuration.For qualitative evaluation, we compare image stitching methods on GV360 and additional unseen datasets, focusing on the presence of visual artifacts (e.g., misalignment, ghosting, and duplication) and the ability to generate natural and seamless images from a human-centric perspective for omnidirectional vision. This analysis includes three different distance parallax scenarios, similar to our quantitative multi-camera VA-OVS evaluation. Commercial stitching methods such as Samsung Gear 360 and PT-Gui, although designed for 360°image stitching, struggle with large distance parallax in multi-camera settings, often resulting in duplicate and misalignment artifacts. Among these methods, VSLA-like is manually adjusted with hand-crafted image alignment and partial SR decisions to achieve more natural and seamless images. However, it is still prone to ghosting artifacts, highlighting its inability to handle depth differences. In contrast, our method effectively mitigates unintended visual artifacts and produces images that are both natural in appearance and seamlessly integrated.5.4.2 Unseen dataset. We extend our comparison to include scenarios involving unseen data, which comprises images from real-world multi-camera settings, as shown in Figure 5 . Despite the training of our framework on virtual samples from simulations, it has demonstrated robustness when applied to unseen datasets from real-world contexts. This result highlights our framework's generalizability to a various omnidirectional vision scenarios.In the ablation study presented in Table 3 , we investigate the effectiveness of different components within our proposed modules in improving the stitching quality. First, we compare the performance of the Stitching Region Maximization (SRM) module, which aims to maximize the stitching region against a traditional method focusing only on partial stitching regions. The result shows a significant improvement in performance metrics such as PSNR and SSIM, indicating numerical and structural differences in image quality. Despite these gains, extending flow estimation to the full region may cause subtle errors, potentially leading to image-wide blurring. It can result in perceptually uncomfortable outcomes, as quantified by the LPIPS metric, highlighting a trade-off between maximizing stitching area and maintaining image sharpness. The DAS module addresses the depth difference caused by distance parallax through three key components: the image pyramid (IP), 3D depth cost (Cost), and learnable forward warping (LFW). Notably, the absence of 3D depth costs leads to a deterioration in LPIPS scores, underscoring the pivotal role of 3D depth information in the pursuit of more natural images in omnidirectional vision applications. In addition, omitting learnable forward warping reduces PSNR and SSIM. It demonstrates the effectiveness of an integral feature of learnable warping in detecting the importance of specific object pixels in situations with substantial depth differences between images. This enables the accurate placement of objects in the foreground, significantly enhancing the image quality.  
conclusion: 
 This paper introduces a novel omnidirectional stitching framework, referred to as OmniStitch, which addresses the challenges of dynamic distance parallax in multi-camera Vehicle-Agnostic Omnidirectional Vision Systems (VA-OVS). The framework incorporates Stitching Region Maximization (SRM) and Depth-Aware Stitching (DAS) modules to enhance stitching quality and robustness across various vehicle configurations by automating stitching region decisions and refining flow estimation with depth information to minimize artifacts. To overcome the lack of suitable datasets for VA-OVS, we provide the GV360 dataset with ground truth images that preserve 360°perspective using virtual environments. Our framework is superior in handling distance parallax variations and provides more accurate and reliable omnidirectional images. Although it currently operates with a limited number of cameras and is not completely resistant to flow estimation errors such as blurring, it provides valuable opportunities for future research to enhance the scalability and efficiency of omnidirectional vision systems for broader applications in ADAS, VR, and other fields.  
 
 Appendix:  


Decision: The paper is written in a clear manner and provides an original contribution to the field, as indicated by almost all of the 4 reviewers.
The main contributions are the new panoramic stitching framework OmniStitch with its two modules Stitching Region Maximization (SRM)  and Depth-aware stitching (DAS), the synthetic dataset DV360 and a rather comprehensive evaluation of the approach.
Based on the reviews, the authors have provided a rebuttal that mostly addresses all points of criticism.
Nevertheless, the overall review score has slightly decreased, as one reviewer (R4) stated that his concerns were not all addressed (but also stated that some were actually addressed). In turn, one reviewer did not modify his / her rating but indicated that he / she would now lean toward acceptance as a key criticism was resolved.
3 out of 4 reviewers agree in their rating providing a "borderline accept", while 1 reviewer now recommends "weak reject" (initially "borderline reject"). From the 3 reviewers that agree, 2 are "very confident" in their vote, as is the reviewer now suggesting "weak reject".

Besides the aforementioned strong points, the following aspects were found positive:
The contribution is specifically relevant for the parallax-induced issues in case of long and dynamic distances between cameras as for long vehicles, as appreciated by some reviewers (R2, R3).
Here, also the dataset represents a specific contribution (all reviewers).

Further negative points mentioned are:
The details desired by some reviewers about runtime and complexity have mostly been addressed in rebuttal, as mentioned by the respective reviewers (R1, 2, 3).
The real-world validation requested by some reviewers (R2, 3, 4) has partially been addressed in rebuttal, but must be more comprehensively covered in case of acceptance of the paper, by including and extending the information from the rebuttal in the camera ready (CR) paper.

Although some of the points by the critical review R4 remain valid, I am not fully agreeing with conclusions such as that datasets are not needed because there already are datasets or that models are not needed because there are certain other models, in case that certain advantages such as increased training data or improved performance may result. If of course neither in application nor performance new datasets or models are meaningful contributions, the merit of the work may be limited. The three other reviewers see this differently.
The overall rating is 3.75. Hence, in case that no sufficient space is available, this paper may also be rejected. However, as the aforementioned detailed points may be best discussed with peers at a conference, in summary, I lean towards accepting the paper as a poster. Here, direct communication with others working on related topics may enable fruitful exchanges and further improvements of existing algorithms.

Reviews: review: paper_summary: paper_summary: The paper presents a new framework designed to create seamless 360-degree panoramas from multiple cameras, specifically for applications like Advanced Driver Assistance Systems and Virtual Reality. The OmniStitch framework includes two key components: the Stitching Region Maximization (SRM) Module for adapting stitching regions dynamically across different vehicle types, and the Depth-Aware Stitching (DAS) Module, which minimizes errors like ghosting by addressing depth disparities. The paper also introduces a new dataset tailored for omnidirectional systems, demonstrating that OmniStitch outperforms existing methods in generating high-quality panoramic images.
strength_weakness: strength_weakness: The evaluation of the OmniStitch framework is thorough, with comprehensive testing using the newly introduced General-purpose Vehicle’s 360-degree (GV360) dataset. This dataset is specifically designed to simulate real-world conditions for vehicle-agnostic systems. The paper provides comparative analyses showing that OmniStitch outperforms existing state-of-the-art methods in terms of image quality metrics such as PSNR, SSIM, and LPIPS. This extensive evaluation demonstrates the effectiveness of the proposed methods under various conditions; 
The paper highlights significant applications of the OmniStitch framework, particularly in areas requiring high-quality panoramic images such as ADAS and VR. By improving the adaptability and quality of image stitching, the framework has the potential to enhance the development of systems in these fields, which require robust and reliable visual information from multiple cameras.
limitations: limitations: Although the workings of the SRM and DAS modules are detailed, the paper lacks an in-depth analysis of the algorithm's complexity and runtime. Besides, I am wondering if you have consider the cyclic consistency inside an ERP image(the output of the Framework) which is a good property for stitching. For real-time applications, such as advanced driver-assistance systems, processing speed is critical. The real-time performance and resource consumption of the algorithm are not explicitly discussed, which could be a potential flaw. To meet the expected length for a conference paper, I suggest including more detailed explanations and additional analysis in your manuscript.
paper_summary: paper_summary: The paper focuses on ADAS and VR systems which utilize ODI camera setup. Usually in the ODI camera, multiple lenses are used to capture the image from different angles and then the images are stitched together to get final ERP image. However in case of larger vehicles, the camera might have blind spot and cannot cover most of the areas surrounding the vehicle. To overcome this, one can mount 4 different cameras at 4 corners of the vehicle and then use those 4 images to generate final ERP image, however existing algorithms will suffer due to larger parallax differences, and also dynamic nature of these camera setup which might change depending on the vehicle. To address this issues, the paper introduces a stitching framework which can work with larger depth parallax and can handle dynamic distances between the cameras involved in the capture process. The authors also opensource the dataset which can further be used to train or benchmark similar models.
strength_weakness: strength_weakness: **1.** The combination of the SRM and DAS modules is innovative, particularly in handling dynamic distance parallax between cameras. 

**2.** The introduction and release of GV360 dataset is a major contribution. This dataset provides a wide range of dynamic distance parallax inputs and maintains a 360-degree FOV perspective. The dataset is particularly valuable for future research and development in vehicle-agnostic omnidirectional vision systems. 

**3.** The adaptability to different vehicle sizes and geometries is a key strength. By eliminating the need for hand-crafted stitching regions and handling depth differences dynamically, the framework can be applied to a wide range of vehicles, enhancing its practical utility.
limitations: limitations: **1.** Since one of the target applications is to use this method on remote or autonomous vehicles where one might require real time stitching of views, the inference time of whole process is a significant measure. In the comparison tables, the inference time is missing. With its dual-module approach and incorporation of depth information, might introduce significant computational overhead. The paper does not provide a detailed analysis of the computational requirements or the potential impact on real-time performance, which is critical for applications like ADAS and autonomous driving. 

**2.** The proposed framework, While the GV360 dataset is comprehensive and valuable, it is generated using a simulator. The paper would benefit from more extensive real-world validation to demonstrate the framework's effectiveness in practical scenarios. Real-world data often introduces additional complexities which are not present in simulated environments. 

**3.** The framework's performance in extreme scenarios with very large depth differences or highly dynamic environments is not thoroughly explored. Additional experiments and analysis in these challenging conditions would strengthen the paper's claims about the framework's robustness. 

***Other Comments***

**1.** In abstract (in the paper), line 26-28, it reads _“in addition we create and release an omnidirectional dataset, called which provides ....”_. Here the name is missing ‘GV360’. 

**2.** In section 5.1 about implementation details, it will be better to mention about number of epochs. (Although iterations are mentioned, and based on batch size and number of training data one can calculate the number of epochs).
paper_summary: paper_summary: 1. This work proposes a new panoramic stitching framework OmniStitch, which includes two main modules:

      a. Stitching Region Maximization (SRM) module: Automatically adapts to multi-camera setups for different vehicles, simplifying the image alignment and perspective conversion process. 

      b. Depth-aware stitching (DAS) module: considers the large depth differences between different cameras to generate seamless panoramic images.

2. This work creates and releases a new panoramic stitching dataset, which provides accurate ground truth of 360-degree panoramic views and is used to evaluate the performance of panoramic stitching algorithms for vehicle-mounted applications.
strength_weakness: strength_weakness: 1. A new depth-sensing panoramic stitching framework OmniStitch is proposed, which can handle the distance parallax problem between different cameras. It has great practical significance.

2. A new panoramic stitching data set is designed, which provides accurate ground truth of the 360-degree panoramic view and can be used to evaluate the performance of the panoramic stitching algorithm.

3. Experimental results show that the OmniStitch framework is better than existing panoramic stitching algorithms in handling the challenges brought by distance parallax.

In general, this work proposes a new panoramic stitching framework for the needs of vehicle-mounted panoramic vision systems, and develops corresponding data sets, which has certain innovation and practical value.
limitations: limitations: 1. The data set proposed in this work can promote the development of the field, but currently we have not seen any open source information related to this data set and code. If it is open source, you can consider accepting it.

2. Although the entire network framework can systematically realize panoramic image stitching, the network innovation points are insufficient. It is recommended to design relevant network modules based on the special imaging characteristics of fisheye and panoramic images to further improve the accuracy of the method.

3. The article lacks analysis and evaluation of algorithm complexity and computational efficiency. Real-time performance is important for many applications.

4. There are few evaluations of the splicing results of real scenes, and we hope to provide more real data results.
paper_summary: paper_summary: This paper targets omnidirectional stitching in ADAS. To this end, a synthetic stitching dataset with GT and a novel stitching system is built. Compared with other methods, it performs better in the 4-camera ADAS case.
strength_weakness: strength_weakness: A new dataset with gt is proposed to implement high-quality stitching in 4-camera ADAS system. It covers a large diversity of scenes and camera baselines. Besides, a novel step-wise stitching pipeline is proposed.
limitations: limitations: 1.	The dataset is a synthetic dataset. Although the authors conduct generalization experiments on real-world setups, the results (several qualitative results) are not convincing enough.
2.	The proposed framework seems to stitch only two images at once. It’s a little strange to me why not stitching all images once like [a][b].
3.	Since there are many unsupervised (UDIS) or weakly supervised ([a]) stitching, is it really meaningful to propose such a supervised method with a synthetic dataset?
4.	There are extensive typos in this manuscript. Please double-check this paper

Overall, I appreciate the dataset of this work. It indeed contributes to this field as there’s no such dataset. But the meaning of this dataset needs further justification. Why we need such a synthetic dataset since the unsupervised/weakly-supervised methods have already performed very well in real-world scenes. For unsupervised methods, collecting unlabeled data to learn stitching patterns under specific baselines also seems feasible.

[a] Weakly-supervised stitching network for real-world panoramic image generation
[b] End-to-end image stitching network via multi-homography estimation