Reconstructing the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Inpainting Model

Authors: ziqi  xie
weidong  zhao
xianhui  liu
jian  zhao
ning  jia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:48:42

Abstract: Deep learning-based image stitching pipelines are typically divided into three cascading stages: registration, fusion, and rectangling. Each stage requires its own network training and is tightly coupled to the others, leading to error propagation and posing significant challenges to parameter tuning and system stability. This paper proposes the Simple and Robust Stitcher (SRStitcher), which 
revolutionizes the image stitching pipeline by simplifying the fusion and rectangling stages into a unified inpainting model, requiring no model training or fine-tuning. We reformulate the problem definitions of the fusion and rectangling stages and demonstrate that they can be effectively integrated into an inpainting task. Furthermore, we design the weighted masks to guide the reverse process in a pre-trained large-scale diffusion model, implementing this integrated inpainting task in a single inference. Through extensive experimentation, we verify the interpretability and generalization capabilities of this unified model, demonstrating that SRStitcher outperforms state-of-the-art methods in both performance and stability.

Main paper: 
 introduction: 
 Image stitching is a fundamental problem in computer vision, which aims to obtain a larger field of view by merging multiple overlapping images [5] . As illustrated in Figure 1 (a), the current deep learning-based image stitching pipeline is typically structured into three sequential stages: (1) Registration Stage. The first stage takes the original image pairs to estimate warping matrices, which are then used to align images. Current learning-based methods focus on designing the homography estimation networks to address the registration problem [22, 21, 7] . (2) Fusion Stage. The second stage merges the aligned images into a single fusion image. Present research in this domain is generally classified into reconstruction-based (recon-based) and seam-based methods. Recon-based methods [32, 36, 33] typically use the encoder-decoder networks to perform pixel-wise reconstruction of the fusion image. While seam-based methods [35, 11] focus on identifying the optimal seams to eliminate the fusion ghosting. (3) Rectangling Stage. The final stage transforms the irregularly shaped fusion image into a standard rectangular format. There are only a few deep learning-based studies for this stage [34, 56] , and they are all supervised methods with requirements for labeling data.Annoyingly, the cascaded structure of current image stitching pipelines poses significant challenges for training optimization and parameter tuning. Furthermore, errors from early stages tend to propagate to later stages, significantly degrading the performance of later processes. The representative Process ① is implemented by UDIS [33] , process ② by UDIS++ [35] , process ③ by DeepRectangling (DR) [34] , process ④ by Eq. 10 and Eq. 11. The corresponding partial images, I and IV, illustrate how SRStitcher effectively corrects the apparent misalignment of a pillar. Similarly, the partial images II and III demonstrate how SRStitcher repairs the blurry coarse rectangling areas. SRStitcher can be applied to both UDIS and UDIS++ aligned images and get similar stitched results.image stitching methods UDIS [33] and UDIS++ [35] both struggle to effectively fuse images with registration errors, as shown in Figure 1 ① and ② . In the rectangling stage (Figure 1 ③), the prominent rectangling method DeepRectangling (DR) [34] also fails to adequately fill gaps, leaving visible black spaces at image boundaries.As shown in Figure 1 (a), the errors originating in the registration stage persist through to the final stitched image, and the existing methods lack effective mechanisms to address these errors (detailed in Appendix A.6). The recon-based [32, 36, 33] methods are proven to inevitably introduce artifacts in the stitched image in the presence of registration errors. Furthermore, the seam-based [35, 11] methods rely on the assumption that there is a perfect seam between the aligned images. When this assumption is not met, the UDIS++ [35] forcibly distorts the images to create a perfect seam, resulting in distortion in the stitched image. Therefore, current fusion methods are unable to effectively handle the registration errors shown in Figure 1 .To address the error propagation problem, we identify image fusion as the key point for improvement.We reconsider the problem definition of the fusion challenge and hypothesize that By determining the appropriate modification region and introducing an inpainting model with strong generalization ability, the abnormal image content caused by registration error can be effectively corrected. We propose to reformulate the fusion problem by overlaying the less distorted aligned image over the more distorted one, and inpainting the seam area between the images to correct the inappropriate image content.Building on reframing the fusion problem, we also revisit the rectangling challenge. The core of the rectangling problem is to fill in the missing rectangling area, which is also essentially an image inpainting problem. Therefore, we question whether fusion and rectangling are truly distinct challenges or if they could be addressed as a unified inpainting task. We recognize that handling fusion and rectangling tasks simultaneously is not a simple matter of determining the inpainting area. More importantly, it requires precise control of the inpainting process. Specifically, the fusion task involves the preservation of original image semantics to the greatest extent possible, while the rectangle task requires more heavy inpainting to fill the missing regions. To effectively manage these varying demands, we introduce weighted masks to guide the reverse process in a pre-trained large-scale diffusion model. This method allows for the adjustment of inpainting intensity across different regions during the reverse process, enabling both tasks to be completed within a single inference.The main contributions of this paper are: (1) We propose SRStitcher, which reformulates the problem definitions of the fusion and rectangling stages to construct a more streamlined and robust image stitching pipeline. (2) SRStitcher is the first to introduce the concept of inpainting to address the image fusion problem. It incorporates prior knowledge from large-scale pre-trained models into the image stitching pipeline, enhancing the robustness of image fusion against registration errors. (3) Without additional fine-tuning or supervision, SRStitcher improves the generalization of the rectangling method in the zero-shot scenario, opening up new possibilities for unsupervised image rectangling research. (4) We conduct extensive experiments to verify the interpretability and generalization of the proposed unified model. The results show that SRStitcher significantly outperforms the state-of-the-art methods in both quantitative and qualitative evaluations.2 BackgroundRegistration parameterization. The goal of the registration stage is to obtain the aligned images based on a transformation matrix. Given inputs I l (x, y), I r (x, y) ∈ R H×W , where x and y represent the pixel coordinates, H, W are the height and width, respectively. And H donates a 3×3 homography matrix between I l (x, y) and I r (x, y), which maps the input images to an uniform plane. To clarify the process of image registration, take the example of the four vertex coordinates (x k , y k ), k ∈ {1, 2, 3, 4} of the input image. The new image stitching-domain R H * ×W * can be obtained by Eq. 1.EQUATIONEQUATIONwhere, (x w k , y w k ) = H×[x r k , y r k , 1] T . Then, the input images are mapped into this new image stitchingdomain by warping operation φ(•) to get the aligned images I wl (x, y), I wr (x, y) ∈ R H * ×W * , as shown in Eq. 2.I wl (x, y), I wr (x, y) = φ(I l (x, y), I), φ(I r (x, y), H),where, I is an identity matrix. The masks M wl (x, y), M wr (x, y) corresponding to the aligned images can be obtained in a similar way by Eq. 2, except that the inputs I l (x, y), I r (x, y) are replaced with two all-one matrixes. The specific design of φ(•) vary slightly among different stitching methods [33, 35] , but the aligned image generation of these methods all follows the architecture of Eq. 2.Diffusion model. The proposed work is based on the Diffusion Model [16] . Since our method does not include the forward process, we only briefly introduce the reverse process. Suppose the x 1 , ..., x T are latents of the same dimensionality as the x 0 ∼ q(x 0 ), where q(•) is the a Gaussian Markov chain forward process with T steps. And, x 0 = E(I 0 (x, y)), where E(•) is the image encoder and I 0 (x, y) is the input image. The joint distribution of t-th inversion step is defined as Eq. 3.EQUATIONwhere, µ θ (x t , t) and Σ θ (x t , t) are parameters of the Gaussian Markov chain in t-th inversion step.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the work are illustrated in section 5 Discussion and conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
unified inpainting model: 
 Fusion parameterization. Unlike previous methods, our method reconceptualizes the image fusion problem to enhance its robustness against registration errors. Precisely, as shown in Eq. 2, the distortion degree of I wl (x, y) is relatively low because it involves only minor warping based on I.This means that even in the presence of registration errors, I wl (x, y) does not introduce large-scale distortions. Therefore, we propose to construct a coarse fusion image I CF (x, y) via Eq. 4.EQUATIONwhere, & and ⊙ denote the bitwise AND operators and element-wise multiplication operator. The coarse fusion image has noticeable seams, as shown in Figure 1 (b). Also, with registration errors, incoherent image content appears around the seams. To solve this problem, we propose to focus on inpainting the image content around the seam, ensuring cohesion and coherence. Therefore, we use the seam mask M seam (x, y) to define the area in the fusion image that needs to be inpainted, as detailed in Eq. 5.EQUATIONwhere, Dilation(•) and Erosion(•) denote the dilation and erosion operations [38] , K s is the kernel sizes, ∨, ⊕ denote bitwise OR and XOR operators. Then, we inpaint I CF (x, y) based on seam mask M seam (x, y) and the inpainting function f θ (•) to obtain inpainted fusion image ÎCF (x, y), as detailed in Eq. 6.EQUATIONRectangling parameterization. Our method also defines the rectangling challenge as an inpainting problem based on the content mask M content (x, y). We use Eq. 7 to obtain the inpainted rectangling image ÎCR (x, y).EQUATIONwhere, M content (x, y) = M wl (x, y) ∨ M wr (x, y).Unified model. Integrating Eq. 6 and Eq. 7, we obtain a unified inpainting model for fusion and rectangling, as shown in Eq. 8.EQUATIONwhere, M inpaint (x, y) = M seam (x, y) ∨ M content (x, y). By combining equations Eq. 3 and Eq. 8, this inpainting problem can be solved by a diffusion model, as detailed in Eq. 9.xtEQUATIONwhere, x 0 = E(I CF (x, y)), and x t-1 ∼ N (µ θ (x t , t), Σ θ (x t , t)).After defining the unified inpainting model for the fusion and rectangling tasks in the previous subsection, we discuss the method to control the inpainting strength in different regions during the reverse process, ensuring that both tasks can be accomplished in a single inference. Specifically, regions under the mask M seam (x, y) that contain the semantics of the original image are preserved as much as possible. In contrast, regions under M content (x, y) may require more powerful inpainting. We propose weighted masks to guide the reverse process to achieve this varying inpainting strength.Weighted masks. Weighted masks are constructed from the weighted initial mask M init (x, y) and inpainting mask M inpaint (x, y).Algorithm 1 Weighted Mask Guided Reverse Process (WMGRP) Weighted initial mask M init (x, y); Weighted inpainting mask M inpaint (x, y)3:prompt p ← "" ▷ Our method does not require prompt guidance 4:I CF R (x, y) ← Telea(I CF (x, y), M content (x, y), R) ▷ Coarse rectangling 5:x N ← E(I CF R (x, y)) ▷ Encode image 6:// Based on the inpainting model, so there is a little difference here with the Eq. 97:x 0 ← E(I CF R (x, y) ⊙ M init (x, y))8:M small inpaint (x, y), M small init (x, y) ← DownSample( M inpaint (x, y), M init (x, y))9:x ′ N ← AddNoise(x N , N )10: xN ← Concat(x ′ N , M small init (x, y), x 0 ) 11: for t = N -1, • • • , 0 do ▷ Reverse process 12:x ′ t ← DeNoise(x t+1 , p, t)13:M small t (x, y) ← 1 -( M small inpaint (x, y) ⪯ N -t N )▷ ⪯ means element-wise less-than ÎCF R (x, y) ← ImageDecoder(x 0 ) ▷ Decode image 17: Output: ÎCF R (x, y)The weighted initial mask M init (x, y) assigns different fidelity levels to each pixel of the fusion image, determining how much to modify each pixel based on its fidelity during the reverse process. The formula of M init (x, y) is given by Eq.10, which is composed of two parts. The left part determines the fidelity levels of pixels in M seam (x, y) region, and the right part determines the fidelity levels of pixels in M content (x, y) region.EQUATIONwhere, DT(•) is the distance transform operation [38] with kernel size K g , ϵ 1 and ϵ 2 are hyperparameters.The weighted inpainting mask M inpaint (x, y), as described in Eq.11, is inspired by the suffix principle [24] . During the reverse process, M inpaint (x, y) is mapped into multiple sub-masks to define the modified regions at each step t. The region corresponding to M content (x, y) contains no image content, so its size remains constant across all sub-masks, ensuring it is repainted throughout the entire process. Conversely, the region corresponding to M seam (x, y) contains a substantial quantity of original image information, and its size gradually increases with each step t, indicating that this region requires progressive modification. This gradual modification method facilitates a more seamless blending of the inpainting content with the original image content.EQUATIONGuided reverse process. We observe that when the missing region of the fusion image is large, the diffusion model very easily generates abnormal content, such as abnormal textures and words. To mitigate this issue, we introduce coarse rectangling. To be specific, we employ the Alexandru Telea Algorithm Telea(•) [46] to generate the coarse rectangling image: I CF R (x, y) = Telea(I CF (x, y), M content (x, y), R), where R is the radius of a circular neighborhood of each point inpainted. The Telea(•) algorithm introduces a weak prior without any specific semantic information to the image I CF (x, y). As shown in the partial image I of Figure 1 , the image of the coarse rectangling region is completely blurred. The experimental result shows that generating images with weak priors significantly reduces the likelihood of producing anomalous content compared to leaving blank areas entirely black. More details regarding the advantages of coarse rectangling can be found in the Appendix A.1.The specific steps of the reverse process are detailed in Algorithm 1. Although this algorithm is based on the Stable Diffusion Inpainting model [41, 3] , which differs slightly from the original Stable Diffusion model [2] , the underlying principles remain consistent. In addition, our method works without the need for prompt, effectively reducing dataset requirements.Appendix A.2 provide more detailed explanation and visualization of the weighted masks and WMGRP algorithm.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental setup: 
 Dataset. To validate the performance of our method, we conducted experiments on the large public dataset UDIS-D [33] . To the best of our knowledge, UDIS-D is the only publicly available large-scale dataset in this field. Appendix D.4 provides more results of our method on other traditional small datasets.Baselines. To our knowledge, no open-source solutions simultaneously address the fusion and rectangle stages of the image-stitching pipeline as comprehensively as our method. Table 1 (a) gives brief statistics of related works, and more related work details provided in Appendix B. Therefore, we have to establish the comparison baselines by combining several existing methods. For the registration and fusion stage, we employ pre-trained models from UDIS [33] and UDIS++ [35] .For the rectangling stage, we utilize pre-trained models from DeepRectangling (DR) [34] , Lama [44] , Stable-Diffusion-v1-5-inpainting (SD1.5) [41] , and Stable-Diffusion-v2-inpainting (SD2) [3] . (1) Stitched image quality. Since UDIS-D is an unsupervised dataset, we use the No-Reference Image Quality Assessment (NR-IQA) metrics to evaluate the image quality. Specifically, we use the HIQA [43] and CLIPIQA [49] . (2) Content consistency. We develop a new metric to evaluate the content consistency between the input images and the stitched image. Specifically, we introduce the CoCa(•) [50] model and Bert(•) [39] model to extract text from the images and generate text embeddings. The similarity between these embeddings is measured by the cosine similarity cosine(•). We design the Content Consistency Score (CCS) metric:EQUATIONCCS n measures the local consistency, which compares the stitched image I Stitched (x, y) and the fusion image I F usion (x, y). Both images are split into n equal parts for detailed comparison:CCS n = cosine(Υ(Split(I Stitched (x, y), n)), Υ(Split(I F usion (x, y), n))), where Υ(•) = Bert(CoCa(•)), and for this test, n = 4. In addition, CCS g assesses the overall content consistency between the I Stitched (x, y) and original input images: CCS g = cosine(Υ(I Stitched (x, y), Υ(I l (x, y), I r (x, y))). Please refer to Appendix C for further information on the metrics.All experiments are performed on a single NVIDIA 4090 GPU. In addition, all experiments of SRStitcher described in this paper are based on these pre-aligned images made by UDIS++ [35] . For hyper-parameters, the guidance scale and inference steps N are set to 7.5 and 50;The K s in Eq. 5 is set to ⌈W * /λ⌉ × δ, where λ = 200 and δ = 10; The K g in Eq. 10 and Eq. 11 are set to 3; The R in Telea(•) is set to 20. The ϵ 1 and ϵ 2 in Eq. 10 are set to 128 and 128. 2 , which illustrates the significant advantages of SRStitcher in terms of the stitched image quality and content consistency.We perform a quantitative evaluation of SRStitcher against other baseline methods, depicted in Figure 2 . The first row of Figure 2 presents a challenging registration scenario involving soft and deformable objects, such as wires, which may have deformed unpredictably between two images. Current registration methods cannot accurately align such objects. Instead of attempting to register or fuse these deformed wires, our method opts to inpaint incorrect wires, effectively overcoming registration errors. A more detailed discussion on this is available in Appendix A. We introduce a user study metric from UDIS [33] . This method allows for a more subjective but insightful visual quality assessment through direct user feedback. For the user study, we display four images simultaneously on a single screen: the two input images, our stitched result, and the stitched result from one of the baseline methods. Participants are asked to determine which result is superior, SRStitcher or Another (comparison baseline). If a clear preference is not apparent, participants can choose Both Good or Both Bad. The study involves 20 participants: 10 researchers (computer vision background) and 10 volunteers (non-computer major). This diverse group ensures a balanced perspective, combining expert technical evaluation with general user impressions. The results are shown in Figure 3 .Hyper-parameter. Figure 4 illustrates the ablation results in hyper-parameter of SRStitcher, demonstrating that these parameters are highly interpretable and easy to adjust. (1) λ: Controls the width of the region in M seam . A smaller λ increases the modification range and decreases the CCS. For stitched pictures with color differences, a lower λ value can better fuse the images. We set λ to 200, considering both image smoothness and CCS. (2) R: Controls the granularity of the coarse rectangling image. A higher R value provides a higher quality weak prior for inpainting, reducing the likelihood of generating abnormal content. Ideally, a larger R is preferable, but due to the limitations in GPU acceleration with the Telea(•), a very high R value can slow down the pipeline. Thus, we balance performance and speed by setting R to 20. (3) ϵ 1 : Controls the inpainting strength of the seam area. At ϵ 1 = 128, the shape of the pillars appears more reasonable compared to ϵ 1 = 64. However, increasing ϵ 1 to 192 significantly alters the image content, so we set it to 128. (4) ϵ 2 : Controls the inpainting strength of the rectangling area. When ϵ 2 is relatively low, the image structure remains largely intact, but increasing it to 192 leads to noticeable structural deficits. Therefore, we set ϵ 2 to 128. Please see Appendix D.2 for more comprehensive hyper-parameters studies. 𝜖 1 =64 𝜖 1 =128 𝜖 1 =192 𝜖 2 =64 𝜖 2 =128 𝜖 2 =192𝑅 = 20 𝜖 1 =64 𝜖 1 =128 𝜖 1 =192 𝜖 2 =64 𝜖 2 =128 𝜖 2 =192 
discussion and conclusion: 
 This paper introduces SRStitcher, which reconceptualizes the fusion and rectangling stages as a unified inpainting model. Through weighted masks, SRStitcher leverages the robust generalization capabilities of a pre-trained large-scale generation model to accomplish this complex inpainting task without additional fine-tuning or task-specific data annotations. Extensive experiments demonstrate that SRStitcher significantly outperforms existing state-of-the-art methods regarding the quality of the stitched images and its robustness to registration errors and abnormal content. Furthermore, the specific effects and adjustments of each hyper-parameter in SRStitcher are detailed in the ablation studies, illustrating its high interpretability and controllability.However, there are still some limitations and open issues in future research: (1)Visible seam. When input images exhibit significant color differences, visible seams may appear with the parameter settings described in the paper. Adjustments to ϵ 1 and λ can partially mitigate this issue, but such modifications can compromise the preservation of original image information. We speculate that a more flexible and appropriately designed hyper-parameter selection scheme could solve this problem.(2) Local blurring. We use coarse rectangling and M init (x, y) to control the content generation. However, this approach introduces a side effect where some challenging scenes appear locally blurred (See Appendix D.5). This issue presents a dilemma: accept local blurring or risk producing anomalous images. We temporarily choose to tolerate local blurring. Future improvements will include a refined coarse rectangling approach or fine-tuning the model. (3) Integrating registration. Is it possible to integrate the registration stage into the unified model? According to Diffusion Features (DIFT) [45] , it is possible. DIFT proves that the geometric correspondence between images can be effectively established by extracting feature maps from their intermediate layers at a specific timestep during the inverse process. Replacing the registration method used by SRStitcher with DIFT is straightforward. However, our ambitions extend beyond simple replacement. We believe there is potential for a more elegant and concise method to integrate concepts proposed by DIFT into our existing method. Stable Diffusion Inpainting model: the black is the area that needs to be retained, and the white is the area that needs to be inpainted. (1) Control of masked image. In our design, M init (x, y) is used to create the masked image, determining the extent to which information from the original image is retained. We aim to fully preserve the image content in M content (x, y). While, for areas outside M content (x, y)(that is the coarse rectangling area), we implement a distance transform to change the M content (x, y). Due to the coarse rectangling regions are blurry (but we must use the weak prior, as explained in A.1), it is undesirable to retain substantial information from these blurry regions. By employing a distance transform, we can gradually reduce the information from the coarse rectangling image starting from the edges of M content (x, y). This approach ensures that only the most relevant information from the edges of the optimal coarse rectangling is retained, avoiding the use of excessive coarse rectangling data. 
acknowledgment: 
 (2) Control of mask. In our design, the inpainting mask is dynamically adjusted throughout the reverse process based on M inpaint (x, y). Although M inpaint (x, y) serves as just one mask, its mask area is modified at each step t by calculating the threshold N -t N and remapping M inpaint (x, y) to M small t (x, y) accordingly (as detailed in Algorithm 1).Figure 7 provides the detailed mask productions. The final M init (x, y) and M inpaint (x, y) contain gradient areas, and we realize the gradual control of the reverse process based on the gradient areas. The M init (x, y) is used to store information about the fusion image. Intuitively, darker places hold more information about the original image. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss the potential positive societal impacts and negative societal impacts of the work in the Appendix E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our method does not involve model training and uses publicly available datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The source code for this paper has been open sourced and can be accessed through the link provided in the Abstract, with detailed instructions for use.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our user study included only 20 volunteer participants and they only rated the image quality. This test should not fall within the scope of crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing or research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our work does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As a supplement to Algorithm 1, we provide Figure 8 to visually illustrate the process of WMGRP using the masks generated from Figure 7 . The M inpaint (x, y) is employed to regulate the intensity of inpainting in distinct regions during the reverse process. Intuitively, the white part is not modified at all, and the black part is modifiable. It can be observed that the black area of inpainting mask is gradually increased during the reverse process, which serves as the guiding process for the gradual modification.Although the above content is mainly based on the observations and experimental results of the Stable Diffusion Inpainting model, our experiments have proved that it also applies to the general Stable Diffusion model [2] with only minor modifications. We provide the detailed implementation of SRStitcher variants in the Supplementary Material. A.4 Why is SRStitcher so effective at overcoming registration errorsUnlike the previous fusion and rectangling methods SRStitcher does not rigidly adhere to the registration results. Figure 9 provides a clear example of how SRStitcher addresses wire registration errors. In the coarse fusion image I CF R (x, y), the misregistration problem is still serious, which is reflected in the significantly misaligned wires.After the inpainting, these misaligned wires are effectively corrected, while the content in other masked areas remains largely unchanged. We attribute this remarkable correction capability to the strong generalization ability of large-scale generative models. This ability to correct incorrect image content underpins our motivation for employing the large-scale pre-trained model. Masked area that is allowed to be modified considering it as a first option. Initially, we try to address the inpainting problem using Generative Adversarial Network (GAN)-based methods, such as AOTGAN [53] , Lama [44] , and FCFGAN [20] .However, our experiments revealed several shortcomings. GAN-based models struggled with poor generalization ability, displayed unstable training outcomes, and demanded high-quality, well-labeled datasets. These factors complicated the pipeline and increased the workload, particularly in data labeling.During the research bottleneck period, we found the concept of Soft-Inpainting proposed by Differential Diffusion [24] , which involves blending parts of the image with the original by making slight modifications. This concept inspired us to adapt and extend it to our needs. We applied Soft-Inpainting near the seams and employed more intense Hard-Inpainting in the rectangling areas.With this idea, we successfully implemented SRStitcher proposed in this paper.Moreover, as mentioned in the previous section, the strong generalization ability of large-scale diffusion models is also vital to implementing our method, which is the key to the ease of implementation of SRStitcher without training or task-specific data annotations.Thus, adopting the diffusion model proved essential for addressing the challenges of this paper. It provided a more fitting solution than any other model available.The registration stage is not the primary focus of this paper. We employ a simplified homography estimation network from UDIS++ [35] to address the registration challenges. It is essential to clarify that this does not imply a devaluation of the registration stage. Registration has been the most extensively researched of the three stitching stages, with significant work devoted to improving homography accuracy. However, perfect homography matrices that precisely align images do not exist for scenes that are non-planar or involve cameras with different projection centers.There are two mainstream methods to overcome these inherent limitations: the multi-homography warp method [52] and the dense match method [47] . However, the multi-homography method faces challenges in parallelization and integration within deep learning frameworks [35] , while dense matching is generally slower and less robust.These limitations inform our decision to leverage the existing homography network and concentrate our efforts on enhancing the robustness of the subsequent stages. Our experimental results show that this decision has been beneficial: SRStitcher exhibits greater robustness to registration errors, thereby reducing the precision requirements of the registration stage.The image stitching pipeline can be divided into three stages, and the following subsections are described based on each stage.Early image registration works [57, 8, 31] are limited by the feature extraction method, which often falters under conditions of rotation, scaling, and illumination changes. To solve the scale changes problem, AutoStitch [6] marks a significant advancement by incorporating the Scale-invariant Feature Transform (SIFT) to extract scale-invariant features. However, this method is challenging to apply to situations with multiple depth layers. To address multi-depth layers condition, DHW [12] proposes a model that assumes the presence of two distinct planes within the image, applying different homography adjustments to each. However, the performance of this method can be severely impacted by the dynamics of camera movement. More recently, NIS [28] introduces the depth map integration to enhance registration accuracy. However, this method relies on accurately estimating depth maps, presenting its own implementation challenges. Yu et al. [51] develop a technique using the epipolar displacement field to improve registration in scenes with significant parallax.Feature-based methods have traditionally been the cornerstone of image registration techniques. However, these methods often need more geometric structure and in low-texture scenarios where traditional feature detection techniques are prone to failure.In recent years, the advent of deep learning has revolutionized the field of image registration by enabling the extraction of rich semantic features through deep neural networks. Hoang et al [17] and Shi et al. [42] both propose the use of Convolutional Neural Networks (CNNs) to enhance feature representations in image stitching registration. Despite their progress, these approaches primarily use deep learning for feature enhancement rather than creating a holistic learning-based framework.VFISNet [32] is the first complete learning-based framework for image stitching, but it is limited by its inability to handle images of arbitrary resolutions. EPISNet [36] is improved on VFISNet by introducing a flexible mechanism that supports the input of any image size through scalable image and homography adjustments. HomoGAN [18] introduces a method based on the Generative Adversarial Network(GAN) to enhance the quality of homography estimations, representing a novel application of GANs in this field. Jiang et al. [22] integrates graph convolutional networks into the image stitching framework to boost the precision of multi-spectral image registration. LBHomo [21] introduces a semi-supervised approach to estimate homography more accurately in large-baseline scenes by sequentially multiplying multiple intermediate homography. RHWF [7] introduces homographyguided image warping and Focus transformer into the recursive homography estimation framework to further refine homography estimation accuracy.The earliest fusion method is weighted fusion [5] , which requires high registration accuracy. If registration is imperfect or there is a color mismatch between the images, visible seams may appear, which can degrade image quality. APAP [52] introduces a smoothly varying projection field to enhance fusion accuracy. However, APAP tends to introduce severe perspective distortions in nonoverlapping areas, limiting its applicability. Inspired by interactive digital photomontage [1] , Gao et al. [13] propose the seam-based fusion method, which involves a seam prediction stage to identify optimal seam lines between overlapping images. Although effective, it is notably time-consuming. Therefore, SEAGULL [29] proposes to improve the previous seam-based methods by using estimated seams to guide local alignment optimization, enhancing seam quality and reducing processing time. However, it struggles with repetitive textures, where it still shows poor performance.The methods above are all traditional fusion methods characterized by limited versatility and difficulty adapting to complex scenarios. To solve the defects of traditional solutions, UDIS [33] proposes a reconstruction-based model to improve the quality of the fused image. This method sometimes produces artifacts and strange blurs in overlapping areas despite its advances. Inspiration from traditional seam-based approaches, UDS++ [35] and Dseam [11] both use deep learning to refine the seam finding process. Though these fusion methods offer more robust and flexible solutions to improve the ability to handle complex scenarios, they cannot still correct the registration error effectively.Image rectangling is a relatively new area of computer vision with limited research to date. Prior to the advent of deep learning in this domain, traditional solutions such as those proposed by He et al. [15] and Li et al. [25] used mesh-based warping techniques to address missing areas in images. DeepRectangling [34] represents the first deep learning-based approach in image rectangling, accompanied by a baseline and a public dataset tailored for this specific task. The method continues to rely on mesh-based warping but incorporates learning algorithms to enhance the fill quality and handle complex scenarios more effectively. While these methods are groundbreaking, they often change the global relative pixel positions, which could lead to suboptimal results, especially in cases with large missing areas, resulting in incomplete fills. A more recent method RecDiffusion [56] that employs a diffusion model to better solve the rectangling. Although this method provides a sophisticated solution for achieving rectangularity, it is complex in design, requires long inference times, and requires significant computational resources for training, which limits its practical applicability.Moreover, the current deep learning-based image rectangling methods are all based on the DIR-D dataset [34] . DIR-D dataset is a strong assumption dataset, which assumes that some challenging scenes are excluded and that the image registration and fusion are flawless. Therefore, current method do not optimize for the robustness of registration and fusion errors, leading to the error propagation problem illustrated in Figure 1 ③.In addition, there are some works on filling irregular edges in other fields [40, 27, 23] . However, due to significant differences between these fields and the varying shapes of missing regions, these studies are not easily applicable to the rectangling task in the current image stitching pipeline.Several prior studies have attempted to integrate fusion and rectangling stages. For instance, RDISNet [55] claims to create an end-to-end image stitching framework that combines all three stages. However, our analysis of its network structure and experimental results suggests that RDISNet may struggle with large parallax scenes, and noticeable noise in its stitching outputs adversely impacts image quality. Chen et al. [10] propose a diffusion-based method to address both fusion and rectangling tasks. However, this method necessitates meticulously prepared datasets and retraining of the diffusion model, leading to substantial cost demands. In summary, while these existing methods are innovative, they present significant limitations with low reproducibility. As a result, we exclude them from our baseline comparisons. (1) NR-IQA metric settings HIQA. HIQA [43] is designed for the 'wild' image. HIQA is particularly suitable for evaluating the predominantly outdoor images in the UDIS-D dataset, making it an ideal choice for our analysis. We implement this metric based on the public source code with default parameters.CLIPIQA. CLIPIQA [49] based on the Contrastive Language-Image Pre-training (CLIP) models, which allows for adaptable evaluations across different datasets. We use IQA-PyTorch Tool [9] to implement this metric with prompts ['nature image', 'stitched image'] to evaluate whether the stitched images appear more natural.(2) Limitation of NR-IQA metricsThrough our analysis of the results obtained by the NR-IQA metrics HIQA and CLIPIQA, we find a discrepancy between these metrics and human sensory preferences for image quality. Sometimes, our method produces visually higher quality and authenticity images, as shown in Figure 11 , but the scores assigned by these NR-IQA metrics are counterintuitively low.We believe that this problem arises from a mismatch between the training datasets and the unique challenges of image stitching. Metrics such as HIQA and CLIPIQA, are trained on IQA-specific datasets such as KonIQ-10k [19] and Live-iWT [14] , which focus primarily on image distortions such as white noise, low-light noise, and JPEG compression artifacts. These types of distortions are very different from those encountered in image stitching, such as artifacts and incongruous inpainting content. As a result, models trained on such data may struggle to perfectly reflect the true perceptual quality of stitched images.While the NR-IQA metric may exhibit inaccuracies in certain scenarios, it typically indicates superior stitching images in most cases, which is why we have chosen to use it. However, we acknowledge the inherent limitations of current NR-IQA metrics, especially their inability to effectively capture the nuanced aspects of image quality improvements in stitching. Therefore, we do not employ these metrics in the ablation experiments. And, we also conduct user study as a supplementary proof. (1) The impact of K gThe K g in in Eq. 10 and Eq. 11 determines the intensity of the distance transform applied within the weighted masks. Despite the potential variations available in the type of distance transform (e.g., L1 vs. L2) and the kernel size, our empirical observations show that these modifications do not significantly impact the stitching results. Therefore, we set a commonly used value using an L2 distance and a kernel size of 3.(2) The impact of seedWe show the impacts of different seeds in Figure 14 . Our method produces more stable results with high quality. With different random seeds, Stable-Diffusion-v1-5-inpainting (SD1.5) [41] and Stable-Diffusion-v2-inpainting (SD2) [3] produce completely different abnormal contents. In contrast, our proposed method consistently demonstrates remarkable stability. (3) The guidance scale and inference stepThe guidance scale and inference step are two classical parameters in diffusion models. The impact of adjusting these parameters has been extensively validated by previous research [48] . Consequently, this paper does not delve into selecting their values but instead adopts two commonly used settings: 7.5 and 50.This section presents the qualitative evaluation comparing SRStitcher variants based on various diffusion models. The version based on the Stable Diffusion 2 model [2] is designated as SRStitcher-S. Additionally, the implementation utilizing the Stable Diffusion 2 Unclip model [4] is termed SRStitcher-U. Finally, the implementation with Controlnet Inpainting model [54] is defined as SRStitcher-C.The test results are shown in the Figure 15 . The Stable-Diffusion-v2-inpainting model exhibits the best performance, which is the primary reason for selecting it as our base model. The Stable-Diffusion-2-1-Unclip, a fine-tuned model based on Stable-Diffusion V2.1, is chosen in an attempt to leverage its CLIP image embedding functionality. However, the structural integrity of the results We provide the source code for all SRStitcher variants in the Supplementary Material. Therefore, we do not describe implementation of them here.In addition to the UIDS-D dataset, there are traditional datasets in the field of image stitching, such as APAPdataset [52] and REWdataset [26] . However, these datasets are very small, containing only dozens of images, which reduces their usefulness for meaningful comparative experiments.To demonstrate the effectiveness of our method on a broader spectrum of data, we present some experimental results on APAPdataset and REWdataset, as illustrated in Figure 16 .In addition, Figure 17 shows the experimental results comparing SRStither and other methods on the traditional datasets.Here, we present examples of local blur and compare them with other baselines, as illustrated in Figure 18 . We contend that occasional local blur is a tolerable side effect of our scheme, especially when weighed against the generation of significant anomalous content seen in other models. Future research could potentially address this issue by fine-tuning the model.Figure 16 : More results on traditional datasets APAPdataset [52] and REWdataset [26] by SRStitcher. Our solution requires only a single inference step, making it significantly faster than more complex models that require two inference steps, such as UDIS+SD1.5 to UDISplus+SD2. Although our method is slightly slower compared to UDIS+DR to UDISplus+Lama, our experimental results demonstrate that it substantially outperforms these methods regarding stitched image quality, robustness, and generalization. Given these advantages, the minor sacrifice in speed is justifiable.In particular, even without acceleration optimizations like TensorRT [37] , our method achieves an average processing speed of 27 it/s on an NVIDIA 4090 GPU, which is sufficient for real-time performance.This paper aims to introduce a novel image stitching pipeline design that integrates large-scale models into the image stitching process. Unlike previous diffusion model-based image stitching methods, our method does not require training or task-specific supervised datasets. This significantly lowers the implementation threshold, facilitating broader adoption of the method and encouraging more researchers to contribute and enhance this work. One potential negative impact of our method is its ability to make minor modifications to stitched images. This capability could be misused in visual fraud tasks, particularly in the context of surveillance videos.We have made significant efforts to ensure the reproducibility of our method. The code and details of the SRStitcher, Stitcher variants, and CCS metric are all uploaded in the Supplementary Material.NeurIPS Paper Checklist 


Decision: The paper proposes to unify the fusion and rectangling stages of image stitching into a single stage, formulated as an inpainting problem. With this formulation, the proposed method effectively incorporates prior knowledge within pre-trained diffusion models to address the unified problem. Initially, there were concerns about whether the inpainting formulation is a suitable solution for rectangling and fusion, as previous work provided counter-arguments. However, these arguments were made before the emergence of diffusion models, which could be a game-changer. Although the results are imperfect, the proposed method offers a novel perspective on utilizing diffusion models for image stitching. Although several other issues were raised, the rebuttal effectively addressed most. Following the discussion stage, the consensus among the reviewers is positive. Please follow the reviewers' suggestions and include the updated clarifications and experiments in the revised version.

Reviews: review: paper_summary: paper_summary: This paper tried to integrate the fusion and rectangling stages in image stitching into a unified model. More concretely, a special fusion, a rectanlging step, and a mask-guided diffusion model are gathered to implement stitching-customized image inpainting, especially for the irregular boundaries. It is worth mentioning that, the inpainting model uses the pre-trained model and requires no more fine-tuning. To evaluate the proposed method, the authors designed a quantitative metric named CCS and conducted extensive experiments.
strength_weakness: strength_weakness: 1. The so-called unified model consists of several steps including a specially-designed fusion step, a rectangling step, and a inpainting step. I don’t think this multi-step design can be regarded as a unified model. 
2. I am skeptical of whether the inpainting model is meaningful to image stitching. As claimed in the DeepRectangling paper, they abandoned the inpainting model because they think it may introduce some contents that are far from the reality. The manuscript seems to ignore the problem. And I do not think the proposed method can address this issue in the so-called unified inpainting model. 
3. The inpainting results are still not perfect, as illustrated in the image boundaries of Fig. 14.
4. The fusion step of Eq. 4 is special. What’s the motivation for that?
questions: questions: My concern lies in my second weakness. Will this inpainting model for irregular boundaries be meaningful? I have another idea for the inpainting model of image stitching. The inpainting model should not concentrate on the boundaries but on the regions where the artifacts and distortion are produced. Let’s put it in this way. The registration stage may introduce artifacts or distortion. So can we eliminate these issues through an inpainting model? It may be more meaningful to locate these regions, mask them, and then inpaint them. The inpainting process can be implemented with the guidance of original contents, thus contributing to a reliable completion model. This is just a simple discussion. But from my perspective, I cannot figure out the meaning of the proposed inpainting model.
limitations: limitations: No limitation is mentioned in the manuscript.
paper_summary: paper_summary: The paper introduces SRStitcher, a novel method that integrates the fusion and rectangling stages of the image stitching pipeline into a unified inpainting model using a pre-trained large-scale diffusion model, eliminating the need for additional training. This approach addresses the issue of error propagation in traditional pipelines, offering a streamlined and robust solution. Strengths of the method include improved performance in image quality and content consistency, as well as robustness to registration errors. The experimental results are extensive, providing both quantitative and qualitative evidence of SRStitcher's superiority over existing state-of-the-art methods.
strength_weakness: strength_weakness: While SRStitcher simplifies the image stitching pipeline by integrating the fusion and rectangling stages into a unified inpainting model, its technical novelty is questionable. The approach primarily combines existing technologies—pre-trained diffusion models and weighted masks—rather than introducing fundamentally new methodologies. The integration, while effective, does not inherently surpass the capabilities of current state-of-the-art techniques used separately for fusion and rectangling. For SRStitcher to be considered truly novel, it should achieve technical goals that were unattainable with the two processes handled independently. As it stands, the method appears to be more of a consolidation of existing practices rather than a groundbreaking innovation, merely reorganizing the workflow without providing substantial new capabilities or overcoming significant limitations of the previous approaches.
questions: questions: - Technical Novelty: How does SRStitcher fundamentally advance the field of image stitching beyond merely integrating existing fusion and rectangling processes into a unified model? Can the authors provide more evidence of novel technical contributions?

- Performance Comparison: While the paper claims superior performance, how does SRStitcher specifically outperform existing state-of-the-art methods in scenarios with extreme registration errors(large parallax)? Are there any edge cases where SRStitcher struggles compared to traditional methods?
limitations: limitations: The authors have adequately addressed the limitations of their work and there appear to be no significant issues with the broader societal impacts. They have demonstrated transparency and responsibility in discussing the constraints and potential improvements of SRStitcher. Overall, their approach seems robust and well-considered, with no apparent concerns regarding its application or societal implications.
paper_summary: paper_summary: This paper proposes an image stitching algorithm that unifies fusion and rectanguling stages of a conventional pipeline with an image inpainting diffusion model applied with a progressive reverse process guided by weighted masks. Their reverse algorithm progressively inpaints seam regions by gradually increasing the size of soft mask holes, while keep outpainting rectanguling regions with the same hard mask holes. The proposed work is compared with previous works by keeping the registration and fusion stages with UDIS/UDIS++ while altering with various models for a rectangling stage.
strength_weakness: strength_weakness: It is unclear if the repeated reverse process with gradually dilating mask holes is really beneficial than a single reverse process guided by a fixed mask. The repeated application of the re-noising + de-noising step makes the algorithm much slower.

Considering the seam regions are usually narrow while the rectanguling regions could be quite wide, the progressive soft mask dilation during the progressive reverse process may be more useful with the rectanguling masks than the seam masks.
questions: questions: Comparing with a case of a single reverse process guided by a fixed mask would make the paper more convincing.

Comparing with a case that applies progressive mask dilation of the rectanguling mask holes would be useful.
limitations: limitations: Technical limitations are properly addressed. No serious potential negative societal impact is expected.
paper_summary: paper_summary: This work proposes to integrate the fusion and rectangling of image stitching into a unified inpainting model. In particular, the weighted masks are designed to guide the reverse process in a pre-trained large-scale diffusion model, which implements this integrated inpainting task in a single inference. Extensive experiments demonstrate the interpretability and generalization capabilities of the proposed unified model.
strength_weakness: strength_weakness: - While this work is well-motivated, the proposed contributions seem to be weak for me. For instance, the authors proposed a weighted mask to guide the reverse process in the diffusion model. However, the current presentation regarding this contribution looks like an experimental trick rather than a technical proposal.
- The contribution of the proposed unified inpainting model (Section 3.1) is also expected to be highlighted. It is still ambiguous how the fusion part and rectangling part are merged into a unified model. I believe this module is much more important than applying cutting-edge generative models.
- This work leverages a **generation** view to address the **reconstruction** problem. Such an interesting trend can be noticed in the recent works from other research regions as well. However, how to balance the generation and reconstruction in the image stitching problem remains unclear. I believe the users may prefer accurate reconstruction instead of introducing novel content (sometimes artifacts). Moreover, some previous works also proposed to fill the irregular boundaries of images or holes by warping with generative models, such as "Free View Synthesis (ECCV'2020)", "Towards complete scene and regular shape for distortion rectification by curve-aware extrapolation (ICCV'2021)", "iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis (SIGGRAPH Asia)", etc. More discussions are expected to be provided.
- There are some typos in the paper: Line 49: "Therefore, We question". The authors are suggested to further polish this work.
questions: questions: - Can the proposed unified model be extended into other research areas?
limitations: limitations: - The proposed unified inpainting model seems to be impractical compared to previous reconstruction-based solutions.

Comments: title: reminder was sent
comment: My concerns are addressed in the rebuttal and I would like to raise my evaluation.
Thanks for the detailed rebuttal, which addressed most of my concerns. I would like to increase my rating. The updated clarifications and experiments are expected to be presented in the final version.
Dear authors,

The reminders were sent to individual reviewers without posting to the public. I hope that they will post soon.

AC
Dear Area Chair,

We write this comment in the hope that you can remind the Reviewers to reply to the Rebuttal. The time is running out, but only one Reviewer has replied now (Thanks for the timely reply from Reviewer pLdt). 

According to Notifications, we as authors cannot rush Reviewers to reply, which should be the responsibility of the Area Chair. We also do not know whether the Area Chair has reminded Reviewers of the deadline, because we have not received any Notification from Area Chair.

Since there is only one day left and there are still three reviewers who have not replied, we hope the Area Chair will remind the reviewers of the deadline. 

Whether the score is changed or not, the reviewers' reply to our Rebuttal will be a good reference for our subsequent work. We believe that this is why Neurips started the discussion session. We don't think a discussion session where most people are silent is what Neurips hopes to see.
Thank you for your rebuttal. It clarifies the need of different masking strateges for the seam and rectangling regions, and the preservation of computational complexity for the inference. Therefore, my major concerns have been resolved.

I find that the proposed scheme, which converts conventional multiple stages for a stiching problem into unified one by adapting diffusion based inpainting models, holds certain values suitable for the venue. Existing inpatining models would not be able to achieve this goal without applying the proposed algorithm.

I would like to update the final rating from from BA to WA.