Reconstructing the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Inpainting Model
Authors: Xieziqi
Weidong Zhao
XianhuiLiu
Jian Zhao
Ning Jia
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Stitching
Image Fusion
Image Rectangling
Diffusion Model
Primary area: machine_vision
TL;DR: An image stitching method that simplifies fusion and rectangling into a unified inpainting model.

Abstract: Deep learning-based image stitching pipelines are typically divided into three cascading stages: registration, fusion, and rectangling. Each stage requires its own network training and is tightly coupled to the others, leading to error propagation and posing significant challenges to parameter tuning and system stability. This paper proposes the Simple and Robust Stitcher (SRStitcher), which 
revolutionizes the image stitching pipeline by simplifying the fusion and rectangling stages into a unified inpainting model, requiring no model training or fine-tuning. We reformulate the problem definitions of the fusion and rectangling stages and demonstrate that they can be effectively integrated into an inpainting task. Furthermore, we design the weighted masks to guide the reverse process in a pre-trained large-scale diffusion model, implementing this integrated inpainting task in a single inference. Through extensive experimentation, we verify the interpretability and generalization capabilities of this unified model, demonstrating that SRStitcher outperforms state-of-the-art methods in both performance and stability.

Reconstructing the Image Stitching Pipeline: Integrating Fusion and Rectangling into a Unified Inpainting Model
Ziqi Xie', Weidong Zh Xianhui Liu Jian Zhao', Ning Jia 1.2
College of Computer Science and Technology, Tongji University
2 College of Electronics and Information Engineering, Tongji University
fxieziqi, wd, 1xh, zjtju1919, jianing7072) www..gjiieuu.com
Abstract
Deep learning- based image stitching pipelines typically divided into three cascading stages: registration, fusion, and rectangling. Each stage requires its own network training and is tightly coupled the others, leading to error propagation and posing significant challenges to parameter tuning and system stability. This paper proposes the Simple and Robust Stitcher (SRStitcher), which revolutionizes the image stitching pipeline by simplifying the fusion and rectangling stages into a unified inpainting model, requiring no model training or fine-tuning. We reformu- late the problem definitions of the fusion and rectangling stages and demonstrate that they can be effectively integrated into an inpainting task. Furthermore, we design the weighted masks to guide the reverse process in pre-trained large- scale diffusion model, implementing this integrated inpainting task in a single inference. Through extensive experimentation, we verify the interpretability and generalization capabilities of this unified model, demonstrating that SRStitcher outperforms state-of-the-art methods in both performance and stability. Code: httts::/gitaen com Saayyooo66/SRSttttt
Introduction
Image stitching is fundamental problem in computer vision, which aims obtain larger field of view by merging multiple overlapping images [5]. As illustrated in Figure (a), the current deep learning-based image stitching pipeline is typically structured into three sequential stages: (1) Registration Stage. The first stage takes the original image pairs to estimate warping matrices, which then used align images. Current lcarning-basee methods focus on designing the homography estimation networks to address the registration problem [22, 21, 7J. (2) Fusion Stage. The second stage merges the aligned images into a single fusion image. Present research in this domain is generally classified into reconstruction--arte based (recon-base based)) and seam- based methods. Recon- based methods [32, 36, 33] typically use the mccder-decoder networks to perform pixel-wise reconstruction of the fusion image. While seam-based methods [35, 11] focus on identifying the optimal seams eliminate the fusion ghosting.
(3) Rectangling Stage. The final stage transforms the irregularly shaped fusion image into a standard rectangular format. There are only a few deep learning- -ased studies for this stage [34, 56), and they are all supervised methods with requirements for labeling data.
Annoyingly, the cascaded structure of current image stitching pipelines poses significant challenges for training optimization and parameter tuning. Furthermore, errors from early stages tend to propagate later stages, significantly degrading the performance of later processes. The representative
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Stage Registration
Stage 2 Fusion
Stage
Images
Rectangling
Recon-based Ftsion Model
Rectumpling
Mockt
Aligned Images
Fused Images
Stitched Images
Seam-based
Fuzion
(a) Current Image Stitching Pipeline (b) SRStitcher (Ours)
Integrated Stage
Fusion and Rectangling
Input
Stage 1
Coarse F Fusion
Images
Registration
M 39
Weightd Mides
lignedd Images
Pre-tzined
DDFM
Guided
Course Rectangling
Process
Stitched Imanp
Figure 1: Comparison between existing pipeline and SRStitcher. Process ① is implemented by UDIS [33]. process @ by UDIS+ [35], process @ by DeepRectangling (DR) (34), process @ by Eq. 10 and Eq. 11. The corresponding partial images, and /V, illustrate how SRStitcher effectively corrects the apparent misalignment of a pillar. Similarly, the partial images // and /// demonstrate how SRStitcher repairs the blurry coarse rectangling areas. SRStitcher can be applied both UDIS and UDIS++ aligned images and get similar stitched results.
image stitching methods UDIS [a3] UDIS+ [35] both struggle to effectively fuse images with registration errors, as shown in Figure 1 ① and 2 In the rectangling stage (Figure I ()) the prominent rectangling method DeepRectangling (DR) [34] also fails adequately fill gaps, leaving visible black spaces at image boundaries.
As shown in Figure (a)), the errors originating in the registration stage persist through to the final stitched image, and the existing methods lack effective mechanisms to address these errors (detailed in Appendix A.6). The recon-based [32, 36, 33] methods are proven to inevitably introduce artifacts in the stitched image in the presence of registration errors. Furthermore, the seam-based [35, 11] methods rely the assumption that there is perfect seam between the aligned images. When this assumption is not met, the UDIS+ [35] forcibly distorts the images to create perfect seam, resulting in distortion in the stitched image. Therefore, current fusion methods are unable to effectively handle the registration errors shown in Figure 1.
To address the етrо propagation problem, we identify image fusion as the key point for improvement. We reconsider the problem definition of the fusion challenge and hypothesize that By determining the appropriate modification region and introducing ability, the abnormal image content caused by registration error can be effectively corrected. We propose to reformulate the fusion problem by overlaying the less distorted aligned image over the more distorted one, and inpainting the seam area between the images to correct the inappropriate image content.
inpainting model with strong generalization
Building on reframing the fusion problem, we also revisit the rectangling challenge. The core of the rectangling problem is to fill in the missing rectangling area, which is also essentially an image inpainting problem. Therefore, we question whether fusion and rectangling are truly distinct challenges or if they could be addressed as a unified inpainting task. We recognize that handling fusion and rectangling tasks simultaneously not a simple matter of determining the inpainting area. More importantly, it requires precise control of the inpainting process. Specifically, the fusion task involves the preservation of original image semantics to the greatest extent possible, while the rectangle task requires more heavy inpainting fill the missing regions. To effectively manage

---

these varying demands, we introduce weighted masks to guide the reverse process in pre-trained large-scale diffusion model. This method allows for the adjustment of inpainting intensity across different regions during the reverse process, enabling both tasks to be completed within a single inference.
The main contributions of this paper are: (1) We propose SRStitcher, which reformulates the problem definitions of the fusion and rectangling stages construct more streamlined and robust image stitching pipeline. (2) SRStitcher is the first to introduce the concept of inpainting to address the image fusion problem. It incorporates prior knowledge from large-scale pre-trained models into the image stitching pipeline, enhancing the robustness of image fusion against registration errors. (3) Without additional fine-tuning or supervision. SRStitcher improves the generalization of the rectangling method in the zero-shot scenario, opening up new possibilities for unsupervised image rectangling research. (4) We conduct extensive experiments to verify the interpretability and generalization of the proposed unified model. The results show that SRStitcher significantly outperforms the state-of-the-art methods in both quantitative and qualitative evaluations.
2 Background
Registration parameterization. The goal of the registration stage to obtain the aligned images based on a transformation matrix. Given inputs [1(x,)) y). (x, y) E R ## where x: and y represent the pixel coordinates, H, W are the height and width, respectively. And H donates 3 x homography matrix between [xx, y) and (x. y). which maps the input images to an uniform plane. To clarify the process of image registration, take the example of the four vertex coordinates (IR; yk), (1,2,3,4) of the input image. The new image stitching-dove -omain R xW"
can be obtained by Eq.
W =
mAx (xE [. min (- ok) k,(1,,,3,4
(1a)
k,(1.2,3,1)
H1 kE(1,233,4
min f s/.. kE(1.23,44
(lb)
where, (x1² vi" Hx [ vE: 1| Then, the input images are mapped into this new image stitching-
domain by warping operation yp()) to get the aligned images (wt(2; y), Jwr (r. y) E R xW' shown in Eq. 2.
as
(2, v), Lur (x, (lli(1, v), 1), (0,,(x 2), H),
where, is an identity matrix. The masks (I y), Mar (x, y) corresponding to the aligned images can be obtained in similar way by Eq. except that the inputs (xx,)) [x y) are replaced with two all-one matrixes. The specific design of (p-) vary slightly among different stitching methods [33, 35], but the aligned image generation of these methods all follows the architecture of Eq. 2.
Diffusion model. The proposed work is based on the Diffusion Model [16]. Since our method does not include the forward process, only briefly introduce the reverse process. Suppose the X1, .... Xy are latents of the same dimensionality as the Xg ~ ((Xp). where q(-) is the a Gaussian Markov chain forward process with 7' steps. And, Xo (lo((, v)). where E(-) is the image encoder and Lo (z, y) is the input image. The joint distribution of t-th inversion step is defined as Eq. 3.
Pe (x = (x (), Ex ()), (1,T),
(3)
where, (x, t) and Ex(x,, t) are parameters the Gaussian Markov chain in t-th inversion step.
3 Methodology
3.1 Unified inpainting model
Fusion parameterization. Unlike previous methods, our method reconceptualizes the image fusion problem to enhance robustness against registration errors. Precisely, as shown in Eq. 2, the distortion degree of (21 y) is relatively low because it involves only minor warping based on I.
3

---

This means that even in the presence of registration errors, [z, y) does not introduce large-scale distortions. Therefore, we propose to construct coarse fusion image (ar y) via Eq.
(cr(s, v) (x. v) Tur (2,, y) (1 (Mui y)& (x. y)))
where, & and @ denote the bitwise AND operators and element-wise multiplication operator. The coarse fusion image has noticeable seams, as shown in Figure 1(b). Also, with registration errors, incoherent image content appears around the seams. To solve this problem, we propose to focus on inpainting the image content around the seam, ensuring cohesion and coherence. Therefore, use the seam mask Meeam y) to define the area in the fusion image that needs to be inpainted, as detailed Eq. 5.
Maeam (2, v) Dilation(MM((11 Ks) E Mwt y)v
(5) where, Dilation()) and Erosion(.) denote the dilation and erosion operations [38], K, is the kernel sizes, V, ① denote bitwise OR and XOR operators. Then, we inpaint (cr y) based on seam mask Maeam (x,z) and the inpainting function fa(-) to obtain inpainted fusion image IcF (z, v), as detailed Eq.
Erosion (Mai y), K,) @ Mid (I, U)&M&rr (vr y);
(e y) - (cF(2.)) @ Maeam (I, 9)) + fo (IFF(( v)) Macam (I, y).
(6)
Rectangling parameterization. Our method also defines the rectangling challenge as an inpainting problem based the content mask Moontent (Ir, y). We use Eq. to obtain the inpainted rectangling image [x.(x.y).
(oR(Z,8) y) (CF(2,)) ① (1 M (z + fe(Icr(x,)) Meontent v).
(7)
where, Moontent(2, y) Mmn (2,z) y) V M (2, y)-
Unified model. Integrating Eq. 6 and Eq. 7, we obtain a unified inpainting model for fusion and rectangling, as shown Eq. 8.
icrR(E,)) v) Ic)(a,)) @ (1 Manpaint (Ix v)) + fo (Tcr(xx,)) y)) O Minpaine (x, y),
(8)
where, M Minpaint (Z, y) Mseam (I, y) V Mcontent (z, y). By combining equations Eq. 3 and Eq. 8, this inpainting problem can be solved by diffusion model, as detailed Eq. 9.
x = xo (1 Minpoint (I, y)) + X 2 Ménpaint (I, y),
where, Xo = (1 y)), and xt-1 ~ N(au(( 1), Ex(xt, 1)).
3.2
Weighted mask guided reverse process
After defining the unified inpainting model for the fusion and rectangling tasks in the previous subsection, we discuss the method to control the inpainting strength different regions during the reverse process, ensuring that both tasks can be accomplished in single inference. Specifically, regions under the mask Mscam (I. y) that contain the semantics of the original image are preserved much as possible. In contrast, regions under Mcontene(2) y) may require We propose weighted masks to guide the reverse process to achieve this varying inpainting strength.
powerful inpainting.
Weighted masks. Weighted masks are constructed from the weighted initial mask Minst (2, y) and inpainting mask Minpaint (z, y).

---

Algorithm Weighted Mask Guided Reverse Process (WMGRP)
1: Input: Coarse Fusion image [or( y); Inference steps N; Radius R:
2: Weighted initial mask Minit (I, y); Weighted inpainting mask Minpaint (2, y)
3: prompt p f-
b Our method does not require prompt guidance
4: ICPR(æ, y) +- Telea
y), Moontent y), R)
Coarse rectangling
5: XN < E(ICFF(Z,,)) v))
> Encode image
6: // Based on the inpainting model, so there is little difference here with the Eq. 9
7:
Xg - E((FFR((z,)) y) Minat y))
8:
M Mipppannt (x, y), M Mist (I, y) + DownSample (Minpaine (z, v), Minit (2r, u))
9:
X'y - AddNoise(xN.N))
10: XN + Concat (xy,, Mamall (x,,) (I, Xo)
11:
for N 1, 1.... do
D Reverse process
12:
X + DeNoise((::11.0)
13: M. (rr t- (Mampati (I, 1 2x2) C-Cocaatxi, M, M.maall (x. (). 15: end for
D x means element-wise less-than
14:
16: (CFR(E, v) +- ImageDecoder(xx)
Decode image
Output: [CFR(M.)))
The weighted initial mask Minor y) assigns different fidelity levels to each pixel of the fusion image, determining how much to modify each pixel based on its fidelity during the reverse process.
The formula of Minir (2, y) is given by 10, which is composed of two parts. The left part determines the fidelity levels of pixels in Maeam (I, y) region, and the right part determines the fidelity levels of pixels in M Monntent (x,, y) region.
DT (I, y), Kg) x
DT "T(Moontent (I, Kg) x 62 max (Meontent (I. y). Kg)*
Mina v)
(10)
Da (I, y). Kg)
where, DT(-) is the distance transform operation [38] with kernel size Kg. 61 and 62 are hyper- parameters.
The weighted inpainting mask M Minpaint (z. y), as described in 11, 1, is inspired by the suffix principle [24]. During the reverse process, Mippannt (x. y) is mapped into multiple sub-maske define the modified regions at each step t. The region corresponding to Mcontent (I, y) contains no image content, so its size remains constant across all sub-masks, ensuring it is repainted throughout the entire process. Conversely, the region corresponding to M.eam (2., y) contains a substantial quantity of original image information, and its size gradually increases with each step t, indicating that this region requires progressive modification. This gradual modification method facilitates a more seamless blending of the inpainting content with the original image content.
Minpaint (x. v) = Meontent (z, v) V DDT-DD (I, Kg)).
(11)
Guided reverse process. We observe that when the missing region of the fusion image is large, the diffusion model very easily generates abnormal content, such abnormal textures and words. To mitigate this issue, we introduce coarse rectangling. To be specific, we em- ploy the Alexandru Telea Algorithm Telea(-) [46] to generate the coarse rectangling image: ICFR(I,)) y) = Telea (z. Mcontena (z. v), R). where R is the radius of circular neigh- borhood of each point inpainted. The Telea(.) algorithm introduces a weak prior without any specific semantic information to the image Icr(æ, y). As shown in the partial image of Figure the image of the coarse rectangling region is completely blurred. The experimental result shows that generating images with weak priors significantly reduces the likelihood of producing anomalous
5

---

content compared to leaving blank areas entirely black. More details regarding the advantages of coarse rectangling can be found in the Appendix A.1.
The specific steps of the reverse process are detailed in Algorithm 1. Although this algorithm is based on the Stable Diffusion Inpainting model [41,3], which differs slightly from the original Stable Diffusion model [2], the underlying principles remain consistent. In addition, our method works without the need for prompt, effectively reducing dataset requiremenss.
Appendix A.2 provide more detailed explanation and visualization of the weighted masks and WMGRP algorithm.
4 Experiments
4.1
Experimental setup
Dataset. To validate the performance of our method, we conducted experiments on the large public dataset UDIS-D [33]. To the best of our knowledge, UDIS-D is the only publicly available large-scale dataset in this field. Appendix D.4 provides more results of our method on other traditional small datasets.
Baselines. To our knowledge, no open-source solutions simultancously address the fusion and rectangle stages of the image-stitching pipeline as comprehensively as our method. Table ((a) gives brief statistics of related works, and more related work details provided in Appendix B. Therefore, we have to establish the comparison baselines by combining several existing methods. For the registration and fusion stage, we employ pre-trained models from UDIS [33] and UDIS+ [35]. For the rectangling stage, we utilize pre-trained models from DeepRectangling (DR) [34], Lama [44], Stable-Diffusion- -ir-iion------iianiiinn- (SD1.5) [41], and Saable-Diffusioonn2-nnnaatttt (SD2) [3]. Table 1(b) presents the detailed configurations of baselines.
Table Statistics related works and details of comparison baselines.
(a) Statistics of related works.
(b) Details of comparison baselines.
Name
Stagel
Stage2
Stage3
Baseline
Stagel and 2
Stage3
VFISNet [32]
X
UDIS+DR
UDIS
DR
EPISNet [36]
X
UDISplus+DR
UDIS+
DR
UDIS [33]
/
X
UDIS+Lama
UDIS
Lama
UDIS+ [35]
X
Dseam [11]
X
/
X
UDISplus+Lama
UDIS+
Lama
Jiang et al. [22]
X
UDIS+SDI.5
UDIS
SD1.5
LBHomo [21]
X
X
UDISplus+SDI.5
UDIS+
SD1.5
RHWF [7]
X
X
UDIS+SD2
UDIS
SD2
HomoGAN [18]
X
X
UDISplus+SD2
UDIS+
SD2
DR [34]
X
Variants. In this paper, mainly present SRStitcher based Stable Diffusion Inpainting model [3]. However, our method versatile and can be readily adapted other diffusion-based models with only minor modifications. In the experiments, we also compare the SRStitcher variants, including: SRStitcher-S based on the Stable Diffusion 2 model [2], SRStitcher- U based on Stable Diffusson Unclip model [4], SRStitcher-C based Controlnet Inpainting model [54]. For further information on SRStitcher variants, please refer to Appendix D.3.
Metrics. (1) Stitched image quality. Since UDIS-D is an unsupervised dataset, we use the No- Reference Image Quality Assessment (NR-IQA) metrics evaluate the image quality. Specifically, we use the HIQA [43] and CLIPIQA [49]. (2) Content consistency. We develop a new metric to evaluate the content consistency between the input images and stitched image. Specifically, we introduce the CoCa(-) [50] model and Bert(.) [39] model to extract text from the images and generate text embeddings. The similarity between these embeddings measured by the cosine similarity cosine(-). We design the Content Consistency Score (CCS) metric:

---

CCS +
(12)
CCS, measures the local consistency, which compares the stitched image Lstitched y) and the fusion image Pusiio (z. y). Both images are split into n equal parts for detailed com- parison: CCS, cosine (SSp((((((S(tssstteeheeee
y), n)), [[Split(TFusion (2. y), n))), where T()) Bert(CoCa(-)), and for this test, n 4. In addition, CCS, assesses the over- all content consistency between the Iststched and original input images: CCS, (o v), (II(F, u)).. Please refer to Appendix C for further information on metrics.
Implement details. All experiments are performed on a single NVIDIA 4090 GPU, In addition, all experiments of SRStitcher described in this paper based on these pre-aligned images made by UDIS+ [35]. For hyper-parre parameters, the guidance scale and inference steps N are set to 7.5 and 50; The K, in Eq. 5 is to [W* where À and 5 10; The Kg Eq. 10 and Eq. 11 are set to 3; The R in Telea(-) is set to 20. The and in Eq. 10 are set to 128 and 128.
Table Quantitative results. The best and second-bess results are highlighted by red and blue. refers to the inference results of this method are not affected by seed. means the inference results of this method are affected by the seed. We tested the results five times by varying the seed, taking the average and standard deviation.
UDIS Drest
UDIS Derain
Method
HIQA 1
CLIPIQA +
CCS($))
HIQA f
CLIPIQA f CCS(%) f
UDIS+DR*
42.53
28.33
89.35
45.31
31.29
90.02
UDISplus+DR*
45.98
31.24
88.45
49.87
33.47
90.69
UDIS+Lama*
42.55
27.17
84.99
45.63
30.15
86.70
UDISplus+Lamax
46.57
31.48
87.73
51.28
33.29
86.12
42.60
28.03
87.42
48.59
28.57
87.74
UDIS+SD1.5f
2.24
2.44
+ 1.08
1 1118
1 0.89
+ 1.36
46.45
27.13
87.16
50.89
30.16
88.12
UDISplu++DDI.5f
11111
1 1.8
1 1.61
1: 2.20
1 1.46
+ 1.35
42.84
28.00
85.97
47.15
34.31
85.72
UDIS+SD21
+ 1.08
0.89
+ 1.33
$ 1.33
0.95
= 1.55
46.98
31.23
89.37
51.49
34.26
91.18
UDISplus+SD2f
1.43
2.18
+ 1.23
$ 1.74
+ 1.24
1.35
SRStitcher
Variants
45.66
32.08
85.91
51.73
35.23
87.32
SRStitcher-St
0.89
0.91
+ 0.74
+ 0.79
43.89
28.35
85.81
48.18
31.38
86.33
SRStitcher-Uf
1.01
i 0.666
1 1.01
1 0.55
1 0.74
1
46.57
31.34
89.47
52.73
34.53
91.41
SRStitcher-Cf
0.89
0.766
0.771
0.74
+
- 0.84
47.82
33.25
91.15
54.74
37.52
93.29
SRStitcheri
0.55
0.577
0.52
# 0.63
+ 0.68
- 0.45
4.2 Quantitative evaluation
We perform comprehensive quantitative analysis by comparing the results of 10,440 sample pairs from the UDIS-D training set UDIS Derain and 1,106 sample pairs from the testing set UDSS Drest- Notably, our method does not require training, so to provide a broader base of comparison, the training set of UDIS-D is also included in the comparison experiments. The comparative results are presented in Table 2, which illustrates the significant advantages of SRStitcher in terms of the stitched image quality and content consistency.

---

4.3 Qualitative evaluation
We perform a quantitative evaluation SRStitcher against other baseline methods, depicted in Figure 2. The first row of Figure 2 presents a challenging registration scenario involving soft and deformable objects, such as wires, which may have deformed unpredictably between two images. Current registration methods cannot accurately align such objects. Instead of attempting to register or fuse these deformed wires, our method opts to inpaint incorrect wires, effectively overcoming registration errors. A more detailed discussion on this is available in Appendix A.4. The second row of Figure illustrates challenges associated with structured and extensive missing areas, where methods like DR and Lama struggle to accurately fill in the image content. The third row addresses the repeated pattern challenge, where a large number of bricks significantly complicates registration accuracy. The fourth row highlights the classic multi-depth layer problem, illustrating how objects like pillars and their backgrounds, being on different depth layers, result in registration inaccuracies. To enhance the clarity of presentation, the results of UDiSplus+DR, UDIS+Lama, UDIS+SD1.5 and UDISplus+SDI. are omitted from this figure. Detailed qualitative evaluations for all comparison methods are provided in Appendix D.
Inputs
UDIS+DR
UDISplus+Lama
UDIS+SD2
UDISplus+SDZ
SRStither
Figure Qualitative evaluation results. All visual results are obtained with seed
4.4
User study
We introduce user study metric from UDIS [33]. This method allows for a more subjective but insightful visual quality assessment through direct user feedback. For the user study, we display four images simultaneously on single screen: the two input images, our stitched result, and the stitched result from one of the baseline methods. Participants are asked to determine which result is superior, SRStitcher or Another (comparison baseline). If a clear preference is not apparent, participants
8

---

can choose Both Good or Both Bad. The study involves 20 participants: 10 researchers (computer vision background) and 10 volunteers (non-computer major). This diverse group ensures balanced perspective, combining expert technical evaluation with general user impressions. The results are shown in Figure 3.
4.5 Ablation study
hyype--prraneter Figure 4 illustrates the ablation results in hyper-parameter of SRStitcher, demon- strating that these parameters are highly interpretable and easy to adjust. (1) A: Controls the width of the region in Maeam- smaller À increases the modification range and decreases the CCS. For stitched pictures with color differences, lower À value can better fuse the images. We set À to 200, considering both image smoothness and CCS. (2) R: Controls the granularity of the coarse rectangling image. A higher R. value provides higher quality weak prior for inpainting, reducing the likelihood of generating abnormal content. Ideally, a larger R is preferable, but due to the limitations in GPU acceleration with the Telea(-), very high R value can slow down the pipeline. Thus, we balance performance and speed by setting R to 20. (3) 61: Controls the inpainting strength of the seam area. At E₁ 128, the shape of the pillars appears more reasonable compared to é1 = 64. However, increasing é to 192 significantly alters the image content, so we set it to 128. (4) e2 Controls the inpainting strength of the rectangling area. When e2 is relatively low, the image structure remains largely intact, but increasing it to 192 leads to noticeable structural deficits. Therefore, we set E2 to 128. Please see Appendix D.2 for more comprehensive hyper-parameters studies.
Both Good SRShttche(Ouus))
Both Bad
2 =100
10
=64
64
Another
C.C.Snn 26.
90
18
785
773
80
723
70
65.3
60
2 200
15
128
=123
CCS-91.03
80E501 10
30
23.5
20
11.8
130
13
148
203
16.6
10
A 300
= 20
192
192
CCSe92,
UDIS-DR
UDISPLUS-DD
UDIS -LAM
UDISPLUS-LAN LLAMA
UDIS+SDI.5
UDISPLUS+SDI
UDIS-SD2
UDISPLUS-BD -SD:
Figure 3: User study on visual quality. Figure 4: Ablation study results. CCS on each image the The results averaged across 20 participants, with the percentage on the ordinate axis.
average score of U UDIS- Drest with this hyper-parameter and seed 0, not a single image.
Weighted Masks. Figure 5 illustrates the ablation results in different masks guided manners. Specifically, Figure 5(a) represents the different effects of using fixed mask and weighted mask in the fusion region. As shown in the red box, the weighted mask better smoothes the image content while preserving the original information of the image by gradually modifying the image content, while the fixed mask significantly modifies the original image content. In addition, Figure 5(b) illustrates the stitched images obtained by the gradually changing weighted masks the rectanguling region. Since the rectanguling region contains no image content, the generator guided by the gradually changing masks repeatedly smoothes the empty region, resulting in blurred noise. Therefore, we use fixed size mask in the rectanguling region.

---

SRSWtther
Fixed mask
Weighted mask
(a) Fusion Region
(b) Rectanguling Region
Figure 5: Ablation of the masks guided manners.
Discussion and conclusion
This paper introduces SRStitcher, which reconceptualizes the fusion and rectangling stages unified inpainting model. Through weighted masks, SRStitcher leverages the robust generalization capabilities of a pre-trained large-scale generation model to accomplish this complex inpainting task without additional fine-tuning or task-specific data annotations. Extensive experiments demonstrate that SRStitcher significantly outperforms existing state-of-the-art methods regarding the quality of the stitched images and its robustness to registration errors and abnormal content. Furthermore, specific effects and adjustments of each hyper-parree parameter in SRStitcher are detailed in the ablation studies, illustrating its high interpretability and controllability.
However, there are still some limitations and open issues in future research: Visibbe seam. When input images exhibit significant color differences, visible seams may appear with the parameter settings described in the paper. Adjustments to and À can partially mitigate this issue, but such modifications can compromise the preservation of original image information. We speculate that more flexible and appropriately designed hyper-parameter selection scheme could solve this problem.
(2) Local blurring. We coarse rectangling and Minit(:: (x, y) control the content generation. However, this approach introduces a side effect where some challenging scenes appear locally blurred (See Appendix D.5). This issue presents a dilemma: accept local blurring or risk producing anomalous images. We temporarily choose to tolerate local blurring. Future improvements will include a refined coarse rectangling approach fine-tuning the model. (3) Integrating registration. Is it possible to integrate the registration stage into the unified model? According to Diffusion Features (DIFT) [45], is possible. DIFT proves that the geometric correspondence between images can be effectively established by extracting feature maps from their intermediate layers at a specific timestep during the inverse process. Replacing the registration method used by SRStitcher with DIFT straightforward. However, our ambitions extend beyond simple replacement. We believe there is potential for more elegant and concise method to integrate concepts proposed by DIFT into our existing method.
6 Acknowledgment
This paper was supported by the National Key Research and Development Project of China (Grant No.2023YFB34008600 and Science and Technology Innovation Program of Shanghai (Grant No. No.18DZ22291100.
References
[1] Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, and Michael Cohen. Interactive digital photomontage. In ACM SIGGRAPH 2004 Papers, pages 294-3292 302-302.40 2004.
10

---

[2] Stability AL Stable diffusion 2. http://hhggggttooooooo/aaa/////d 2023.
[3] Stability AL Stable diffusion 2 inpainting.
http:://uuginngaaee..o/ssaaaaa1p stable-diffusiee diffusionn------ 2-i--a-iiiininng..... 2023.
[4] Stability AL Stable diffusion 2.1 unclip.
stable diffusion 2 uncl ip. 2023.
[5] Alex Andrew. Multiple view goometry in computer vision. Kybernetes, 3009/10):1333- 2001.
[6] Matthew Brown and David G Lowe. Automatic panoramic image stitching using invariant features. International journal of computer vision, 74:59-73, 2007.
[7] Si-Yuan Cao, Runmin Zhang, Lun Luo, Beinan Yu, Zehua Sheng, Junwei Li, and Hui-Liang Shen. Recurrent homography estimation using homography-guideed image warping and focus transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages June 2023.
[8] David Capel and Andrew Zisserman. Automated mosaicing with super-resolution zoom. In Proceed- ings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No. 98CB36231), pages 885-891. IEEE, 1998.
[9] Chaofeng Chen and Jiadi Mo. IQA-PyTorch: Pytorch toolbox for image quality assessment. [Online]. Available: https:/ gith... com/ chaofenge/ IQA --yTorch 2022.
[10] Jun Chen, Yongxi Luo, Jie Wang, Honghua Tang, Yixian Tang, and Jianhui Li. Elimination of irregular boundaries and seams for uav image stitching with a diffusion model. Remote Sensing, 16(9):1483, 2024.
[11] Senmao Cheng, Fan Yang. Zhi Chen, Nanjun Yuan, and Wenbing Tao. Deep seam prediction for image stitching based on selection consistency loss. arXiv preprint arXiv:2302. 05027, 2023.
[12] Junhong Gao. Seon Joo Kim, and Michael S Brown. Constructing image panoramas using dual-homography warping. In CVPR 2011, pages 49-56. IEEE, 2011.
[13] Junhong Gao, Yu Li, Tat-Jun Chin, and Michael S Brown. Scam-driven image stitching. In Eurographics (Short Papers), pages 45-48, 2013.
[14] Deepti Ghadiyaram and Alan C. Bovik. Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions on Image Processing, 25(1):372 387, 2016.
[15] Kaiming He, Huiwen Chang, and Jian Sun. Rectangling panoramic images via warping. ACM Transactions on Graphics (TOG), 3204):1-10 2013.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
[17] Van-Dung Hoang, Diem-Phue Tran, Nguyen Gia Nhu, The-Anh Pham, and Van-Huy Pham. Deep feature extraction for panoramic image stitching. In Intelligent Information and Database Systems: 12th Asian Conference, ACIIDS 2020, Phuket, Thailand, March 23-26, 2020, Proceedings. Part 11 12, pages 141-151. Springer, 2020.
[18] Mingho Hong, Yuhang Lu, Nianjin Ye, Chunyu Lin, Qijun Zhao, and Shuaicheng Liu. Unsupervised homography estimation with coplanarity-wware gan. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 17663-17672, June 2022.
[19] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe. Koniq- An ccologically valid database for deep lcarning of blind image quality assessment. IEEE Transactions on Image Processing. 29:4041-400 -006,, 2020.
[20] Jitesh Jain, Yuqian Zhou, Ning Yu, and Humphrey Shi. Keys to better image inpainting: Structure and texture go hand in hand. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 208-217, 2023.
[21] Hai Jiang, Haipeng Li, Yuhang Lui, Songehen Han, and Shuaicheng Liu. Semi-supervised deep large- baseline homography estimation with progressive equivalence constraint. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1024-1032, 2023.
[22] Zhiying Jiang. Zengxi Zhang, Jinyuan Liu, Xin Fan, and Risheng Liu. Multi Multi-spectral image stitching via spatial graph reasoning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 472 480, 2023.
11

---

[23] Yash Kant, Aliaksandr Siarohin, Michael Vasilkovsky, Riza Alp Guler, Jian Ren, Sergey Tulyakov, and Igor Gilitschenski. invs: Repurposing diffusion inpainters for novel view synthesis. In SIGGRAPH Asia 2023 Conference Pupers, pages 1-12, 2023.
[24] Eran Levin and Ohad Fried. Differential diffusion: Giving each pixel its strength, 2023.
[25] Dongping Li, Kaiming He, Jian Sun, and Kun Zhou. A geodesic- preserving method for image warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 213-221, 2015.
[26] Jing Li, Zhengming Wang, Shiming Lai, Yongping Zhai, and Maojun Zhang. Parallax-tolerant image stitching based robust elastic warping. IEEE Transactions on multimedia, 20(7) 2017.
[27] Kang Liao, Chunyu Lin, Yunchao Wei, Feng Li, Shangrong Yang, and Yao Zhao. Towards complete scene and regular shape for distortion rectification by curve-aware extrapolation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14569-14578, 2021.
[28] Tianli Liao and Nan Li. Natural image stitching using depth maps, 2023.
[29] Kaimo Lin, Nianjuan Jiang. Loong-Fah Cheong, Minh Do, and Jiangho Lu. Seagull: Seam-guided local aligmment for parallax-tolerant image stitching. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 385-385. Springer, 2016.
[30] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models, 2022.
[31] Philip McLauchlan and Allan Jaenicke. Image mosaicing using sequential bundle adjustment. Image and Vision computing. 20(9-100 (:0-9-1)7511-1111 1599, 2002.
[32] Lang Nie. Chunyu Lin, Kang Liao, Meiqin Liu, and Yao Zhao. view-free image stitching network based on global homography. Journal of Visual Communication and Image Representation, 73:102950, 2020.
[33] Lang Nie, Chunyu Lin, Kang Liao, Shuaicheng Liu, and Yao Zhao. Unsupervised deep image stitching: Reconstructing stitched features to images. IEEE Transactions on Image Processing, 30:61844 84-6197, 2021.
[34] Lang Nie, Chunyu Lin, Kang Liao, Shuaicheng Liu, and Yao Zhao. Deep rectangling for image stitching: learning baseline. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 5740-5748, 2022.
[35] Lang Nie, Chunyu Lin, Kang Liao, Shuaicheng Liu, and Yao Zhao. Parallax-tolerant toleeran unsupervised deep image stitching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7399-7408 2023.
[36] Lang Nie, Chunyu Lin, Kang Liao, and Yao Zhao. Learning edge-preserved image stitching from multi- scale deep homography. Neurccoppttnng 491 533-553,,4,, 2022.
[37] nvidia. Tensorrt. https: thub, com/ Sttm//VIDAA/Teeno/t 2023.
[38] OpenCV. Open source computer vision library, 2015.
[39] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019.
[40] Gernot Riegler and Vladlen Koltun. Free view synthesis. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XIX 16, pages 623 640. Springer, 2020.
[41] Runway. Stable diffusion 1.5 inpainting. stable-diffusion diffusionn----- hew.s/iiinnbboobmmii//ee/ee vi-e-vvv------ ww----npainting.......mmmm ckpt, 2022.
httpe://nuggiingace caleee..co/uumaee co/runvaym1/
[42] Zaifeng Shi, Hui Li, Qingjic Cao, Huizheng Ren, and Boyu Fan. An image mosaic method based convolutional neural network semantic features extraction. Journal of Signal Processing Systems, 92:435-444 2020.
[43] Shaolin Su, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. Blindly assess image quality in the wild guided by self-adaptive hyper network. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
12

---

[44] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov. Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robusi large mask inpainting with fourier convolutions. arXiv preprint urXiv:2109.07611, 2021.
[45] Luming Tang. Menglin Jia, Qianqian Wang. Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[46] Alexandru Telea image inpainting technique based on fast marching method. Journal of graphics tools, 9(1):23 34, 2004.
[47] Prune Truong, Martin Danelljan, and Radu Timofte. Glu-net: Global-local universal network for dense flow and correspondences. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 6258-6266 6268, 2020.
[48] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https:// htt///gitttt tighu........ hatp/huggingg waw.aag./eee.c.c aate/a//e seeeeiffffeeee 2022.
[49] Jianyi Wang. Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In AAAI, 2023.
[50] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[51] Jian Yu, Yi Yu, and Feipeng Da. Parallax tolerant image stitching with epipolar displacement field, 2023.
[52] Julio Zaragoza. Tat-Jun Chin, Michael Brown, and David Suter. ppprjjcttiveaa-posiiie image stitching with moving dlt. In Proceedings of the IEEE conference computer vision and pultern recognition, pages 2339 2346, 2013.
[53] Yanhong Zeng. Jianlong Fu, Hongyang Chao, and Baining Guo. Aggregated contextual transformations for high-resolution image inpainting. IEEE Transactions Visualization and Computer Graphics, 2022.
[54] Lvmin Zhang. Control v1 lp sd15 inpaint. httpss//huughngaaa...ooo/aaaaaa sd15 sa15_inpaint, 2023.
_v11p-
[55] Hongfei Zhou. Yuhe Zhu, Xiaoqian Lv, Qinglin Liu, and Shengping Zhang, Rectangular-ouppt image stitching. In 2023 IEEE International Conference on Image Processing (ICIP), pages 2800-2804, IEBE, 2023.
[56] Tianhao Zhou, Haipeng Li, Ziyi Wang, Ao Luo, Chen-Lin Zhang, Jiajun Li, Bing Zeng. and Shuaicheng Liu. Recdiffusion: Rectangling for image stitching with diffuusinn models, arXiv preprint arXiv:2403. 191644 2024.
[S7] Imad Zoghlami, Olivier Faugeras, and Rachid Deriche. Using geometric corners to build a 2d mosaic from set images. In Proceedings of IEEE computer society conference on computer vision and pattern recognition, pages 420 425, IEEE, 1997.
13

---

Appendix
A More details of SRStitcher
A.1 More analysis of the designs described in the main paper
To elucidate the specific role of each design element in SRStitcher, Figure illustrates the results after sequentially removing designs:
(a) SRStitcher result. The result is obtained when coarse rectangling, Minit (2, v), and Minpaine (z, y) are all used.
(b) Remove coarse rectangling, retain Mini( y) and Minpaint (z, y). Removing coarse rect- angling while maintaining (zn y) results in the incomplete filling. As mentioned above, this because Msnst (z, y)is still working and retains the original image information. But, the area previously filled by the coarse rectangling returns pure black, which affects the final stitched result.
(c) Remove coarse rectangling and Minit (z, y), retain M Mmpaine (It y). Replacing Manat (z, y) with Moontent (r, y) eliminates its effect, allowing the rectangling area to be completely filled. However, compared to (a), the content changes significantly, and abnormal content emerges. This result indicates the importance of coarse rectangling in providing weak priors for guiding the generation.
(d) Remove coarse rectangling, Minit (I, y), and Mimpoine (2, y). By removing all design elements, SRStitcher becomes simple inpainting model based on Minpoint (2z, y). This results in higher probability of generating abnormal content, with substantial alterations near the seam (indicated by the red box).
Inputs
SRStitcher Result
Remove
Remove Mini (x, y)
Remove M Mappaine (x.y)
Coarse Rectangling
(a)
(b)
(c)
(d)
Figure 6: The effects of each design on SRStitcher results. We demonstrate how each design element influences the results of SRStitcher by removing them one time and observing the changes.
A.2 Physical implications of weighted masks Minit (2, y) and Minpaint (æ,, y)
We design Minse (I, y) and Minpaine (x, y) is illuminated by the following observations:
(1) The input structure of the inpainting model. Unlike the general Stable Diffusion model[2], the input of the Stable Diffusion Inpainting model comprises the original image, mask and masked image. Through experimental tests, we found that variations in any part of this composite input significantly influence the final output results.
(2) Impact of masked image. The masked image retains and continuously provides the unmasked area information of the original image throughout the reverse process, ensuring that the unmasked area of the final generated image remains consistent with the original image.
(3) Impact of mask. The mask is employed to identify the modified regions during the reverse process. Adjusting the scope of the mask during the process allows for the controlled modification of the inpainting intensity across different regions. One point note is that in the input mask of the
14

---

Stable Diffusion Inpainting model: the black is the area that needs to be retained, and the white is the area that needs to be inpainted.
DT(Merr (l. Won (h.Kg) max MocctiiMseeee
DT (T Monteeee (1,
Mwi (x.
Mcontene (x.y)
M content y) Mur (x.y) (%,
Eq. 9
Mar (X,
M seam (x,y)
M Minnains y)
Eq. 5
Eq.
(a) Memmtens V M7(Mauna (a. K2).
(8 MM,y) (...
Figure 7: Visual production of masks. We provide the correlation between equations and masks to facilitate comprehension.
Based on the above findings, we propose the construction of weighed masks.
(I) Control of masked image. In our design, (x, y) is used to create the masked image, determining the extent to which information from the original image is retained. We aim to fully preserve the image content in Mcontent (I. 3). While, for areas outside M Mcontent y) (that is the coarse rectangling area), we implement distance transform to change the Moontens (x, y). Due to the coarse rectangling regions are blurry (but must use the weak prior, explained in A. I), it undesirable to retain substantial information from these blurry regions. By employing a distance transform, we can gradualyy reduce the information from the coarse rectangling image starting from the edges Mcontent (x, y). This approach ensures that only the most relevant information from the edges of the optimal coarse rectangling is retained, avoiding the use of excessive coarse rectangling data.
(2) Control of mask. In our design, the inpainting mask is dynamically adjusted throughout the reverse process based on Minpaina (z, y). Although M Minpainal (I, y) serves as just mask, its mask area is modified at each step by calculating the threshold N-t and remapping Minpaint (I. y) to N
Mymail (x, y) accordingly (as detailed in Algorithm 1).
Figure provides the detailed mask productions. The final Minit y) and Minpaint (2, y) contain gradient areas, and we realize the gradual control of the reverse process based on the gradient areas. The M2 y) is used to store information about the fusion image. Intuitively, darker places hold more information about the original image.
A.3 Visualization of WMGRP
As supplement to Algorithm 1, we provide Figure 8 to visually illustrate the process of WMGRP using the masks generated from Figure 7. The Minpeine (z, y) is employed to regulate the intensity of inpainting in distinct regions during the reverse process. Intuitively, the white part is not modified at all, and the black part is modifiable. It can be observed that the black area of inpainting mask is gradually increased during the reverse process, which serves as the guiding process for the gradual modification.
Although the above content mainly based on the observations experimental results of the Stable Diffusion Inpainting model, our experiments have proved that it also applies to the general
15

---

Stable Diffusion model [2] with only minor modifications. We provide the detailed implementation of SRStitcher variants in the Supplementary Material.
x
x
X
Icr y),
ICFR (CFR(x,)) y)
Minpaint (x, y)
XN
M
smalll (x,y) N
Mymall (x,y)
M (x,y)
Figure 8: Visualization of WMGRP. For simplicity, we omit the masked image, which is invariant
after initialization throughout the process.
A.4 Why is SRStitcher so effective at overcoming registration errors
Unlike the previous fusion and rectangling methods SRStitcher does not rigidly adhere to the registration results. Figure 9 provides a clear example of how SRStitcher addresses wire registration errors. In the coarse fusion image [CPR(2, y), the misregistration problem is still serious, which is reflected in the significantly misaligned wires.
After the inpainting, these misaligned wires are effectively corrected, while the content in other masked areas remains largely unchanged. We attribute this remarkable correction capability to the strong generalization ability of large-scale generative models. This ability to correct incorrect image content underpins our motivation for employing the large-scale pre-trained model.
x/
x
Iaputs
Tesa (x,y)
Masked area that is alloved modified
Inpointing
Mamall
Momall (x,y)
Ramall
Figure 9: How SRStitcher addresses the issue of registration errors. Due to the disparate parameter settings, the figure differs from the main paper. The display effect and parameters of the main paper shall prevail.
For visual illustration, Figure 10 compares the fusion effects of SRStitcher and other methods in large-parallax scene. The result shows that, SRStitcher's inpainting- based fusion method achieves the best continuity of image content in this challenging scenario.
A.5 The necessity of using a large-scale diffusion model
Although the diffusion model has been shown to outperform Generative Adversarial Network (GAN) methods on the inpainting problem [30], its high hardware requirements have prevented from
16

---

Inputs
liigned Images
UDIS
UDIS+
SRStitcher Fusion
Inpainting on Seam mask
Fusion image
Figure 10: Comparison of fusion effects of SRStitcher and other methods in the scene.
considering first option. Initially, we try to address the inpainting problem using Generative Adversarial Network (GAN)-based methods, such as AOTGAN [53]. Lama [44]. and FCFGAN [20].
However, experiments revealed several shortcomings. GAN-based models struggled with poor generalization ability, displayed unstable training outcomes, and demanded high-quality, well-labeled datasets. These factors complicated the pipeline and increased the workload, particularly in data labeling.
During the research bottleneck period, we found the concept of Soft-Inpainting proposed by Dif- ferential Diffusion [24], which involves blending parts of the image with the original by making slight modifications. This concept inspired us to adapt and extend it to our needs. We applied Soft-Inpainting near the seams and employed more intense Hard Hard-Inpon Irrrnddnpiniing the rectangling areas. With this idea, successfully implemented SRStitcher proposed in this paper.
Moreover, mentioned in the previous section, the strong generalization ability of large- scale diffu- sion models is also vital to implementing our method, which is the key to the ease of implementation of SRStitcher without training or task-specific data annotations.
Thus, adopting the diffusion model proved essential for addressing the challenges of this paper. provided more fitting solution than any other model available.
A.6 Why not focus on the registration stage
The registration stage is not the primary focus of this paper. We employ simplified homography estimation network from UDIS+ [35] to address the registration challenges. It is essential to clarify that this does not imply a devaluation of the registration stage. Registration has been the most extensively researched of the three stitching stages, with significant work devoted improving homography accuracy. However, perfect homography matrices that precisely align images do not exist for scenes that are non-planar involve cameras with different projection centers.
There are two mainstream methods to overcome these inherent limitations: the multi-houe caatii-hmoorrpphyy warp method [52] and the dense match method [47]. However, the multi-homoorapp method faces challenges in parallelization and integration within deep learning frameworks [35], while dense matching is generally slower and less robust.
These limitations inform our decision leverage the existing homography network and concentrate efforts on enhancing the robustness of the subsequent stages. Our experimental results show that
17

---

this decision has been beneficial: SRStitcher exhibits greater robustness to registration errors, thereby reducing the precision requirements of the registration stage.
B A brief survey of the image stitching pipeline
The image stitching pipeline can be divided into three stages, and the following subsections described based on each stage.
B.1 Image registration
Early image registration works [57, 8, 31] are limited by the feature extraction method, which often falters under conditions of rotation, scaling, and illumination changes. To solve the scale changes problem, AutoStitch [6] marks significant advancement by incorporating the Scale-invariant Feature Transform (SIFT) to extract scale-invariant features. However, this method is challenging to apply to situations with multiple depth layers. To address multi-depth layers condition, DHW [12] proposes a model that assumes the presence of two distinct planes within the image. applying different homography adjustments to each. However, the performance of this method can be severely impacted by the dynamics of camera movement. More recently. NIS [28] introduces the depth map integration to enhance registration accuracy. However, this method relies on accurately estimating depth maps, presenting its own implementation challenges. Yu et al. [51] develop technique using the epipolar displacement field to improve registration in scenes with significant parallax.
Feature-based methods have traditionally been the comeertoone of image registration techniques. However, these methods often need more geometric structure and in low-texture scenarios where traditional feature detection techniques are prone to failure.
In recent years, the advent of deep learning has revolutionized the field of image registration by enabling the extraction of rich semantic features through deep neural networks. Hoang et al [17] and Shi et al. [42] both propose the use of Convolutional Neural Networks (CNNs) to enhance feature representations in image stitching registration. Despite their progress, these approaches primarily use deep learning for feature enhancement rather than creating holistic learming-based framework. VFISNet [32] is the first complete learning-based framework for image stitching, but it is limited by its inability to handle images of arbitrary resolutions. EPISNet [36] is improved VFISNet by introducing flexible mechanism that supports the input of any image size through scalable image and homography adjustments. HomoGAN [18] introduces method based on the Generative Adversarial Network (GA)) to enhance the quality of homography estimations, representing a novel application of GANs in this field. Jiang et al. [22] integrates graph convolutional networks into the image stitching framework to boost the precision of multi-spectral image registration. LBHomo [21] introduces a semi-supervised approach to estimate homography more accurately in aureebaseline scenes by sequentially multiplying multiple intermediate homography. RHWF [7] introduces homography- guided image warping and Focus transformer into the recursive homography estimation framework to further refine homography estimation accuracy.
B.2 Image fusion
The earliest fusion method is weighted fusion [5], which requires high registration accuracy. If registration is imperfect or there is a color mismatch between the images, visible seams may appear, which can degrade image quality. APAP [52] introduces a smoothly varying projection field to enhance fusion accuracy. However, APAP tends to introduce severe perspective distortions in non- overlapping areas, limiting its applicability. Inspired by interactive digital photomontage [1], Gao et al. [13] propose the seam-based fusion method, which involves seam prediction stage to identify optimal seam lines between overlapping images. Although effective, it is notably time-consuming. Therefore, SEAGULL [29] proposes to improve the previous based methods by using estimated seams to guide local alignment optimization, enhancing seam quality and reducing processing time. However, it struggles with repetitive textures, where it still shows poor performance.
The methods above are all traditional fusion methods characterized by limited versatility and difficulty adapting to complex scenarios. To solve the defects of traditional solutions, UDIS [33] proposes a reconstruction-based model to improve the quality of the fused image. This method sometimes produces artifacts and strange blurs in overlapping areas despite its advances. Inspiration from
18

---

traditional Seam-based approaches, UDS++ [35] and Dseam [11] both use deep learning to refine the seam finding process. Though these fusion methods offer more robust and flexible solutions to improve the ability to handle complex scenarios, they cannot still correct the registration error effectively.
B.3 Image rectangling
Image rectangling is a relatively new area of computer vision with limited research to date. Prior to the advent of deep learning in this domain, traditional solutions such as those proposed by He al.[15] and Li et al. [25] used mesh-based warping techniques to address missing areas in images. DeepRectangling [34] represents the first deep barning-based approach in image rectangling. accompanied by baseline and a public dataset tailored for this specific task. The method continues to rely mesh-based warping but incorporates learning algorithms to enhance the fill quality and handle complex scenarios more effectively. While these methods are groundbreaking, they often change the global relative pixel positions, which could lead to suboptimal results, especially in cases with large missing areas, resulting in incomplete fills, A more recent method ReeDiffusion [56] that employs diffusion model to better solve the rectangling. Although this method provides sophisticated solution for achieving rectangularity, it is complex in design, requires long inference times, and requires significant computational resources for training, which limits its practical applicability.
Moreover, the current deep learning-basse -baeed image rectangling methods are all based on the DIR-D dataset [34]. DIR-D dataset is strong assumption dataset, which assumes that some challenging scenes are excluded and that the image registration and fusion are flawless. Therefore, current method do not optimize for the robustness of registration and fusion errors, leading to the error propagation problem illustrated in Figure 1 3.
In addition, there are some works on filling irregular edges in other fields [40, 27, 23]. However, due sigmificant differences between these fields and the varying shapes of missing regions, these studies are not easily applicable to the rectangling task in the current image stitching pipeline.
B.4 Other similar work
Several prior studies have attempted to integrate fusion and rectangling stages. For instance, RDISNet [55] claims create an end-to-end image stitching framework that combines all three stages. However, our analysis of its network structure and experimental results suggests that RDISNet may struggle with large parallax scenes, and noticeable noise in its stitching outputs adversely impacts image quality. Chen et al. [10] propose a diffusion based method to address both fusion and rectangling tasks. However, this method necessitates meticulously prepared datasets and retraining of the diffusion model, leading to substantial cost demands. In summary, while these existing methods are innovative, they present significant limitations with low reproducibility. As a result, we exclude them from our baseline comparisons.
Imputs
UDIS+SD1.5
UDISplus+SD1.5 HIQA: 52.8
UDIS+SD2
UDISpuu+5D2
SRSütcerr
HIQA: 5423
HIQA: 51.76
HIQA: 53.07
HIQA: 53.30
CLIPIQA: 17.59
CLIPIQA: 22,67
CLIPIQA: 16552
CLIPIQA: 20.18
CLIPIQA: 19:47
Figure 11: Examples of unreasonable NR-IQA scores.
19

---

C Detail of metrics
C.1 NR-IQA metric
(I) NR-IQA metric settings
HIQA. HIQA [43] designed for the 'wild' image. HIQA is particularly suitable for evaluating the predominantly outdoor images in the UDIS-D dataset, making it an ideal choice for our analysis. We implement this metric based on the public source code with default parameters.
CLIPIQA. CLIPIQA [49] based on the Contrastive Language-Iman Image Pre-training (CLIP) models, which allows for adaptable evaluations across different datasets. We use IQA-PyTorch Tool [9] to implement this metric with prompts l'nature image', 'stitched image"] to evaluate whether the stitched images appear more natural.
(2) Limitation of NR-IQA metrics
Through our analysis of the results obtained by the NR-IQA metrics HIQA and CLIPIQA, we find discrepancy between these metries and human sensory preferences for image quality, Sometimes, our method produces visually higher quality and authenticity images, as shown in Figure 11, but the scores assigned by these NR-IQA metrics are counterintuitively low.
We believe that this problem arises from a mismatch between the training datasets and the unique challenges of image stitching. Metries such HIQA and CLIPIQA, are trained on IQA-specific datasets such as KonIQ- 10k [19] and Live-iWT [14], which focus primarily on image distortions such white noise, low-light noise, and JPEG compression artifacts. These types of distortions are very different from those encountered in image stitching, such as artifacts and incongruous inpainting content. result, models trained on such data may struggle to perfectly reflect the true perceptual quality of stitched images.
While the NR-IQA metric may exhibit inaccuracies in certain scenarios, it typically indicates superior stitching images in most cases, which why we have chosen to use it. However, we acknowledge the inherent limitations of current NR-IQA metries, especially their inability to effectively capture the nuanced aspects of image quality improvements in stitching. Therefore, we do not employ these metries in the ablation experiments, And, also conduct user study supplementary proof.
Inpuls
Fusion
UDISplus +DD Result
SRSticher Result
CCS,
Eor diee
Boskn Audlw
Compare similarity
penania
slndow
Bnitt Bing
CCS, 6130
B9168
CCS
Compare the similaatty
Ion swidow
torofobkeee DAREEA
boding of
CCS, 25.98
CCS, #8882
Figure 12: Visual presentation of the CCS metric. Inside the blue box is the text extracted using CoCa(.).
20

---

C.2 CCS metric
We design the CCS metric measure the image content consistency before and after stitching, which is based on the idea of an image-to-tex model. We first extract text information based on the image through the (-Ca(-) model, then map the into the embedding space through the Bert(.) model, and finally measure the cosine similarity of the embedding before and after stitching to calculate the CCS. We give an intuitive evaluation of this metric in Figure 12.
In defining CCS,,, we set n=4, a value we believe is most effective. Using ==1 loses local meaning. For 29, scenes with large areas missing (as illustrated the second scene of Figure 2), parts of the local image may lack semantic content. This absence hampers the extraction of text information, thus undermining the credibility of the CCS metric. Additionally, excessively small input images can negatively impact the performance of the CoCa(-) model.
We also give the souree code for the CCS metric implementation in the Supplementary Material, as shown in metrics/ccs.pp.
D Additional experiments and results
D.1 Evaluation results of the example in Figure 2 on all baselines
To ensure the presentation effect, all baseline results are not provided in the main paper Figure 2. Here, offer the complete results, shown in Figure 13.
Our results in the figure can also be reproduced using the source code and provided data in the Supplementary Material.
21

---

UDIS+DR
UDISpha+DR
UDIS + .ama
UDISphus+Lamm
Inpul Images
Seeattcher (Ours)
UDIS+SD1.5
UDDSp
UDIS+5D2
UDISpkssSSD
UDIS+DR
UDISplas4DR
UDIS-Lama
UDISplus-Lame
Input
Images
SeSüicir
(Ours)
UDIS+801.5
UDDSSplssSSD1.
UDIS+5D2
UDDSplus+SDD
UDIS+DR
UDDSSla=+DR
UDIS-LLama
UDISplusLLame
Input Images
SRSWitherr
(Ours)
UDIS-SDL.
UDD5plus+5D15
UDIS+SD2
UDDSplus+D2
UDIS
UD DD
UDIS +amma
UDISplas+Lamm
Input Images
SRSWiicee (Ouns)
UDDS+5D115
UDISplus+SDDL
UDIS+602
UDISplus-SD2
Figure 13: Evaluation results of the example in Figure 2 on all baselines.
22

---

D.2 Additional hyper- parameers study results
(I) The impact of Kg
The Kg in in Eq. 10 and Eq. 11 determines the intensity of the distance transform applied within the weighted masks. Despite the potential variations available the type of distance transform (e.g.. L1 vs. 1.2) and the kernel size, our empirical observations show that these modifications do not significantly impact the stitching results. Therefore, we set commonly used value using an L.2 distance and a kernel size of 3.
(2) The impact of seed
We show the impacts of different seeds in Figure 14. Our method produces more stable results with high quality. With different random seeds, (SD2) [3] produce completely different abnormal contents. In contrast, proposed method consistently demonstrates remarkable stability.
--i--------iiiiniinggg-- (SD1.5) [41] and
Inguts
Seed 1
Seed 2
Seed 3
Seed 4
SRStither
USD5
UDISphu+6D2
Figure 14: Ablation study the seed.
(3) The guidance scale and inference step
The guidance scale and inference step are two classical parameters in diffusion models. The impact of adjusting these parameters has been extensively validated by previous research [48]. Consequently. this paper does not delve into selecting their values but instead adopts two commonly used settings: 7.5 and 50.
D.3
Qualitative evaluation results of SRStitcher variants
This section presents the qualitative evaluation comparing SRStitcher variants based on various diffusion models. The version based on the Stable Diffusion model [2] is designated as SRStitcher- S. Additionally, the implementation utilizing the Stable Diffusion 2 Unclip model [4] is termed SRStitcher-UU U. Finally, the implementation with Controlnet Inpainting model [54] defined as SRStitcher-C.
The test results are shown in the Figure 15. The the best performance, which is the primary reason for selecting it as our base model. The Stable- 2--ssionn2---nnliin fine-tuned model based on Diffl--Dif V2.1, chosen an attempt to leverage its CLIP image embedding functionality. However, the structural integrity of the results
model exhibits
23

---

Inguts
SRStitcher-S
SRStitcer-U
SRStitcee-CC
SRStitcher
NH
Figure 15: Qualitative evaluation results SRStitcher variants.
generated by this model is significantly inferior to those of the other two models, likely due to compromises introduced during fine-tuning.
Notably, the performance of the SRStitcher-C based on ControlNet has exceeded our expectations. While the model does exhibit more pronounced issue with local blurring, it demonstrates exceptional capability in preserving the original image information. In future work, should model fine-tuning be employed to further optimize the stitching effect, we may consider beginning our enhancements with the ControlNet model.
We provide the source code for all SRStitcher variants in the Supplementary Material. Therefore, we do describe implementation of them here.
D.4 Generalization on other datasets
In addition to the UIDS-D D dataset, there are traditional datasets in the field of image stitching, such as APAPdataset [52] and REWdataset [26]. However, these datasets are very small, containing only dozens of images, which reduces their usefulness for meaningful comparative experiments. To demonstrate the effectiveness of our method on broader spectrum of data, we present some experimental results APAPdataset and REWdataset. illustrated in Figure 16.
In addition, Figure 17 shows the experimental results comparing SRStither and other methods on the traditional datasets.
D.5 Examples of local blurring
Here, we present examples local blur and compare them with other baselines, as illustrated in Figure 18. We contend that occasional local blur is a tolerable side effect of our scheme, especially when weighed against the generation of significant anomalous content seen in other models. Future research could potentially address this issuc by fine-tuning the model.

---

Figure 16: More results on traditional datasets APAPdataset [52] and REWdataset [26] by SRStitcher.
25

---

Trputs
UDISpla=DR
UDSS
UDSSpuu5522
SRSHither
APAP dataset
REW dataset
Figure 17: Comparison results on traditional datasets APAPdataset [52] and REWdataset [26].
Irpus
UDIS+SDL5
UDIS+SD2
SDDS
UDISphs4SD2
SeStither
Figure 18: Examples of local blurring.
D.6 Speed
Our solution requires only a single inference step, making it significantly faster than more complex models that require two inference steps, such UDIS+SDI.5 UDISplus+SD2. Although our method is slightly slower compared to UDIS+DR to UDISplus+Lama, our experimental results demonstrate that substantially outperforms these methods regarding stitched image quality, ro- bustness, and generalization. Given these advantages, the minor sacrifice in speed is justifiable. In particular, even without acceleration optimizations like TensorRT [37], our method achieves an average processing speed of 27 it/s on an NVIDIA 4090 GPU, which is sufficient for real-time performance.
E Broader impact
This paper aims to introduce novel image stitching pipeline design that integrates large-scale models into the image stitching process. Unlike previous diffusion model-based image stitching methods, our method does not require training or task-specific supervised datasets. This significantly lowers the implementation threshold, facilitating broader adoption of the method and encouraging more researchers to contribute and enhance this work. One potential negative impact of our method is its
26

---

ability to make minor modifications to stitched images. This capability could be misused in visual fraud tasks, particularly in the context of surveillance videos.
F
Reproducibility
We have made significant efforts to ensure the reproducibility of our method. The code and details of the SRStitcher, Stitcher variants, and CCS metric are all uploaded in the Supplementary Material.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are illustrated in section 5 Discussion and conclu-
sion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [Yes]
Justification:
We provide proof' of the proposed model in the main paper and Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We give detailed explanation of all the experimental results in the section Experiments and provide supplementary notes in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
29

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The source code for this paper has been open sourced and can accessed through the link provided in the Abstract. And, we use the publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (uttps://nips.co
as public/guin heellii/ggiies/Caaee wwweee//eooeebeeieeeeeee as sisnoolicy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (httpa:
/ //nips.coo wttpccccpppccccccccc. geides/ CodeSubmi ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We give all experimental details in the section 4 Experiments and provide supplementary notes in the Appendix D.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the standard deviation under different seeds in Table 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
30

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We give all implement details including computer resources in the Experimen- tal setup of section 4 Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:r/pps.ccppppceeeeeet....e. ines?
Answer: [Yes]
Justification:
Our research respects the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential positive societal impacts and negative societal impacts of the work in the Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
31

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our method does not involve model training and uses publicly available datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have explicitly stated the data and model sources used in the main text. All the models and data used are open source and shared.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi th code. . cww.coo//a//a// aattettsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The source code for this paper has been open sourced and can be accessed through the link provided in the Abstract, with detailed instructions for use.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our user study included only 20 volunteer participants and they only rated the image quality. This test should not fall within the scope of crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing or research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "flCBNmlD58", "reviewer": "Reviewer_GE1w", "summary": "This paper tried to integrate the fusion and rectangling stages in image stitching into a unified model. More concretely, a special fusion, a rectanlging step, and a mask-guided diffusion model are gathered to implement stitching-customized image inpainting, especially for the irregular boundaries. It is worth mentioning that, the inpainting model uses the pre-trained model and requires no more fine-tuning. To evaluate the proposed method, the authors designed a quantitative metric named CCS and conducted extensive experiments.", "questions": "My concern lies in my second weakness. Will this inpainting model for irregular boundaries be meaningful? I have another idea for the inpainting model of image stitching. The inpainting model should not concentrate on the boundaries but on the regions where the artifacts and distortion are produced. Let’s put it in this way. The registration stage may introduce artifacts or distortion. So can we eliminate these issues through an inpainting model? It may be more meaningful to locate these regions, mask them, and then inpaint them. The inpainting process can be implemented with the guidance of original contents, thus contributing to a reliable completion model. This is just a simple discussion. But from my perspective, I cannot figure out the meaning of the proposed inpainting model.", "limitations": "No limitation is mentioned in the manuscript.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The concept of integrating multiple stages of image stitching into a single stage is novel and promising. \n2. The experiments are abundant and convincing. \n3. The authors leverage a pre-trained model to implement stitching-customized inpainting without any extra training.", "weaknesses": "1. The so-called unified model consists of several steps including a specially-designed fusion step, a rectangling step, and a inpainting step. I don’t think this multi-step design can be regarded as a unified model. \n2. I am skeptical of whether the inpainting model is meaningful to image stitching. As claimed in the DeepRectangling paper, they abandoned the inpainting model because they think it may introduce some contents that are far from the reality. The manuscript seems to ignore the problem. And I do not think the proposed method can address this issue in the so-called unified inpainting model. \n3. The inpainting results are still not perfect, as illustrated in the image boundaries of Fig. 14.\n4. The fusion step of Eq. 4 is special. What’s the motivation for that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for raising these issues and your comments. Please see below for the responses.\n\n## Weaknesses\n\nWe start with weakness 2, which is most concerned by the reviewer, and then discuss weaknesses 1, 3 and 4.\n\n **W2: DeepRectangling's experimental results of inpainting models are outdated.** \n\nThe DeepRectangling paper claimed that: \n\n> Nevertheless, there is currently no work to design a mask for irregular boundaries in image stitching, and even SOTA completion works [26, 28] show unsatisfying performance (Fig.1d) when processing the stitched images.\n\nHowever, it is important to note that DeepRectangling paper was published in 2022, and its references 26[A] and 28[B], were completed in 2021 and 2019, respectively. Notably, since 2022, there has been substantial progress in diffusion-based inpainting methods[C, D, E], which demonstrate far superior performance in terms of image quality and semantic coherence. Therefore,  [A, B] in DeepRectangling paper are not SOTA now.\n\nAlso, a recent work[F], published in June 2024 (and therefore not cited in this paper), has begun exploring inpainting-based methods to address the rectangling problem, yielding promising results. Thus, the potential and value of the inpainting-based method for rectangling problem should not be denied based only on the results in DeepRectangling paper.\n\nTo address the unrealistic content problem mentioned in the DeepRectangling paper, we propose an innovative solution, based on the weighted initial mask and coarse rectangling, that effectively controls abnormal content. The unrealistic content problem has been successfully resolved in our method.\n\n**W1: The unified model is clearly defined in the paper by Eq. 8.**\n\nIn Section 3.1 Unified inpainting model,  we first define the image fusion problem(Eq. 4), and then define the image rectangling problem(Eq. 6). Finally, we explain how these two originally separate problems can be abstracted into one unified problem that can be solved by one single learning-based model (Eq. 8). This is not a \"multi-step design\", but rather a conceptual process of integrating two individual problems into a unified framework.\n\n**W3: No method can perfectly stitch the images in Fig. 14.**\n\nIn the newly added PDF in Global Response, we provide the Rebuttal Fig. 1 to show the performance comparison of our method with UDISplus+DR, UDISplus+SD1.5, and UDISplus+SD2 under the scenes in Fig. 14.  \n\nThe results show that DeepRectangling causes distortion in the overall structure of the image, while both SD1.5 and SD2 introduce content in the upper-left corner of the image that is not present in the original. In contrast, our method exhibits only minor issues with edge clarity.\n\nLocal blurring is a limitation of our method, which is discussed in the 5 Discussion and conclusion of the main paper (Lines 249-254). This blurring results from a trade-off to reduce the likelihood of generating abnormal content. The presented limitations provide directions and possibilities for future research. So, we do not shy away from our imperfections in Fig. 14, they do not imply that our method is inferior to other methods.\n\n**W4: The motivations of Eq. 4 are in the text directly above this equation and Introduction section.**\n\nThe text directly above Eq. 4 (Lines 90-93) : *\"Precisely, as shown in Eq. 2, the distortion degree of $I_{wl}(x, y)$ is relatively low because it involves only minor warping based on $\\texttt{I}$. This means that even in the presence of registration errors, $I_{wl}(x, y)$ does not introduce large-scale distortions. Therefore, we propose to construct a coarse fusion image $I_{CF} (x, y)$ via Eq. 4.\"*\n\nAlso, in the Introduction section (Lines 44-46) : *\"We propose to reformulate the fusion problem by overlaying the less distorted aligned image over the more distorted one, and inpainting the seam area between the images to correct the inappropriate image content.\"*\n\n## Questions\n\n**The \"another idea\" is the core idea of our paper.**\n\nIn the Questions, the reviewer proposes an \"another idea\". We believe it is a super excellent idea, because this \"another idea\" is highly consistent with the core idea of our paper introduced in the Introduction section (Lines 41-44) : *\"Therefore, we reconsider the problem definition of the fusion challenge and hypothesize that: By determining the appropriate modification region and introducing an inpainting model with strong generalization ability, the abnormal image content caused by registration error can be effectively corrected.\"*\n\nWe have detailed how our core idea is implemented in the paper. Therefore, this so-called \"another idea\" proposed by the reviewer has been realized in our paper.\n\n**Overall, our unified inpainting model is meaningful.**\n\n1.  Our response to \"weakness2\" has proved that the results from DeepRectangling do not indicate that inpainting-based methods are ineffective for rectangling problem.\n\n2. The so-called \"another idea\" proposed by the reviewer has been implemented in our paper.\n\n3. Experiments  (Fig. 2 and Rebuttal Fig. 1)  show that our method significantly outperforms DeepRectangling in the ability to preserve the original structure of the fusion image.\n\nBased on the aforementioned arguments, we are confident that the unified inpainting model is effective and has a solid theoretical foundation and significant experimental results.\n\n## Limitations\n\nWe mentioned the limitations in section \"5 Discussion and conclusion\" of the manuscript.\n\n**references**\n\n[A] Resolution-robust large mask inpainting with fourier convolutions, 2021.\n\n[B] Boundless: Generative adversarial networks for image extension, 2019.\n\n[C] Repaint: Inpainting using denoising diffusion probabilistic models, 2022.\n\n[D] Palette: Image-to-image diffusion models, 2022.\n\n[E] Smartbrush: Text and shape guided object inpainting with diffusion model, 2023.\n\n[F] RectanglingGAN: Deep rectangling for stitched image via image inpainting, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFDenbfvIk", "reviewer": "Reviewer_m57w", "summary": "The paper introduces SRStitcher, a novel method that integrates the fusion and rectangling stages of the image stitching pipeline into a unified inpainting model using a pre-trained large-scale diffusion model, eliminating the need for additional training. This approach addresses the issue of error propagation in traditional pipelines, offering a streamlined and robust solution. Strengths of the method include improved performance in image quality and content consistency, as well as robustness to registration errors. The experimental results are extensive, providing both quantitative and qualitative evidence of SRStitcher's superiority over existing state-of-the-art methods.", "questions": "- Technical Novelty: How does SRStitcher fundamentally advance the field of image stitching beyond merely integrating existing fusion and rectangling processes into a unified model? Can the authors provide more evidence of novel technical contributions?\n\n- Performance Comparison: While the paper claims superior performance, how does SRStitcher specifically outperform existing state-of-the-art methods in scenarios with extreme registration errors(large parallax)? Are there any edge cases where SRStitcher struggles compared to traditional methods?", "limitations": "The authors have adequately addressed the limitations of their work and there appear to be no significant issues with the broader societal impacts. They have demonstrated transparency and responsibility in discussing the constraints and potential improvements of SRStitcher. Overall, their approach seems robust and well-considered, with no apparent concerns regarding its application or societal implications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "SRStitcher's primary strength lies in its innovative integration of the fusion and rectangling stages into a unified inpainting model, which addresses the long-standing issue of error propagation in traditional image stitching pipelines. By leveraging a pre-trained large-scale diffusion model, SRStitcher eliminates the need for stage-specific training, thereby simplifying the pipeline and enhancing its robustness. This approach ensures superior performance in handling registration errors, which are typically propagated and amplified in multi-stage pipelines. Additionally, the use of weighted masks to guide the inpainting process allows for precise control over inpainting intensity, significantly improving image quality and content consistency. These advancements address the limitations of existing methods, which often struggle with the independent optimization of each stage and the associated parameter tuning challenges. The extensive experimental results, including both quantitative metrics and qualitative assessments, robustly demonstrate SRStitcher's superiority over state-of-the-art methods, showcasing its ability to produce high-quality stitched images with greater stability and fewer artifacts.", "weaknesses": "While SRStitcher simplifies the image stitching pipeline by integrating the fusion and rectangling stages into a unified inpainting model, its technical novelty is questionable. The approach primarily combines existing technologies—pre-trained diffusion models and weighted masks—rather than introducing fundamentally new methodologies. The integration, while effective, does not inherently surpass the capabilities of current state-of-the-art techniques used separately for fusion and rectangling. For SRStitcher to be considered truly novel, it should achieve technical goals that were unattainable with the two processes handled independently. As it stands, the method appears to be more of a consolidation of existing practices rather than a groundbreaking innovation, merely reorganizing the workflow without providing substantial new capabilities or overcoming significant limitations of the previous approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and careful reading. Please see our responses below.\n\n## Weaknesses\n\nWe would like to clarify that SRStitcher achieves technical goals that were unattainable with the two processes handled independently, especially in image rectangling. \n\n**1. SRStitcher improves the robustness of the fusion technique**\n\nThe image fusion method can be broadly categorized into two types: reconstruction-based method and seam-based fusion method. The SOTA  reconstruction-based model is UDIS[A], which has been observed to introduce blurring parallax regions, particularly in scenes with large parallax. This issue is thoroughly documented in UDIS++[B]. To address this, the SOTA seam-based model UDIS++ attempts to improve the fusion image quality by the soft seam. However, in scenarios with registration errors between images, a perfect seam does not exist. In such scenarios, UDIS++ forces the image distortion to \"create\" a perfect seam, which results in the distortion of the pillar as illustrated in Figure 1② of our paper.\n\nTo address the aforementioned issues, we propose a new solution for image fusion that uses the inpainting-based method to smooth the image. In contrast to reconstruction-based methods, SRStitcher does not introduce blurred regions when handling large parallax scenes. Furthermore, in comparison to seam-based methods, SRStitcher does not depend on the existence of perfect seams in the image. Consequently, SRStitcher exhibits greater robustness in dealing with registration errors compared to previous methods.\n\n**2. SRStitcher  is the first rectangling solution that does not require supervised data and has higher generalization performance**\n\nSRStitcher represents a major breakthrough for the rectangling problem. Existing rectangling methods [C, D, E] are all supervised learning methods and require labeled datasets. And the only labeled dataset for image rectangling is DIR-D[C]. DIR-D is an ideal dataset that excluded some challenging scenes during its production process, and its scale is relatively small. Models trained on DIR-D have limited generalization ability, as evidenced by zero-shot experimental results of DeepRectangling.\n\nOur solution is the first rectangling method that does not require supervised data. It generalizes much better than the SOTA rectangling model DeepRectangling,  due to the prior knowledge from large-scale pre-trained models.\n\nWe appreciate the reviewer's comments, which helped us identify the areas for improvement in the writing of our contributions section. We modify the contributions in the Introduction section as follows:\n\n> The main contributions of this paper are: \n(1) We propose SRStitcher, which reformulates the problem definitions of the fusion and rectangling stages to construct a more streamlined and robust image stitching pipeline. \n(2) SRStitcher is the first to introduce the concept of inpainting to address the image fusion problem. It incorporates prior knowledge from large-scale pre-trained models into the image stitching pipeline, enhancing the robustness of image fusion against registration errors.\n(3) Without additional fine-tuning or supervision, SRStitcher improves the generalization of the rectangling method in the zero-shot scenario, opening up new possibilities for unsupervised image rectangling research.\n(4) We conduct extensive experiments to verify the interpretability and generalization of the proposed unified model. The results show that SRStitcher significantly outperforms the state-of-the-art methods in both quantitative and qualitative evaluations.\n\n## Questions\n\n**Q1: SRStitcher has three novel technical contributions**\n\n1. SRStitcher first proposed the inpainting-based fusion method, a new thinking direction.\n\n2. SRStitcher first incorporates prior knowledge from large-scale pre-trained models into the image stitching pipeline.\n\n3. Weighted masks are not existing techniques, we design them to get a more refined inpainting control.\n\nThanks to the reviewer' suggestions, we have revised the description in the Introduction section to emphasize the significant contributions of our method in both fusion and rectangling techniques.\n\n**Q2: We provide a case for an extreme registration error scenario in the Rebuttal Fig. 2**\n\nPlease see the newly added PDF in Global Response for Rebuttal Fig. 2.\n\nIn this scene, due to the large parallax and complex floor tile texture, other comparison fusion methods exhibit noticeable misalignment and artifacts. SRStitcher addresses these issues using an inpainting-based method. Although SRStitcher cannot guarantee perfect results in the presence of such significant registration errors, it markedly outperforms previous methods in terms of tile continuity.\n\nAccording to the definition of $K_s$ (Lines 187), our method adaptively increases the width of $M_{seam}(x,y)$ in large parallax scenarios. Therefore, our method performs effectively in large parallax.\n\nEdge cases exist. Specifically, in \"small parallax scenes with substantial color differences between the stitched images\", our method may exhibit more pronounced stitching seams compared to existing methods such as UDIS and UDIS++.  Because, the current hyper-parameter settings are relatively conservative, resulting in a relatively small value of $K_s$ in small parallax scenes. The width of $M_{seam}(x,y)$ is insufficient to provide enough space for color difference smoothing. We think that this issue can be mitigated by pre-calculating the color differences and design a more flexible hyper-parameter setting method.\n\n**references**\n\n[A] Unsupervised deep image stitching: Reconstructing stitched features to images, 2021.\n\n[B] Parallax-tolerant unsupervised deep image stitching, 2023.\n\n[C] Deep rectangling for image stitching: A learning baseline, 2022.\n\n[D] Recdiffusion: Rectangling for image stitching with diffusion models, 2024.\n\n[E] RectanglingGAN: Deep rectangling for stitched image via image inpainting, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "66vSyAxuNa", "reviewer": "Reviewer_pLdt", "summary": "This paper proposes an image stitching algorithm that unifies fusion and rectanguling stages of a conventional pipeline with an image inpainting diffusion model applied with a progressive reverse process guided by weighted masks. Their reverse algorithm progressively inpaints seam regions by gradually increasing the size of soft mask holes, while keep outpainting rectanguling regions with the same hard mask holes. The proposed work is compared with previous works by keeping the registration and fusion stages with UDIS/UDIS++ while altering with various models for a rectangling stage.", "questions": "Comparing with a case of a single reverse process guided by a fixed mask would make the paper more convincing.\n\nComparing with a case that applies progressive mask dilation of the rectanguling mask holes would be useful.", "limitations": "Technical limitations are properly addressed. No serious potential negative societal impact is expected.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In their setup of experiments, theirs generally showed higher quality quantitatively and quantitatively than compared methods.\n\nThe proposed reverse process results in a high quality generation, suitable to inpaint and outpaint the seam and rectangling regions naturally.", "weaknesses": "It is unclear if the repeated reverse process with gradually dilating mask holes is really beneficial than a single reverse process guided by a fixed mask. The repeated application of the re-noising + de-noising step makes the algorithm much slower.\n\nConsidering the seam regions are usually narrow while the rectanguling regions could be quite wide, the progressive soft mask dilation during the progressive reverse process may be more useful with the rectanguling masks than the seam masks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review of our work. Please see below for the responses.\n\n## Weaknesses\n\nPlease see the newly added PDF in Global Response. We have provided the Rebuttal Fig. 3 and Rebuttal Fig. 4 to explain your confusion.\n\n**W1: Gradually dilating mask holes in seam regions are based on content preservation consideration.**\n\nImagine that there is a painter, and this painter has an unfinished painting(Coarse fusion image). There are two things the painter needs to finish on this painting:\n\n1. The painter is not satisfied with the finished part of the painting and needs to modify it (seam regions).\n2. There are still parts of the painting that are not painted (rectanguling regions).\n\nLet us do the first thing: if this painter smears out everything in the seam region at once (fixed mask), the modified image differs significantly from the original content, see Rebuttal Fig. 3(a) red box.\n\nSo this painter needs to modify it slowly, a little at each time (step $t$), so that the painter can refer to what is still in the seam regions to ensure that the modified content still has the original content, see Rebuttal Fig. 3(b) red box. This is why we use Gradually dilating mask holes in seam regions.\n\n**W2: Using gradually dilating mask holes in rectangling regions is disappointing.**\n\nLet us do the second thing: the painter wants to paint the rectangling regions using the same strategy as seam regions. However, there is nothing in this region. When the painter starts drawing from a small mask, the painter finds the information near the mask is insufficient.  \n\nOne side of the mask has the content, and the other has no content (Rebuttal Fig. 4(b)). Alternatively, one side of the mask has no content, and the other is directly out of the image (Rebuttal Fig. 4(a)). \n\nThe painter must smooth the contents of both sides of the mask with each step, and smoothing the image without content introduces blurring noise.\n\nIf the painter were a human, he would know that he only needed to consider the pixels with content. Unfortunately, the painter is a robot whose execution program is inflexible.\n\nTherefore, in rectangling regions, we use fixed mask. One side of the mask has content, and the other is directly out of the image.  Our smart robot painter knows that there is no need for smoothing the content beyond the image.\n\n**Speed: Our method does not increase the inference time.**\n\nOur design does not increase the computational complexity of the model, and the re-noising + de-noising step is the standard process of the diffusion model. The original diffusion model still needs multi-step sampling without our method [A]. We achieve fine-grained control for local adjustment of different image regions by using weighted masks in the reverse process. This approach keeps the basic computational structure of the model the same. \n\nThus, the inference time of the model with our method remains relatively consistent compared to the original model.\n\n\n## Questions\n\n**Q1: We provide a case of the reverse process guided by a fixed mask in the new pdf.**\n\nIt is shown in Rebuttal Figure 3. There is no single reverse process. The reverse process of the diffusion model is multi-step [A].\n\n**Q2: We provide a case that applies progressive mask dilation of the rectanguling mask holes in the new pdf.**\n\nIt is shown in Rebuttal Figure 4.\n\n**references**\n\n[A] Denoising diffusion probabilistic models, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eRujla8zUv", "reviewer": "Reviewer_yn5D", "summary": "This work proposes to integrate the fusion and rectangling of image stitching into a unified inpainting model. In particular, the weighted masks are designed to guide the reverse process in a pre-trained large-scale diffusion model, which implements this integrated inpainting task in a single inference. Extensive experiments demonstrate the interpretability and generalization capabilities of the proposed unified model.", "questions": "- Can the proposed unified model be extended into other research areas?", "limitations": "- The proposed unified inpainting model seems to be impractical compared to previous reconstruction-based solutions.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The motivation for reconstructing the image stitching pipeline into a unified model is sound and clear.\n+ Some discussions on image stitching are insightful. For example, in the Introduction, the authors claimed that \"To address the error propagation problem, we identify image fusion as the key point for improvement\". I agree it is an accurate point to inspire this work.\n+ The reviews of previous methods are detailed and comprehensive (in Tab. 1 and Section B in the Appendix), which clearly shows their limitations and potential improvements.", "weaknesses": "- While this work is well-motivated, the proposed contributions seem to be weak for me. For instance, the authors proposed a weighted mask to guide the reverse process in the diffusion model. However, the current presentation regarding this contribution looks like an experimental trick rather than a technical proposal.\n- The contribution of the proposed unified inpainting model (Section 3.1) is also expected to be highlighted. It is still ambiguous how the fusion part and rectangling part are merged into a unified model. I believe this module is much more important than applying cutting-edge generative models.\n- This work leverages a **generation** view to address the **reconstruction** problem. Such an interesting trend can be noticed in the recent works from other research regions as well. However, how to balance the generation and reconstruction in the image stitching problem remains unclear. I believe the users may prefer accurate reconstruction instead of introducing novel content (sometimes artifacts). Moreover, some previous works also proposed to fill the irregular boundaries of images or holes by warping with generative models, such as \"Free View Synthesis (ECCV'2020)\", \"Towards complete scene and regular shape for distortion rectification by curve-aware extrapolation (ICCV'2021)\", \"iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis (SIGGRAPH Asia)\", etc. More discussions are expected to be provided.\n- There are some typos in the paper: Line 49: \"Therefore, We question\". The authors are suggested to further polish this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful assessment of our work. Please see our responses below.\n\n## Weaknesses\n\nWe start with weakness 3, and then discuss weaknesses 1, 2 and 4.\n\n**W3: The reconstruction-based image fusion model has significant limitations and has fallen into the development bottleneck.**\n\n*Users may prefer accurate reconstruction, but only if the reconstruction is truly accurate.*\n\nFirst reconstruction-based image fusion model is a UNet-like model proposed by VFISnet [A] in 2020. The subsequent classical learning-based image stitching works [B, C, D] generally follow this model structure, and there are few structural breakthrough innovations. \n\nNow, the SOTA reconstruction-based model is UDIS [C]. However, it is mentioned in UDIS++ that UDIS has significant defects, and UDIS may introduce blurred parallax regions in the fusion image with large parallax. Therefore, UDIS++[D] focused on finding a suitable soft seam and ultimately gave up improving the reconstruction-based fusion model. \n\nBased on the above discussion, in image stitching, the reconstruction-based image fusion model research has not been a fundamental breakthrough for many years, and its SOTA method UDIS still has apparent defects. Therefore, we propose another idea to explore a new direction of image fusion based on the generative model. \n\nA single research perspective limits the breadth of the development of a research field. We believe our method provides a new research perspective for the fusion problem in image stitching.\n\n**Related work of irregular boundaries** Thank you for reminding us of the work related to irregular boundaries. We will add the discussion about these works in the related work.\n\n**W1: The construction of the weighted mask has the theoretical basis.**\n\nDue to the space limitation of the main paper, we put the detailed theoretical basis and design principle in Appendix A \"More Details of SRStitcher\".\n\nTo address this weakness, we thoroughly rewrote and supplemented Weighted masks parts in Section 3.2 Weighted mask guided reverse process in the new manuscript, as shown in the gray box.\n\nWeighted inpainting mask is inspired by the suffix principle [E]. This principle allows for the customization of the variation for each pixel or image region during the reverse process. We extend this concept by applying it to the diffusion-based Inpainting model and incorporating the requirements of image stitching task to construct weighted masks.\n\n> The weighted inpainting mask, as described in Eq. 11, is inspired by the suffix principle [E]. During the reverse process, weighted inpainting mask is mapped into multiple sub-masks to define the modified regions at each step $t$. ....This gradual modification method facilitates a more seamless blending of the inpainting content with the original image content.\n\nHowever, the suffix principle is a scheme to smooth the transition between the inpainting and non-inpainting areas, which cannot perceive the original content. Inspired by the attention map, we design the weighted initial mask, which assigns different fidelity to each pixel in the original image.\n\n> The weighted initial mask assigns different fidelity levels to each pixel of the fusion image, determining how much to modify each pixel based on its fidelity during the reverse process. The formula of weighted initial mask is given by Eq. 10, which is composed of two parts. The left part determines the fidelity levels of pixels in $M_{seam}(x,y)$ region, and the right part determines the fidelity levels of pixels in $M_{content}(x,y)$ region.\n\nWeighted masks are a further extension of the suffix principle, which discusses how to achieve more fine-grained and content-fidelity inpainting control and provides some technical contributions to image stitching and the control of diffusion-based inpainting process.\n\n**W2: We believe that the challenge of unified models is not in definition but in implementation.**\n\nOur unified model definition is straightforward and intuitive (Eq. 8). The real difficulty lies in finding a suitable model to implement this unified problem efficiently. The image stitching is a field where data is extremely scarce. All datasets used in this paper are unlabeled, which makes it difficult to train the model. \n\n*So, our first task is not to design a theoretically elegant model, but to design a practically feasible model.*\n\nWithout prior knowledge provided by large-scale generative model, corrected registration errors and unsupervised rectangling are a struggle to succeed. Therefore, in the description of the main paper, we emphasize how to be compatible with the structure of existing large-scale generative models in the unified problem definition. In doing so, our method can introduce large-scale models into image stitching, which opens a new research direction for image stitching with extreme data shortage.\n\n**W4: Thank you for your careful check. We have re-checked the paper and revised all typos.**\n\n## Questions\n\nOur unified model is specifically designed for image stitching pipelines. Given that we have yet to conduct a systematic study of the application of the model to other domains, we are careful to avoid asserting its universality in other domains. The core of our technique lies in its fine processing ability of image content to correct abnormal content while preserving the original content, which is potentially valuable for applications in multiple low-level vision domains, such as image restoration, artifact removal, and low-light enhancement.\n\n## Limitations\n\nAnswered, see W3.\n\n**references**\n\n[A] A view-free image stitching network based on global homography, 2020.\n\n[B] Learning edge-preserved image stitching from multi-scale deep homography, 2022.\n\n[C] Unsupervised deep image stitching: Reconstructing stitched features to images, 2021.\n\n[D] Parallax-tolerant unsupervised deep image stitching, 2023.\n\n[E] Differential diffusion: Giving each pixel its strength, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their careful comments and agreement with our motivation. Here, We address some of the concerns shared by multiple reviewers and upload a PDF with rebuttal figures.

**1. We clarified key contributions**

Due to the previous manuscript's limited space, our contributions are not clear enough, leading reviewers to underestimate our work's technical contributions to the image stitching. Here, we revised the contributions at the end of Introduction section as follows:

>The main contributions of this paper are: 
>
>1. We propose SRStitcher, which reformulates the problem definitions of the fusion and rectangling stages to construct a more streamlined and robust image stitching pipeline.
>
 >2. SRStitcher is the first to introduce the concept of inpainting to address the image fusion problem. It incorporates prior knowledge from large-scale pre-trained models into the image stitching pipeline, enhancing the robustness of image fusion against registration errors. 
>
>3.  Without additional fine-tuning or supervision, SRStitcher improves the generalization of the rectangling method in the zero-shot scenario, opening up new possibilities for unsupervised image rectangling research.
>
 >4. We conduct extensive experiments to verify the interpretability and generalization of the proposed unified model. The results show that SRStitcher significantly outperforms the state-of-the-art methods in both quantitative and qualitative evaluations.

**2. Feasibility of the unified inpainting model**

1. Feasibility of applying the Inpainting model to the fusion problem

At present, image fusion models are mainly divided into two categories: reconstruction-based and seam-based. SOTA reconstruction-based model UDIS [A] has been shown to introduce blurring parallax regions when dealing with large parallax inevitably. When there are registration errors between images, SOTA seam-based method UDIS++ [B] forcibly searches for seams, which leads to severe image distortion, as shown in Fig.1 ② in our paper.

To solve the above problems, we propose a new solution to the image fusion problem based on inpainting model. Compared with reconstruction-based methods, our method does not introduce blurred regions when dealing with large parallax scenes. In contrast to seam-based methods, our method does not rely on the assumption that perfect seams exist in the fusion image. Therefore, our inpainting-based method improves the fusion robustness for registration errors. 

Our method needs to paint some image pixels, and we use the seam mask to limit the scope of modification strictly. Secondly, we introduce weighted masks to control the intensity of modification, ensuring the image's semantic consistency before and after modification. Extensive experiments confirm the effectiveness of our method. Therefore, we firmly believe that the proposed method is practical for the fusion problem.

2. Feasibility of applying the Inpainting model to the rectangling problem

One reviewer questioned the feasibility of applying the inpainting model to the rectangling problem based on the DeepRectangling paper. However, the DeepRectangling [C] paper was published in 2022, and diffusion-based inpainting methods have substantially progressed after 2022. Therefore, the possibility of applying the current diffusion-based Inpainting model in the image rectangling problem can not be denied based only on the conclusion of DeepRectangling. Also, RectanglingGAN [D], a paper published in June 2024, verifies the feasibility of the inpainting model for the rectangling problem.

Our method adopts the inpainting-based model to solve the rectangling problem. To deal with abnormal content generation, we design a combination strategy of coarse rectangling and weighted initial mask, which has solved the concern proposed by DeepRectangling. Experiments (Fig. 2 and Rebuttal Fig. 1) show that our method significantly outperforms DeepRectangling in the ability to preserve the original structure of the fusion image. Therefore, we firmly believe that the proposed method is practical for the rectangling problem.

**Improvement introduction:** Due to the space limitation, the Introduction section of the previous paper does not discuss in detail the inherent limitations of the existing fusion and rectangling methods and the breakthrough contribution of our method to them. We realize that it may lead to the reviewers underestimating the contribution of our method. Therefore, we have integrated the above discussion into our Introduction section in the revised paper.

**3. Controllability of generative model**

Some reviewers expressed concern that generative models might introduce uncontrollable content. We believe that the controllability of generative models is an optimistic research direction. 

In recent years, the study of controllability of generative models has made remarkable progress and produced many widely influential works. For us, an important insight is provided by the work of Diff-Plugin[E], which verifies that large-scale pre-trained diffusion models and lightweight plugin networks can effectively handle low-level tasks in various visual domains, including de-rain, de-fog, and low-light enhancement while maintaining high-fidelity content consistency. Diff-Plugin confirms the ability of generative models in terms of content fidelity, giving us confidence to SRStitcher.

Therefore, we hold a positive attitude toward the controllability of generative models and believe that the research prospects in this area are up and coming.

**references**

[A] Unsupervised deep image stitching: Reconstructing stitched features to images, 2021.

[B] Parallax-tolerant unsupervised deep image stitching, 2023.

[C] Deep rectangling for image stitching: A learning baseline, 2022.

[D] RectanglingGAN: Deep rectangling for stitched image via image inpainting, 2024.

[E] Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks, 2024.

Meta-review: The paper proposes to unify the fusion and rectangling stages of image stitching into a single stage, formulated as an inpainting problem. With this formulation, the proposed method effectively incorporates prior knowledge within pre-trained diffusion models to address the unified problem. Initially, there were concerns about whether the inpainting formulation is a suitable solution for rectangling and fusion, as previous work provided counter-arguments. However, these arguments were made before the emergence of diffusion models, which could be a game-changer. Although the results are imperfect, the proposed method offers a novel perspective on utilizing diffusion models for image stitching. Although several other issues were raised, the rebuttal effectively addressed most. Following the discussion stage, the consensus among the reviewers is positive. Please follow the reviewers' suggestions and include the updated clarifications and experiments in the revised version.
Decision: Accept (poster)