Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Authors: Xiancheng Sun
Senmao Ma
Shengxi Li
Mai Xu
Jingyuan Xia
Lai Jiang
Xin Deng
Jiali Wang
Venue: ICML 2025 poster (2025)
Keywords: Panoramic image outpainting
spherical convolution
ERP format
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Panoramic image outpainting acts as a pivotal role in immersive content generation, allowing for seamless restoration and completion of panoramic content. Given the fact that the majority of generative outpainting solutions operates on planar images, existing methods for panoramic images address the sphere nature by soft regularisation during the end-to-end learning, which still fails to fully exploit the spherical content. In this paper, we set out the first attempt to impose the sphere nature in the design of diffusion model, such that the panoramic format is intrinsically ensured during the learning procedure, named as spherical-nested diffusion (SpND) model. This is achieved by employing spherical noise in the diffusion process to address the structural prior, together with a newly proposed spherical deformable convolution (SDC) module to intrinsically learn the panoramic knowledge. Upon this, the proposed method is effectively integrated into a pre-trained diffusion model,  outperforming existing state-of-the-art methods for panoramic image outpainting. In particular, our SpND method reduces the FID values by more than 50\% against the state-of-the-art PanoDiffusion method. Codes are publicly available at \url{https://github.com/chronos123/SpND}.

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Xiancheng Sun
Senmao Ma Shengxi Li Mai Xu
Jingyuan Xia
Lai Jiang
Xin Deng 12 Jiali Wang"
Abstract
Planar Grid
BE
Seterical Jos
Completed Panoramic Image
Panoramic image outpainting acts as a pivotal role in immersive content generation, allowing for seamless restoration and completion of panoramic content. Given the fact that the majority of gen- erative outpainting solutions operates on planar images, existing methods for panoramic images address the sphere nature by soft regularisation during the end-to-end learning, which still fails to fully exploit the spherical content. In this paper, set out the first attempt to impose the sphere nature in the design of diffusion model, such that the panoramic format is intrinsically ensured dur- ing the learning procedure, named as spherical- nested diffusion (SpND) model. This achieved by employing spherical noise in the diffusion pro- cess address the structural prior, together with newly proposed spherical deformable convo- lution (SDC) module to intrinsically learn the panoramic knowledge. Upon this. the proposed method is effectively integrated into a pre-trained diffusion model, outperforming existing state-of- the-art methods for panoramic image outpaint- ing. In particular, our SpND method reduces the FID values by more than 50% against the state-of-the PanoDiffusion method. Codes are publicly available at https:/ / gig.tu.b.......g...... com/ chronos 1 23/ SpND.
Partial Image
Pianar Noise
Splecrical Grid "
Completed Panoramic Image
Partial Image
Structural
Figure I. Comparison between the proposed and existing meth- ods for panoramic image outpainting. (a) Existing methods learn panoramic content by soft regularisations, in a macro way, thus out- painting irregular 3D arrangement and artefacts. (b) The proposed method incorporates the sphere nature within the design, thus in micro way, which is able outpaint realistic and high-quality panoramic images. Note that E' denotes for the encoder and D denotes for the decoder.
2019; Zhang & Kou, 2022), augmented reality (AR) (Lee & Tsai, 2015; Eiris et al., 2018), and autonomous driving (Huang et al., 2018; Yurtsever al., 2020; Gao al., 2022). Although existing methods (Rombach et al., 2022) are ca- pable of generating high-quality content, they designed for planar images and fail to generalise well to panoramic images, due to the spherical distortion introduced by the equinectangular projection (ERP). Consequently, the gener- ation panoramic images has emerged as a pivotal area of research, drawing extensive research interest seeking to pro- duce high-quality panoramic content. We also notice several text-to-pant text-to-panorama panorama generation methods, including PanFusion (Zhang et al., 2024) and MVDiffusion (Tang et al., 2023) by incorporating attention mechanism to partially address panoramic cues. They still fail to address spatial continuity and contextual coherence for panoramic image outpainting.
1. Introduction
Most recently, panoramic images have received increasing research efforts due to their critical roles in applications such as virtual reality (VR) (Wan Abd Arif et al., 2009; Wang,
Department of Electronic Information Engineering, Beihang University, Beijing, China "State Key Laboratory Virtual Real- ity Technology and Systems, Beihang University, Beijing, China ? "Department of Electronic Science and Technology, National Uni- versity of Defense Technology, Changsha, China, Location, Coun- try *Cainiao Technology, Hangzhou, China. Correspondence to: Shengxi Li <LiShengxie @ bua...du.cn>.
As one of the main-stream tasks of panoramic image gener- ation, panoramic image outpainting has been pioneered by works (Akimoto et al., 2019; Sumantri & Park, 2020), which are able to complete scenes with the 360 x 1 180 field-of- view (FOV). To ensure spherical continuity of the generated edges, one of the core characteristics panoramic images, existing solutions include the usage of circular padding tech- nique (Hara et al., 2021; Akimoto et al., 2022) and rotation
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
within the latent space (Wang et al., 2023a). A further cru- cial problem is the accurate capturing against the panoramic content, which is distorted by ERP during the training. Ex- isting methods relieve this by employing the panoramic- guided loss (Wang et al., 2024), extra depth modality cues (Wu et al., 2024b), or unconstrained deformable convolution operations (Wu et al., 2024a), which essentially impose soft regularisations in an macro way to guide the overall training procedure. Generative models trained in end-to-end style by the soft regularisations, however, may not be able to op- erate as expected, thus failing to fully exploit the spherical geometry and panoramic content as illustrated in Figure 1.
accomplished by the CME ensure the high-quality panoramic image outpainting.
2. Related Works
Image Outpainting. The image outpainting task aims to fill unknown regions beyond the boundaries of given con- tent. Based on advancements of generative models, such as variational autoencoders (VAEs) (Kingma & Welling, 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2014), a series of image outpainting methods has been developed (Pathak et al., 2016; lizuka et al., 2017; Yu et al., 2018; Zheng et al., 2019; Wang al., 2019; K. & Ali, 2020; Zhao et al., 2021). The generative models allow for out- painting semantically meaningful content when completing images based on a provided portion from an overall image. The most recent advancements in diffusion model (Rom- bach et al., 2022) with the transformer architecture (Vaswani et al., 2017) have further improved completed images with consistent structure and semantics (Dosovitskiy et al., 2021; Esser et al., 2021: Wan et al., 2021; Zheng et al., 2022; Yao et al., 2022; Lugmayr et al., 2022; Xie et al., 2023; Wang et al., 2023b; Li et al., 2024). However, regarding panoramic images, these methods overlook the unique characteristics of panoramic images, including omnidirectional continuity across edges and spherical distortion.
In this paper, we set out the first attempt to outpaint panoramic images by incorporating their intrinsic sphere nature in an micro way, i.e. by the sophisticated design of generative models. More specifically, we first propose to sample identically and independently distributed (i.i.d) spherical noise on the 3D surface in the diffusion model, corresponding to the sphere nature of panoramic images. To the best of our knowledge, all the existing methods sample noise on the planar latent space. Though, the i...d. spherical noise essentially results into non-i.idd planar noise after the ERP projection; this ensures that our method suits well in the panoramic image outpainting task. More importantly, given the fact that an irregular grid is required to maintain the regular grid on the sphere, we propose to adaptively perceive the receptive field on the sphere, instead of on the plane, in which the spherical deformable convolution (SDC) layer is developed in particular for the panoramic content. Our SDC layer introduces fundamentally novel convolution operation to mitigate panoramic deformation, which is distinct from existing strategies such feature re-organisation (Shen et al., 2022) or sphere-guided loss (de La Garanderie et al., 2018; Jaus et al., 2023) that are employed for panoramic depth estimation and segmenta- tion tasks. Built upon the SDC layers, we propose our spherical-neste nested diffusion (SpND) model, based on a newly developed circular mask encoder (CME) for the consistency across edges. The experimental results have verified the sig- nificantly superior performances of our SpND model. Our contributions are three-fold:
Panoramic Image Outpainting. Different from planar images with a narrow FOV, panoramic images with a 360 x 180 FOV exhibit spherical distortion due to the ERP projection. Consequently, objects in the ERP format undergo significant distortion, particularly for those located near the top and bottom regions of the image. Image com- pletion methods for panoramic images are thus required to address the spherical distortion inherent in their structure and ensure the preservation of omnidirectional continuity. Early studies (Sumantri & Park, 2020; Akimoto et al., 2019; Lu et al., 2021) primarily focus on ensuring omnidirectional continuity by adopting convolution operation with the cir- cular padding strategy. To address the spherical distortion, there are strategies such as using extra depth information (Oh et al., 2022), adding projection loss (Somanath & Kurz, 2021), and leveraging scene-symmetry properties (Hara et al., 2021). In order to generate visually pleasant content, Omnidreamer (Akimoto et al., 2022) employs transformer- based sampling strategy to implicitly learn the spherical structure. To achieve a smooth transition between the real and generated content, incoporates learnable positional encoding cues with existing features that are still processed by standard 2D convolu- tion. Leveraging the powerful diffusion model, PinoDiff (Wang et al., 2023a) employs the ControlNet (Zhang et al., 2023) with the positions as an additional input and also uses spherical loss to regularise panoramic content. To further ac- comodate intricate visual details, AOG-Net (Lu et al., 2024)
We seamlessly incorporate the spherical noise by inves- tigating ERP distortion when training diffusion models, such that the structural prior of panoramic images can be well accommodated.
(Liao et al., 2024)
We propose the SDC layer that is the first successful attempt to satisfy the intrinsic sphere nature within generative model architectures, with adaptive and con- sistent receptive fields.
We develop the SpND model by incorporating the spherical noise and SDC layer as fundamental modules,
2

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
proposes an autoregressive pipeline for panoramic image outpainting, utillssig feature remapping for panoramic con- tent. On the other hand, PanoDiffusion (Wu et al., 2024b) introduces the depth modality during training to enhance panoramic awareness. Unfortunately, the above methods dominantly address the distortion by soft regularisations, from the perspective of guiding the overall end-to-end train- ing, i.e. in a macro way. This may result in suboptimal performance since the macro constraints may not fully con- tribute to learning the panoramic content as expected.
5=5
5 =30
5 = 1000
Figure 2. Illustration of the i.dd spherical noise which exhibits non-i.id. nature after the ERP projection. We illustrate examples with different sampling densities ( on the 3D spherical surface resolution of 128 x 64.
Based on the above analysis, we propose to sample noise in 3D space and transform it into the ERP format to capture the intrinsic structural prior. The uniform spherical noise is sampled by adopting sampling density per unit length in 3D space. Accordingly, the number of sampling points mi at latitude & on the spherical surface can be formulated by
3. Methodology
3.1. Spherical Noise for Structural Prior
The noise plays an essential role that relates to the random- ness of outpainting, which benefits the overall performance by considering the panoramic structural prior. More specifi- cally, since panoramic images commonly projected into the ERP planar format (Ye et al., 2018) for processing, we aim to build an explicit structural prior knowledge for the ERP format to guide the panoramic image outpainting pro- cess. Given the fact that images in the ERP format exhibit higher pixel density near the equator and lower pixel den- sity near the poles, modelling the coherence of panoramic images using i.i.d. Gaussian noise, the de facto choice for planar images, is mismatched with the intrinsic character- istics of panoramic ERP format. Instead, by modelling panoramic images as pixels uniformly distributed over the surface of a sphere, we are able to capture pixel coherence in the ERP format by sampling i.i.d. noise uniformly dis- tributed in 3D space and projecting it into the ERP format at given resolutions.
m = I< 2n cos Φ1
(2)
where 2n cos denotes the geodesic length and [-] is the ceiling function. Note that the number of sampling point at poles is 1. essentially controls the sampling density of the i.i.d. Gaussian noise on 3D spherical surface. Therefore, given my the fixed number of points for a row in the ERP format, for latitudes where my Ma,, one sampling point on the spherical surface corresponds to multiple points within a single row in the ERP format. This one-to-many mapping inherently introduces spatial coherence in the ERP domain. As a result, i.i.d. noise in 3D space is no longer i.i.d. when projected into the ERP format. In contrast, for latitudes where m& > Max the i..dd property within row in the ERP representation can be preserved. Both the sampling density Ç and the resolution, which determines the number of sampling points mw play an important role in modelling the sphere nature of the ERP format. Since the resolution of feature is set as 128 x 64 in our model, we further illustrate the non-i.i.d noise (aka. structural prior) with different Ç in Figure 2. We also illustrate detailed analysis in Appendix A, by choosing < 30 for our SpND model.
Firstly, we take analysis the spherical property of the ERP format. We denote u, as the coordinates in the ERP format and 0, & are the spherical coordinates on the surface of the sphere. two widely used coordinates for panoramic images. Correspondingly, the transformation from spherical format the ERP planar can be formulated by 9u//2x u= (6+ 5 ///F, where w and h are the width and height of the image, respectively. Also, the spheri- cal coordinates (0, Φ) are defined by x = cos & cos f, V.V= = cos sin 0, = sin ó where I, y, 2 are the Cartesian coor- dinates in 3D space. Then, we assume that there are m sampling points per row in the ERP format. The actual geodesic distance de between two adjacent sampling points on the spherical surface, height in the ERP format, is given by
3.2. Spherical Deformable Convolution
Generally speaking, the deformable convolution (Dai et al., 2017) decomposes the standard convolution operation into two steps: 1) Sampling certain points from the input feature Fin using a regular grid R; 2) Summing the sampled points weighted by W. In the standard convolution, the grid
R- R-((-L--D,(-,,,
(0, 1))
(0, 1),(1.
2TT
TV
π 2
(3)
dg
COS
(1)
defines 3 x 3 kernel with stride 1 and determines the receptive field. For each location po on the output feature F Fouts the deformable convolution can be formulated by
ma
Therefore, given the points at poles (v = 0 and = h), d, = 0 means that all points essentially correspond to a
single point in 3D space. We summarise this property as the sphere nature of the ERP format.
F Fout (po) = Σ W(Px) Fin(Po + Pa + Apa), (4)
Pr,
3

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
where P.. enumerates the locations in R and [Apa/n 1. ,N] denote the learnable offsets, where N = -
input feature Fin. the SDC layer can be formulated
Four(Pu) Σ
W(p.) Fm Pn + Apn): (7)
Although deformable convolution improves the flexibility of the convolution operation, it still operates as the planar convolution, thus falling short in highly complex geometric scenes, such as spherical data of panoramic images. This issue stems from the fact that directly applying deformable convolution to panoramic images results in suboptimal ini- tialization, as it relies the regular grid R as the initial grid (Dai et al., 2017);; this misaligns with the spherical structure inherited from panoramic images.
P.
where Pn. enumerates the locations in the spherical grid R W denotes the learnable weights of the kernel, and the constraint for the learnable offsets is formulated by Apml < a h where denotes the height and a is a hyperparameter which defines the ratio against the overall height.
3.3. Spherical-Nested Diffusion Model
To address the mismatch, we propose the novel SDC layer that leverages the spherical gird, while integrating dynamic adaptability of deformable convolution for precise local feature aggregation. More specifically, the spherical neigh- bourhood of a kernel at position Po can be defined as spherical grid R2* as introduced in (Coors et al., 2018), us- ing gnomonic projection (Frederick Pearson, 1990) and a tangent plane centred at Po on sphere. The coordinate for Po = -f0, Φo) is defined by its latitude Φ E [- 2. 21 and longitude &g E [- x]. Recall that w and h are the width and height of the image in ERP format (Ye et al., 2018). Then, the 3 x 3 grid R' on the tangent plane (Coors et al., 2018) is reformulated from (3) as
Considering the sphere nature of panoramic images, our SpÑD model achieves panoramic image outpainting by in- tegrating the structural prior and the newly proposed SDC layer. The overall architecture of our SpND model is illus- trated in Figure 3. We further develop the CME module to preserve omnidirectional continuity and the spherical structure, thereby significantly enhancing the quality of the generated content. More specifically, our CME is devel- oped to encode the masked input image, which contains the known portion within panoramic images. Then, the trainable CME module employs 3 x 3 circular convolu- tion layers with the sigmoid linear unit (SiL.U) activation function (Elfwing et al., 2018), to transform the 3-channel masked image into the 256-channel feature, with an overall downsampling ratio of 8. Subsequently, the 256-channel latent feature is fed into the prior injection layer, which aggregates the structural prior for the ERP format. This process produces an intermediate output containing both ERP structural information and image completion details. We denote Xm as the masked input and PERP the ERP structural prior. The prior injection layer operats as follows,
R' = [[(- tan Ao, - sec Ao tan Ap), (- tan Ao, 0), (-tan Do, sec Ao tan A&), (0, tan A p), ... (0, tan Ao), (tan Ag, sec As tan }
(5)
where A6 = T and D3 = 2 denote unit steps on the tangent plane. By representing the elements within R' as R' = ((Ai, Aja)(n n = 1.....,, we can obtain the spherical gird R = {(0n, )n) n= = 1,... ,N centred at position po through
pei m O CME + (1 m) C P PERP
(8)
where F pri denotes the aggregated feature, m is the mask, represents the element-wise production, and CME[-] means the encode process of the CME module. Note that the mask m is provided by the input.
O
Ain
8- 0o + ta
cos cos
Ain sin 0g
(6)
1
8² sin-1
(sin Φu
Ain cos (b))
1
+
Ad
The spherical net is then established based the spherical- nested blocks (SpNBs), which is introduced to extract panoramic features that guide the pre-trained diffusion model for panorama outpainting. More specifically, each SpNB incorporates an SDC layer to effectively encode spherical information, catering for the sphere nature with panoramic images. Furthermore, a vanilla deformable convolution layer is employed to bridge the spherical fea- ture and the pre-trained diffusion model. This additional layer enhances compatibility with the pre-trained diffusion model, allowing for effective integration and improved per- formances.
where Ad V(Ain) Jz (Coors al., 2018).
Therefore, the spherical grid R defines the 3x x 3 neighbour- hood around position Po on the sphere in our SDC layers. While a kernel size of 3 is used here as an example, the kernel size can be arbitrarily chosen in practice.
Furthermore, since the spherical grid serves as the initial grid for the SDC layer, the usage of unconstrained learnable offsets Ap.. may deteriorate the deformable convolution to neglect the spherical prior knowledge. To address this issue, we impose constraints on the offsets to ensure that the SDC layer consistently integrates the spherical prior knowledge. For each location po on the output feature F Fout and the
Aligned with the architecture of ControlNet (Zhang et al., 2023), we adopt the SiLU activation function (Elfwing et al.,
4

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Masked Imput
Spherical
Spherical Deformable Convolution (SDC)
Legends
Conv Kernel
Grrdd
Deform
Spherical Kernel
Frozen
Struetural Prior
Offect Kernel
Trainable
Ourput
Circular Conv
Diffussin Block
Eaput
Circular Mask
Encoder (CMB)
Circular
Prior Injection
Spherical
Apherical-Nested Nested Block (SpNB)
Constrain for Offset
Conv 3x3
TimeEmbell bed
Deformable
SpNB
CrossAtia
Spherical Net
SILU
SDC
SiLL
Conv
Equi-rectangular Projection
SpNB
Circular Cony 3x3
Spherical Grid
SpNB
SpaB
Prompt
Spherical Net
Image
Decoder
Z,
Z,
Zo
Denoising U-Net
Denoising U-Net
Output Panorama
Figure 3. The overall architecture of our proposed SpND model. The input of our SpND model is the masked images, which are first encoded by our circular mask encoder (CMF) module. Subsequently, the prior injection layer integrates structural prior knowledge tailored for the ERP format into the encoded masked features. These features subsequently refined by the spherical net, composed of SpNBs, which incorporate spherical prior knowledge inherent the SDC layers the SpNBs. The refined features then guide a pre-trained diffusion model to perform panoramic image outpainting using the repaint technique.
2018) to process features encapsulating the spherical prop- erty, combined with information from the original image, and sequentially integrate into the middle and output layers of the diffusion model. Assume that fsD denotes the pre- trained diffusion model and fPDN denotes the spherical net. The process can formulated as
mask aligned to the input, as perform the image out- painting task. We provide more details for the repainting technique in Appendix B. Utilizing the repaint technique for the pre-trained diffusion model ensures that the masked area remains consistent with the input, thereby enabling seamless image outpainting without introducing distortion to the provided regions.
E /SD Le. T, JPDN F pri, T)}:
where is the embedding of the input prompt and evp is the output of our SpND model. Note that vp denotes the learnable parameters. Furthermore, the final output ey is de- coded by the image decoder to get the completed panoramic image. Note that the image decoder is a pre-trained VAE model (Rombach et al., 2022). Moreover, the text embed- ding T is derived by encoding the text prompt using the widely adopted CLIP model (Radford et al., 2021).
4. Experimental Results
4.1. Experimental Settings
Datasets. To evaluate the performance of our SpND model, we employed the widely applied Matterport3D (Chang et al., 2017) and Structured3D) (Zheng et al., 2020) dataset for com- parison. Similar to (Lin et al., 2019), we obtained 10912 panoramic images with size 1024 512 for the Matter- port3D dataset. A total of 9, 820 images were selected for the training, and all 1, 0912 images were used for evaluation to compute the sufficient statistics. For the Structured3D dataset, we followed the methodology outlined in (Wu et al., 2024b) to obtain 21,133 images, of which 19,019 images were used for training and all 21,133 images were used for evaluation compute the sufficient statistics. Prompts generated by the BLIP-2 (Li et al., 2023) model for both datasets.
During the training process, the pre-trained diffusion model and the VAE for encoding panoramic images are frozen, whilst the CME and the spherical net are trainable, as illus- trated in Figure 3. We follow the standard process (Rombach et al., 2022) to train the conditional diffusion model via the default diffusion loss, which can be formulated by
C = E. |||e Egb [Z, t, T
(10)
where is the Gaussian noise, denotes the time, and Z, is the latent feature in the pre-trained diffusion model.
Implementation Details. Our SpND model was trained based on the pre-trained weights of (Zhang et al., 2023). The hyperparameter Ç was set to 30 to obtain the struc-
During the inference stage, the repaint technique (Lugmayr al., 2022) is employed in our SpND model, with the
5

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Table Quantitative comparisons with state-of-the-art methods Matterport3D and Structured3D. The hest results and the second-best results are highlighted in bold, underline.
Matterporta
Structured3D
Methods
FID 4
Fort
Avg f
Pre f
Recall f
FIID 4
FloDiri 4
Avg t
Pre t
Recall
SIG-SS (Hara et al., 2021)
64.81
50.53
0.22
0,42
0.01
117.84
72.51
0.15
0.29
0.01
TFill (Zheng et al,, 2022)
77.69
85.28
0.15
0.01
85.00
84.49
0.04
0.06
0.01
OmniDreamer (Akimoto al., 2022)
51.69
44.44
0.31
0.45
0.16
20.63
24.94
0.35
0.51
0.19
PanoDift (Wang et al., 2023a)
13.45
13.11 11
0.50
0.24
0.75
11.11
9.75
0.53
0.36
0.69
AOG-Net (Lu et al., 2024)
38.56
44.27
0.10
0.16
0.04
85.39
98.11
0.10
0.19
0.01
PanoDiffusion (Wu et al., 2024b)
39.43
17.34
0.49
0.55
0.42
9.80
10.84
0.49
0.55
0,43
SpND (Ours)
9.08
10.64
0.62
0.52
0.72
7.37
6.05
0.62
0.62
0.62
SpND proeuge (Ours)
6.67
6.58
0.64
0.52
0.76
4.60
4.75
0.64
0.66
0.62
(a) Ground Truth
(b) Input Masked Image
(c) OmniDreamer
PanoDiffusion
Sp (ours)
()) AOG-Net
R TFill
(lh) SIG-SS
ii PanoDiff
(i) SpND gromps (ewEs
(a) Ground Truth
(b) Input Masked Image
(c) Dreamrramer
(d) PamoDiffusion
(c) SpND (ours)
(1) AOG-Net
( TFil
(b) SIG-SS
(i) PanoDiff
(i) SpNDoeage (ours)
Figure 4. Qualitative comparisons of panoramic image outpainting on the Matterport3D dataset. The centre mask is employed to generate the masked input image, and the results the corresponding completed images obtained by different methods.
tural prior for the ERP format. During training, we used the AdamW optimizer (Loshchilov & Hutter, 2019) with a leam- ing rate of 10-5 and batch size of 4. During inference, the classifier-free guidance scale (Ho & Salimans, 2021) was set to 3.0 with DDIM (Song et al., 2021) step of 50. Addition- ally, the hyperparameter a in SDC layers was set to 0.04 to constrain the learnable offset. To rigorously evaluate the effi cacy of integrating prior knowledge, we utilised the general prompt "Ouspanniing"
Baselines. For panoramic image outpainting, we mainly compared with the existing state-of-the- panorama out- painting methods including SIG-SS (Hara et al., 2021), Om- niDreamer (Akimoto et al., 2022), PanoDiff (Wang et al., 2023a), AOG-Net (Lu et al., 2024), and PanoDiffusion (Wu et al., 2024b). Additionally, we compared with the image inpaingting method TFill (Zheng et al., 2022) following Pan- oDiffusion (Wu al., 2024b). We fine-tuned all the baseline methods on the Matterport3D dataset, whilst fine-tuning al- most all the baseline methods on the Structured3D datasets (except for the PanoDiffusion model where the pre-trained weights exist on Structured3D) for fair comparison.
validate our approach. This choice was motivated for fair comparisons, by the fact that several baseline methods do not incorporate prompts as part of their input. Furthermore, to demonstrate that performance could potentially be further improved with application-spei speciifc prompts, we trained an additional model incorporating vary- ing text prompts, denoted prompt-
Metrics. We utilised the commonly adopted Fréchet in- ception distance (FID) metric assess the quality of the completed panoramic images. More specifically, we fol-
6

---

Spherical-Nester Diffusion Model for Panoramic Image Outpainting
(a) Ground Truth
(b) Input Masked Image
(c) Omniereamer
PanoDiffusion
(c) SpND(ours)
(1) Net
(g) TFill
(h) SIG-SS
(i) PanoDiff
(i) SpNDgrnnee grompe (ours)
(a) Ground Truth
(b) Input Masked Image
(c) Oreameemmer
(d) PanoDiffusion
(c) SpND (ours)
(f) ADG-Net
(g) TFill
(h) SIG-SS
(i) PanoDiff
(i) SpND (ours)
Figure 5. Qualitative comparison of panoramic image outpainting on the StructuredDD dataset. The centre mask is employed to generate the masked input image, the results are the corresponding completed images obtained by different methods.
lowed the method in (Wu et al., 2024b) directly compute the FID score. To further refine the evaluation of image quality for panoramic images, we adopted the approach pro- posed in (Tang et al., 2023) and computed FiD This was achieved by calculating the FIDhori scores using 8 equally spaced 512 x 512 cropped images, extracted from a panoramic image with size 1024 x 512. Additionally, the precision (denoted as Pre) and recall metries (Kynkäänniemi al., 2019) also calculated to comprehensively assess outpainting performance. A higher precision value indicates better image quality, while higher recall value reflects improved diversity in the outpainted images. We average the precision and recall metric (denoted as Avg) to compre- hensively evaluate the performance for panoramic image outpainting.
Qualitative Results. We randomly chose scenes from both dataset and reported the subjective results in Figures 4 and 5. As can be seen from those figures, panoramic images outpainted by our SpND model demonstrate distinct superior quality and optimal alignment with the provided input sections, also achieving the smoothest transitions at the boundary between the real and generated components compared to other methods. Furthermore, with the newly proposed structural prior and SDC layers, our SpND model preserves the spherical structure well and outpaints visually compelling panoramic content. In addition, image quality and semantic continuity can be further improved through the usage of specific prompts in our SpND model.
Outpainting from View Images. To further evaluate the performance and the robustness of the proposed SpND model, we conducted panoramic image outpainting exper- iments using view images. For comparison, we selected the best two existing methods based on their performance in prior experiments and performed the comparison on the Structured31 dataset!. As illustrated in Figure and Table 2, our SpND model demonstrates superior performances, achieving the best results both subjectively and objectively.
4.2. Comparison for Panoramic Image Outpainting
Quantitative Results. The widely adopted centre mask was employed in performing the panoramic image outpainting task for all comparing methods. As reported in Table 1, the proposed SpND model consistently achieves the best perfor- mance by taking into account both the quality and diversity of the outpainted images. More importantly, with specific prompts as input, the overall outpainting performance can be even better across all metrics, as shown in the last two rows of Table While PifoDiff achieves superior recall for the Structured3D dataset, its FID and the precision are quite limited. In addition, the GAN-based methods including SIG- SS and TFll exhibit extreme low recall scores, indicating that the panoramic outpainting diversity is poor.
Outpainting with Mask near the Pole. To further evalu- ate the performance of our method different masks, conducted additional experiments masks located near the poles, whereby the spherical distortion of masks becomes apparent. More specifically, we projected a 256 x 256 image
We also conducted experiments on a small Martian image dataset using SpND prompt, which is detailed in Appendix C.

---

-eested Diffusion Model for Panoramic Image Outpainting
(n) Ground Truth
(b) Masked Input
(c) PanoDiffusion
(d) PanoDiff
(c) SpND
Figure Qualitative comparison of panoramic image outpainting from view images on the Structured3D dataset. The centre view mask employed to generate the masked input panoramic view, and the results are the completed images obtained by different methodss.
(a) Ground Truth
(b) Masked Input
(c) PanoDiffusion
(d) PanoDiff
(e) SpN
Figure 7. Qualitative comparison of panoramic image outpainting with mask near the pole on the Matterport3d dataset. The distorted mask near the pole is employed to generate the masked input image, and the results are the completed images obtained by different methods.
Table Comparisons for panoramic image outpainting from view images the Structured3D dataset. The hest results are bold.
Table 4, Ablation study of core components in SpND model. The comparison includes three variants af SpND model: without the structural prior (denoted as w./o. PERP); without the SDC layer (denoted w./o. R²) and the one without the learnable offsets (denoted as w:/o. Ap..) The best results are in bold.
Methods
FID 1
FID 4
Avg
Pre f
Recall f
PinoDislusion
0.36
0.38
PanoDiff
9.20
8.19
0.54
0.40
0.68
SpNDpr (ours)
4,23
3.98
0.64
0.66
0.61
Methods
FID +
FIDDoo
Avg t
Pre f
Recall
w/o. PERP
9,93
12.05
0.53
0.42
0.63
Table Comparisons for panoramic image outpainting with mask near the pole on Matterport3D. The best results are in bold.
w//o R*
11.64
15.65
0.52
0,41
w/oo Apn.
12.76
11.17
0,56
0,46
0.65
SpND (ours)
9.08
10.64
0.62
0.52
Methods
FID 1
FID
Avg
Pre f
Recall f
PanoDiffusion
15.92
17.76
0.48
0.54
0.42
prior knowledge embedded in the grid R2 of the SDC layer. The integration of structural prior knowledge enhances the quality and diversity of completed images, as evidenced by improvements in FID, precision, and recall metrics Table 4. The enhancements in image quality are shown in Figure 8. Furthermore, leveraging the spherical grid R within the SDC layer as the spherical prior knowledge yields addi- tional improvements in image quality and diversity. These experimental results underscore the critical components in substantially enhancing the performance of panoramic im- age outpainting. Please note that, the learnable offsets, in conjunction with the spherical grid R*, play a crucial role in generating high-quality panoramic content, whereas the spherical prior R alone does not yield comparable results. As reported in Table 4, using the spherical grid R without the learnable offsets results in significant degradation in both outpainting quality and diversity.
PanoDiff
13.97
14.91
0.48
0.30
0.65
SpND prompt (ours)
8.49
8.84
0.57
0.44
0.69
onto the ERP format at the viewpoint 0 = 90000090 = 60%, obtain a highly-distorted mask near the pole. We then re- trained our SpND method, along with the best two baseline methods (i.e.. PanoDiff and PanoDiffusion) on the Matter- port3l) dataset for comparison. We report the quantitative re- sults in Table 3 and the qualitative results in Figure 7, which verifies the superior performance of our SpND method. This also verifies the advantages of our SpND method that arises from the spherical noise and SDC operation to accommo- date panoramic deformation.
4.3. Ablation Study
Effectiveness of Core Components. We first conducted an ablation study on the Matterport3D dataset to evaluate the impact of integrating the structure prior PERP, the spherical grid R and the learnable offsets Ap,. for the SDC layer, all of which consist of the components in our SpND model. For simplicity, all ablation studies were conducted using the general prompt "Outpainting" and report the re- sults in Table 4 and Figure 8. More specifically, we verified the effectiveness incorporating the structural prior knowl- edge tailored for the ERP format, as well as the spherical
Effectiveness Packed Modules. We further evaluated the effectiveness of the architecture employed our spherical net and the prior injection layer on Matterport3D dataset, Note that without the spherical noise and the proposed SpNB, our SpND model reduces to variant of Control- Net (Zhang et al., 2023) with the CME module (denoted as Repaint). Moreover, since our SDC layer uses the spherical grid similar to SphereNet (Coors et al., 2018), we further compared variant model which replaces the deformable
8

---

Spherical-Nestee Diffusion Model for Panoramic Image Outpainting
(a) Ground Truth
(b) Input Masked Image
(c) w.oo, Ap,
(d) w.o. Pize
(c) w.oo R'
(1) RePaint
(g) SphereNet
SpND (ours)
Figure 8. Qualitative comparison result of the ablation study Matterport3D dataset including most our ablation settings and the ground truth. Specifically, this comparison includes 3 variants of our SpND model described in Table 4, including SpND without the structural prior (denoted as w./o. PERP): SpND without SDC layer (denoted as w./o. R2) ) and SpND without the learnable offsets (denoted as w/o. Ap..) and two variants RePaint and SphereNet described in Table 5.
Table 5. Ablation study of packed modules the SpND model. The comparison includes variants: Repaint that utilises cir- cular convolution without integrating the prior knowledge and SphereNet that replaces the deformable convolution and the SDC layer with spherical convolution proposed in (Coors et al., 2018). The best results are bold.
ponent of our SpND model, with the parameter ( playing a pivotal role as mentioned in Section 3.1. To conduct a quantitative analysis of how sampling density influences the performance, we further performed an ablation study by varying Ç and report the results Table Here, (= oo cor- responds the w./o. PERP configuration in Table which employs the widely applied i.i.d. noise. As shown in Table 4, our SpND method achieves optimal performance and = 30, consistent with the analysis in Appendix A. In particular, Ç 30 achieves performance comparable to Ç = 45 while offering improved computational efficiency. A reduced sampling density (ç = 15) marginally degrades performance but still surpasses existing baseline methods.
Methods
FII) 4
Fhor $
Avg
Pre
Recall f
45
Repaint
10.10
12.89
0.58
0.42
0.74
SphereNet
11.78
15.74
0.54
0.41
0.67
SpND (ours)
9.08
10.64
0.62
0.52
0.72
Table 6. Ablation study evaluating the impact of varying sampling densities quantitatively on the Matterport3D dataset. The best results are in bold.
These results confirm the rationale and robustness of proposed spherical noise.
Sampling Density
FID J
FID
Avg 1
Pre f
Recall
C= 15
12.74
15.52
0.52
0.38
0.65
5. Conclusion
=
9.08
10.64
0.62
0.52
0.72
In this paper, we have proposed the novel spherical-nested diffusion (SpND) model, that for the first time integrated panoramic characteristics within the design of network ar- chitecture, namely, in micro way. More specifically, first developed the spherical noise catering for the struc- tural prior of panoramic images. To capture correct content within panoramic format, we have adopted the spherical grid within deformable convolution and have proposed the spherical deformable convolution (SDC) layers. Leveraging the structural prior and the SDC layer, we have introduced the SpND model which integrated the spherical net with pre-trained diffusion model to achieve superior outpainting for panoramic images. Extensive experiments have verified the superior performances for outpainting high-quality and diverse panoramic images, surpassing all the baselines. We may also need to point out our limitations by using Repaint technique in the latent space, which leads to slight distor- tion between the input and output against masks. Future works expected to further refine the mask areas during the diffusion process.
8.84
10.27
0.51
0.72
(w//o, Powp)
9.93
12.05
0.53
0.42
0.63
convolution and the SDC layer in the SpNB with the spheri- cal convolution proposed in (Coors et al., 2018) (denoted as SphereNet). We report the results in Table 5 and Figure 8. From the table and figure, the proposed SpND model, which integrates the sphere nature in micro way, significantly enhances the quality of the generated content. However, this improvement comes at the cost of slight reduction in out- painting diversity. Overall, the outpainting performance is notably improved, as evidenced by the FID metric in Table 5. More importantly, our model consistently achieves the best performance, demonstrating that directly employing spherical convolution fails to effectively integrate with the pre-trained diffusion model.
Quantitative Evaluations for Spherical Noise. The non- i.i.d. noise, which serves as explicit structural prior knowl- edge tailored for the ERP format, constitutes key com-

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Acknowledgement
Eiris, R., Gheisari, M., and Esmaeili, B. Using panoramic augmented reality develop virtual safety training environment. 04 2018. doi: 10. 10.1061/97807844888888 004.
This work was supported by Natural Science Founda- tion of China (NSFC) under Grants 62450131, 62206011, 1, 62231002 and 62401027, and Beijing Natural Science Foun- dation under Grant L223021.
Elfwing, S., Uchibe, E., and Doya, K. Siggoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 1,, 2018.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Esser, P., Rombach, R., and Ommer, B. Taming Trans- formers for High-Resolution Image Synthesis. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 12868- 12878, Los Alamitos, CA, USA, June 2021. IEEE Computer Society, doi:
References
10. 10.1 N0.109/CVPR44497 200p446633..021..00 202...01............
Akimoto, N., Kasai, S., Hayashi, M., and Aoki, Y. 360- degree image completion by two-stage conditional gans. In 2019 IEEE International Conference on Image Pro- cessing (ICIP), pp. 4704-4708, 2019. doi: 10.1 2019,8803435.
Frederick Pearson, I. Map ProjectionsTheory and Applica- tions. CRC Press, 1990.
Gao, S., Yang, K., Shi, H., Wang, K., and Bai, J. Review on panoramic imaging and its applications in scene derstanding. IEEE Transactions on Instrumentation and Measurement, 71:1-34, 2022. doi: 10. 10.1109/T1M.202.. 3216675.
Akimoto, N., Matsuo, Y., and Aoki, Y. Diverse plausible 360-degree image outpainting for efficient 3deg back- ground creation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 11441-14450, June 2022.
Goodfellow, I., Pouget-Abadie, J., Mirza, M.. Xu, B. Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014.
Chang, A.. Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A., and Zhang, Y. Matter- port3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2017.
Hara, T., Mukuta, Y., and Harada, T. Spherical image generation from single image by considering scene symmetry. Proceedings of the AAAI Conference on Ar- rificial Intelligence, 35(2):1513-1522 May 2021. doi: 10. 1..16699aaiii..m v33:22..24.3:22..22222.
Coors, B., Condurache, A. P., and Geiger, A. Spherenet: Learning spherical representations for detection and clas- sification in omnidirectional images. In Proceedings of the European conference on computer vision (ECCV), pp. 518-533, 2018.
Ho, and Salimans, T. Classifier-free diffusion guidance. In NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H.,, and Wei, Y. Deformable convolutional networks. In Proceedings of the IEFE international conference on computer vision,
Huang, X., Cheng, X., Geng, Q. Cao, B., Zhou, D., Wang, P., Lin, Y., and Yang, R. The apolloscape dataset for autonomous driving. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018
pp. 764-773, 2017.
de La Garanderie, G. P., Abarghouei, A. A., and Breckon, T. P. Eliminating the blind spot: Adapting 3d object de- tection and monocular depth estimation 360 panoramic imagery. In Proceedings of the European Conference Computer Vision (ECCV), pp. 789-807- 0099807,,22 2018.
lizuka, S., Simo-Serra, E., and Ishikawa, H. Globally and locally consistent image completion. ACM Trans. Graph., 36(4), July 2017. ISSN 0730-0301. doi: 10. 10.1145307299993077666
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Jaus, A., Yang, K., and Stiefelhagen, R. Panoramic panoptic segmentation: Insights into surrounding parsing for mo- bile agents via unsupervised contrastive learning. IEEE Transactions on Intelligent Transportation Systems, 24 (4):4438 4453, 2023.
10

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
K., V. and Ali, R. Image outpainting and harmonization us- ing gans. International Journal of Scientific Research in Computer Science, Engineering and Information Technol- ogy, pp. 294-297, 05 2020. doi: 10.32628/CSEIT226677,
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte,
R., and Van Gool, L.. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1 11461-1 11471, June 2022.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd Interna- tional Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Oh, C., Cho, W. Chae, Y., Park, D., Wang, L., and Yoon, K.-
J. Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning. Avidan, S., Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision ECCV 2022, pp. 352-371, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031- 19787-1.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metric for assess- ing generative models. Advances in neural information processing systems, 32, 2019.
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A. A. Context encoders: Feature learning by in- painting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Lee, I.-C. and Tsai, F. Applications of panoramic images: From 720 panorama to interior 3d models of augmented reality. The International Archives of the Photogranmetry, Remote Sensing and Spatial Information Sciences, XL- 4/W5:189-193 19289999-222011 2015.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8762 PMLR, 2021.
Ji,J., D.,
D., Savarese, S., and Hoi, S. 2 Bootstrapping language-iman pre- pre-training with frozen image encoders and large language models. In International conference on machine learning. pp. 19730-19744 PMLR, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Li, W., Yu, X., Zhou, K., Song. Y., and Lin, Z. Image inpainting via iteratively decoupled probabilistic model- ing. In The Twelfth International Conference on Learning Representations, 2024.
Liao, K., Xu, X., Lin, C., Ren, W., Wei, Y., and Zhao, Y. Cylin-painting: Seamless 360 panoramic image outpaint- ing and beyond. IEEE Transactions on Image Processing, 33:382-394, 2024.
Shen, Z., Lin, C., Liao, K., Nie, L., Zheng, Z., and Zhao, Y. Panoformer: panorama transformer for indoor 360 depth estimation. In European Conference Computer Vision, pp. 195-211. Springer, 2022.
Lin, C. H., Chang, C., Chen, Y., Juan D., Wei, W., and Chen, H. COCO-GAN: generation by parts via condi- tional coordinating. IEEE International Conference on Computer Vision (ICCV), 2019.
Somanath, G. and Kurz, D. Hdr environment map estimation for real-time augmented reality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11298-11306, 11306, June 2021.
Loshchilov, and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. International Conference on Learning Representations, 2021.
Lu, C.-N., Chang, Y.-C., and Chiu, W.-C. Bridging the visual gap: Wide-range image blending. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 843 851, June 2021.
Sumantri, J. and Park, I. K. 360 panorama synthesis from a sparse set of images with unknown field of view. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 2386-2395, 2020.
Lu, Z., Hu, K., Wang, C., Bai, L.,, and Wang, Z. Autoregres- sive omni-awar aware outpainting for open-vocabulary 360- degree image generation. Proceedings of the AAA/ Con- ference on Artificial Intelligence, 1421111142,,,112 Mar. 2024. doi: 10. 1009//a//aaai.... 23322..
Tang, S., Zhang, F., Chen, J., Wang, P., and Furukawa, Y. Mvdiffuuion: enabling holistic multi-view image genera- tion with correspondence aware diffusion. In Proceedings of the 37th International Conference on Neural Informa- tion Processing Systems, NIPS 23, 2023.

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In Guyon, I., von Luxburg, U.. Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan, S. V. N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017.
Wu, T., Zheng, C., and Cham, T.-J. Panodiffusion: 360- degree panorama outpainting via diffusion. In The Twelfth International Conference on Learning Representations, 2024b.
Xie, S., Zhang, 7.,, Lin, Z.., Hinz, T., and Zhang, K. Smart- brush: Text and shape guided object inpainting with diffu- sion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22428 22437, June 2023.
Wan, Z., Zhang, J., Chen, D., and Liao, J. High-fidelity pluralistic image completion with transformers. In Pro-
ceedings of the IEEE/CVF Interaational Conference on Computer Vision (ICCV). pp. (692-47711 October 2021.
Yao, K., Gao, P., Yang, X., Sun, J., Zhang, R., and Huang, K. Outpainting by queries. In Avidan, S., Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision ECCV 2022, pp. 153-166 Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20050-2.
Wan Abd Arif, W. N.. Wan Ahmad, W. F., Nordin, S. M., Abdullah, A., and Sivapalan, S. Designing 3 dimensional virtual reality using panoramic image. In Badioze Zaman, H., Robinson, P., Petrou, M., Olivier, P., Schröder, H., and Shih, T. K. (eds.), Visual Informatics: Bridging Research and Practice, pp. 404 414, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. ISBN 978-3-642-05036-7.
Ye, Y., Alshina, E.,, and Boyce, J. Algorithm descriptions of projection format conversion and video quality metrics in 360lib. In Joint Video Exploration Team of ITU-T SG, 07 2018.
Wang, C. Research on panoramic image processing technol- ogy based on virtual reality technology. In 2019 Interna- tional Conference on Virtual Reality and Intelligent Sys- tems (ICVRIS), pp. 55-58, 2019, doi: 10.1 10.11IS/ICSRIS 2019.00020
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., and Huang, T. S. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 2018.
Yurtsever, E., Lambert, J., Carballo, A., and Takeda, K. survey of autonomous driving: Common practices and emerging technologies. IEEE Access, 8:58443-58469, 2020. doi: 10.109/ACCESSS2200222881
Wang, H., Xiang, X., Fan, Y., and Xue, J. Customizing 360-degree panoramas through text-to-image diffusion models. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pp. 2024.
Zhang, C., Wu, Q., Cruz Gambardella, C., Huang, X., Phung. D., Ouyang, W., and Cai, J. Taming stable dif- fusion for text to 360e panorama image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
Wang, J.. Chen, Z., Ling, J., Xie, R., and Song, L. 360- degree panorama generation from few unregistered nfov images. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 6811 6821, 2023a.
Wang, S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y., Laszlo, S., Fleet, D. J., Soricut, R., Baldridge, J., Norouzi, M., Anderson, P., and Chan, W. Imagen editor and editbench: Advancing and evaluat- ing text-guided image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18359-18369, June 2023b.
Zhang, G. and Kou, X. Research and implementation of digital 3d panoramic visual communication technology based on virtual reality. International Journal of Commu- nication Systems, 35(5):48802, 2022.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In JEEE Inter- national Conference on Computer Vision (ICCV), 2023.
Wang, Y., Tao, X., Shen, X., and Jia, J. Wide-context semantic image extrapolation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Zhao, S.,, Cui, J., Sheng, Y., Dong, Y., Liang, X., Chang, E. I., and Xu, Y. Large scale image completion via co- modulated generative adversarial networks. In Interna- tional Conference on Learning Representations (ICLR), 2021.
WU, T., Li, X., Qi, Z., Hu, D., Wang, X., Shan, Y., and Li, X. Spherediffusion: Spherical geometry- aware dis- tortion resilient diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6126 6134, 2024a.
Zheng, C., Cham, T.-J., and Cai, J. Pluralistic image comple- tion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1438-1447, 2019.
12

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Zheng, C., Cham, T.-.,, Cai, J., and Phung, D. Bridging global context interactions for high-fidelity image com- pletion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11512-11522, June 2022.
Zheng. J., Zhang, J., Li, J., Tang, R.,, Gao, S., and Zhou, Z. Structured3d: A large photo-realistic dataset for struc- tured 3d modeling. In Proceedings of The European Conference on Computer Vision (ECCV), 2020.
13

---

Apherical-Nested Diffusion Model for Panoramic Image Outpainting
A. Analysis on the Structural Prior for the ERP Format
Compared to the equatorial region, rows closer the poles correspond to a smaller number of points on the sphere being mapped onto the fixed mw points in the ERP format. This discrepancy results in a one-to-many mapping, where multiple pixels in the ERP representation correspond to the same point the 3D space, leading to redundancy and distortion in these regions. Since the number of points per (mw) in ERP format is given by the width w, further analysis on (2) reveals that the resolution of the ERP format decreases the sampling density Ç in 3D space increases, the distribution of the uniform spherical noise more closely approximates i..d Gaussian noise within the ERP representation.
To comprehensively analyse the properties and the influence the hyperparameter Ç and the width w on the structural prior, we conducted an experiment and present the results Figure A.I. Note that images with resolution of 128 64 are consistent with the resolution of latent features in our SpND model, while images with a resolution of 1024 x 512 match the resolution of the images in datasets. By comparing the columns Figure A.I, it can be concluded that structural prior with higher resolution produced using the same Ç in the ERP format exhibit fewer rows that satisfy the i.i.d. property, consistent with our analysis. Additionally, by comparing rows with the same resolution in Figure A.1, it can be observed that noise in 3D space with smaller Ç tends to produce a structural prior with fewer rows satisfying the i.idd property at the same resolution. For Ç = 1000 and resolution of 128 x 64, the structural prior closely approximates i.i.d. Gaussian noise, with only the top and bottom rows significantly violating the i..dd property. Since the noise is adopted in the latent space in our SpND model, we adopt the noise distribution the ERP format with a resolution of 128 64 our structural prior.
To determine the optimal value of the hyperparameter Ç, we conducted comparison between the visualised noise and the latent features, illustrated in Figure A.2. This analysis led us select 30, as it most accurately preserves the structural characteristics of the feature map with a resolution of 128 x 64 while maintaining the highest efficiency in panorama outpainting.
128x
5
5=
45
==100
ç-= 1000
512x
=
11
Ç = 30
45
C = 100
C
1000
Figure A.I. Analysis on the impact of resolution (width ws height h) and sampling density ( 3D space to the structural prior for the ERP format. The structural prior is obtained by projecting the iidd Gaussian noise in 3D space with varying density C, into the ERP format. first row shows projections with a resolution of 128 x 64, while second row presents those with resolution of 1024 x 512.
(a) Noise with ( 15
(b) Noise with ( 30
(c) Real feature in latent space
(d) Noise with ( - 45
Figure A.2. Comparison of noise with different values of ( and the real feature in the latent space encoded by the VAE. The red line is used distinguish that approximate the i.i.d. property from those that do not, thereby highlighting regions where the noise significantly deviates from the characteristics of i..dd Gaussian noise.
Since specially designed noise pattern was employed structural prior knowledge for the ERP format, this study aimed to analyse its impact on the resulting outputs. As demonstrated in (a) and (c) of Figure A.3, changing the noise used to realize the structural prior P PERP does not affect the generated content. In contrast, changing the noise Z used for the pretrained diffusion model primarily impacts the generated content, as illustrated (b) and (d) in Figure. A.3. This indicates that the noise pattern associated with PERP primarily captures the spherical curvature, instead of controlling the specific details of the generated content. Note that the masked input identical in (a) and (b), as well as in (c) and (d) in Figure A.3.
14

---

Spherical-Nesee -Nested Diffusion Model for Panoramic Image Outpainting
(d)
Figure A.3. The impact of the structural prior PERP the outputs is examined by sampling different noise patterns in 3D space as the structural prior PERP, while keeping the feature consistent across rows (a) and (c). rows (b) and (d), the same 3D noise used for the structural prior, but the feature Z is varied. Rows (a) and (b) present samples from the Structured3D dataset, while rows (c) and (d) display samples from the Matterport3D dataset.
B. Repaint Operation with Mask Control
The repaint operation (Lugmayr al., 2022) involves adopting a mask to regulate specific regions within the generated sample while preserving the remaining areas unchanged. Specifically, the mask determines the regions that should remain unchanged by assigning a value of 1, while the remaining areas, represented by value of 0, are updated during the diffusion process. At each timestep f, our SpND model predicts the denoised sample Z prodicted which can be formulated as
2 predicted
1
Z predicted
1 - 0tt
predicted
VI Cze
Exp
(B.1)
where a; = I a; and The parameter a serves as a hyperparameter that controls the variance of the noise. Assume that ?) denotes for the i.i.d. Gaussian noise and Zo corresponds the ground-truth feature. Subsequently, to achieve mask control, update the denoised sample predicted given Z according to the formula as
Z predicted = m @ 2
predicted + (1 m) @ Z
(B.2)
Z 0 Zn v1 04-1 1)
(B.3)
where Zprodietor represents the outpainted feature and ( denotes element-wise multiplication. Note that (B.2) formulates the repaint process and (B.3) formulates the forward process of the diffusion model. The repaint operation, as defined (B.1),(B.3), ensures that the masked regions retain their original values, while the unmasked regions are selectively updated. Note that this effect of the repaint operation aligns seamlessly with the objective of the image outpainting task. By leveraging the repaint technique, our SpND model effectively performs outpainting for panoramic images while preserving the integrity of the original part of the image.
C. Experiment on Martain Image Outpainting
We further conducted experiments on a small dataset consisting of Martian panoramic images to evaluate the generalizability and the robustness of our SpND model. Specifically,, we collected 145 Martian panoramic images captured by the
15

---

Spherical-Nested Diffusion Model for Panoramic Image Outpainting
Perseverance rover, with resolution of 512 172, from publicly available sources. These images were padded with black borders final size of 512 256. As shown in Figure C.1, the proposed ppoDpr model effectively addresses the black borders at the top and bottom of the image, generating high-quality panoramic content for Martian images with limited number of training samples. Consistent with our other experiments, we employed the BLIP-2 model (Li et al., 2023) to generate the prompts for the Martian images. Note that the masked input can be obtained by projecting view images into the ERP format.
Masked Input Ground Truth
Output Image
Figure C.I. Qualitative results of panoramic image outpainting on the Martian dataset based on the given perspective view.
D. Experiment Input Versatility
In addition, we have demonstrated that our model can accommodate multiple partial-view and overlapping inputs by adjusting the training masks accordingly on the Matterport3D dataset. More specifically, for the multi-input scenario, we evaluated our SpND method based
viewpoints centred at (01 -90%, Φ1 -15)) and (02 90 d2 15%) within the ERP format, thus denoted Dual. Moreover, for the overlapping scenario, evaluated our SpND method on two viewpoints centered at (0) 20", Φ1 = 09) and (02 90% 02 15%) with overlapped regions, also using the ERP format, thus denoted as Overlapping. We then report the subjective results in Figure D. and Figure D.2. As illustrated in the figures, the proposed SpNDD model capable of achieving high-quality panoramic image completion under varying masks and multiple input conditions. We can conclude that our method can generalize to multiple overlapping and partial-view inputs.
Ground Truth
(a) Input Masked Image
bb) Dual
Ground Truth
(a) Input Masked Image
(b) Dual
Figure D.I. Qualitative results of SpND promge with multiple partial-view inputs, including ground truth, masked input, and images completed by our SpNDpr
16

---

Spherical-Nesee -Nested Diffusion Model for Panoramic Image Outpainting
Ground Truth
(a) Input Masked Image
(b) Overlapping
Ground Truth
(a) Input Masked Image
(b) Overlapping
Figure D.2. Qualitative results of SpND pronage with multiple overlapping inputs, including ground truth images, masked input images, and images completed by our SpNDpoont-
17


Reviews:
[{"review_id": "P0yM9uVu3h", "reviewer": "Reviewer_QS7j", "summary": "This work proposes to impose the sphere nature in the design of the diffusion model, such that the panoramic format is intrinsically ensured during the learning procedure, named the spherical-nested diffusion (SpND) model. In particular, the authors design to employ spherical noise in the diffusion process to address the structural prior, together with a newly proposed spherical deformable convolution (SDC) module to intrinsically learn the panoramic knowledge. Experimental results demonstrate the effectiveness of the proposed methods on various datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the valuable comments! \n- **Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef**\n- **Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253**\n\n> ### **1 Weakness 1: Novelty of Our SDC Layer**\n\nTo the best of our knowledge, the novelty of our SDC layer lies in the first successful attempt of proposing a new **basic convolution operation** to address the panoramic deformation, the key to achieve high-quality panoramic outpainting. Our spherical noise setting also contributes to our overall novelty in addressing this task. In contrast, existing methods, including those for other tasks such as panoramic image segmentation and depth estimation, typically address the panoramic deformation via soft loss regularization or feature re-sampling strategies. More specifically, Ref. [1] developed existing rectilinear architectures via pre-defined rectilinear-panoramic projection, to achieve 3D object detection and monocular depth estimation. Ref. [2] employed spherical resampling solely for the value feature within Transformer, to enhance panoramic image depth estimation. Ref. [3] added learnable positional encoding cues with existing features that are still processed by standard 2D convolution, for panoramic image outpainting. Ref. [4] regularized the network by spherical contrastive loss to effectively capture panoramic content for the segmentation of 360° images. However, Refs. [1, 2, 4] are incapable of generating panoramic content and thus unsuitable for the panoramic image outpainting task. More importantly, all the existing methods rely on either extra feature re-organization (e.g., Refs [2, 3]) or sphere-assisted losses (e.g., Refs. [1, 4])  to accommodate panoramic images. In contrast, our SDC layer essentially alters the basic convolution operations, which is fundamentally different from the existing strategies adopted for both outpainting and other panoramic-related tasks. Indeed, the fundamental improvement of our SDC layer on the convolution operation is much effective and straightforward to enforce geometry cues in addressing panoramic deformation, thus benefiting remarkable improved quality of outpainted images.\n\nRef. [1]: Eliminating the blind spot: Adapting 3D object detection and monocular depth estimation to 360 panoramic imagery.\n\nRef. [2]: Panoformer: Panorama transformer for indoor 360° depth estimation.\n\nRef. [3]: Cylin-Painting: Seamless 360 panoramic image outpainting and beyond.\n\nRef. [4]: Panoramic panoptic segmentation: Insights into surrounding parsing for mobile agents via unsupervised contrastive learning.\n\n> ### **2 Weakness 2: Quantitative Evaluations on Non-*i.i.d.* Noise in ERP Format**\n\nYes. Our Non-*i.i.d.* noise setting caters for the panoramic format during the outpainting, and we further conducted quantitative evaluations as suggested by the reviewer. More specifically, we further ablated two settings by varying sample density $\\zeta$, i.e., via $\\zeta=15$ and $\\zeta=45$, and report the results in Tab. 1 in the link. As shown in the table, our SpND method achieves optimal performance at $\\zeta = 45$ and $\\zeta = 30$, consistent with the analysis in Supplementary-A, where the ERP noise closely aligns with  the ground-truth features/images. Compared to $\\zeta = 45$, $\\zeta = 30$ offers the highest efficiency. Choosing $\\zeta = 15$ slightly degrades outpainting performance but still outperforms existing baselines. These results confirm the rationale and robustness of our proposed non-*i.i.d.* ERP noise, derived from *i.i.d.* spherical noise. \n\n> ### **3 Weakness 3: Limited Capability Regarding Input Versatility**\n\nWe agree with the reviewer that outpainting from multiple input views can further improve the practicality and applicability of our method. Indeed, our model can readily accommodate multiple partial-view and overlapping inputs by adjusting the training masks accordingly. More specifically, \n\n- For the multi-input scenario, we newly evaluated our SpND method based on two viewpoints centered at $(\\theta_1=-90^\\circ, \\phi_1=-15^\\circ)$ and $(\\theta_2=90^\\circ, \\phi_2=15^\\circ)$ within the equirectangular projection (ERP) format, thus denoted as  **Dual**.\n- For the overlapping scenario, we newly evaluated our SpND method based on two viewpoints centered at $(\\theta_1=20^\\circ, \\phi_1=0^\\circ)$ and $(\\theta_2=90^\\circ, \\phi_2=15^\\circ)$ with overlapped regions, also using the ERP format. We denote this as **Overlapping**.\n\nWe report the results in Tab. 2, together with the subjective results in Figs. 1-2 in the link. From this table and figure, we can conclude that our method can generalize to multiple overlapping and partial-view inputs. Note that due to limited rebuttal period, the model was trained by adequate convergence, and performance improvements can be expected with further training. Even though, our SpND method still achieves superior performances for the panoramic image outpainting task.", "claims_and_evidence": "See the weaknesses for more details.", "methods_and_evaluation_criteria": "See the weaknesses for more details.", "theoretical_claims": "See the weaknesses for more details.", "experimental_designs_or_analyses": "See the weaknesses for more details.", "relation_to_broader_scientific_literature": "It is highly related to the panoramic image outpainting literature.", "essential_references_not_discussed": "See the weaknesses for more details.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is generally well-structured, with illustrative diagrams effectively clarifying key concepts and model architectures. The motivation, design choices, and experiments are clearly presented, aiding reader comprehension.\n- By directly incorporating spherical constraints into the diffusion process, the method improves quantitative metrics (FID) compared to existing state-of-the-art approaches. \n\nWeaknesses:\n- Limited Novelty of the SDC Layer Compared to Existing Panoramic Vision Approaches: The paper proposes a Spherical Deformable Convolution (SDC) layer, aiming to better capture the spherical nature of panoramic images. Yet, the technical design closely resembles prior methods proposed in existing panoramic vision literature. For example, \"Eliminating the blind spot: Adapting 3d object detection and monocular depth estimation to 360 panoramic imagery\", \"Panoformer: panorama transformer for indoor 360° depth estimation\", \"Cylin-Painting: Seamless 360 panoramic image outpainting and beyond\", \"Panoramic panoptic segmentation: Insights into surrounding parsing for mobile agents via unsupervised contrastive learning\", etc. Clarifying how the modifications substantially differ from the above approaches through more explicit analyses and discussion would significantly strengthen the novelty claim.\n- Lack of Quantitative Evaluations on Non-i.i.d. Noise in ERP Format with Different Densities: The authors introduce the notion of non-i.i.d. spherical noise after ERP projection as a key component in their model. However, the paper does not quantitatively analyze how varying the sampling density impacts the performance or characteristics of the generated panoramic images. Although visual examples and qualitative insights are provided, rigorous numerical evaluations illustrating the effect of different densities and their impact on model performance (e.g., through ablation studies on various values) are missing. Such quantitative evaluations are crucial for understanding the practical robustness and sensitivity of the proposed method.\n- Limited Capability Regarding Input Versatility (Multiple Images Support): The proposed SpND model primarily focuses on scenarios where the panoramic image outpainting input is a single masked ERP image. However, many realistic applications, particularly those involving panoramic image reconstruction in AR/VR or autonomous driving scenarios, require handling multiple overlapping or partial-view inputs simultaneously. The current formulation and experimentation do not clearly address or demonstrate whether or how the proposed approach could effectively scale to or support scenarios involving multiple input images or viewpoints. Such limitations constrain the practicality and applicability of the method to real-world panoramic image generation tasks.", "other_comments_or_suggestions": "The ablation study lacks the important qualitative evaluations.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E27IgO1cLW", "reviewer": "Reviewer_GNkK", "summary": "This paper proposes a spherical reformulation of diffusion models for panoramic image outpainting. It focuses on the fact that the processing unit and the spatial stochastic patterns in plain diffusion models do not align well with panoramic image outpainting. To handle this problem, the paper proposes redesigning some components, such as spherical noise, spherical deformable convolution, and circular mask. This design choice is different from existing works, which utilize soft constraints. Experimental results clearly show that the proposed method outperforms the existing methods.\n\n## update after rebuttal\n\nAll the reviewers have acknowledged the contribution of the paper. Although I had some doubts regarding the broader impact, I also think that this is a valuable reformulation for an important use case. I maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer very much for the positive opinion and insightful suggestions. \n- **Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef**\n- **Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253**\n\n> ### **1 Weaknesses: Our Broader Impact.**\n\nYes. Although belonging to the broader fields of panoramic image processing, the panoramic outpainting task still possesses rich applications, especially for nowadays VR/AR applications such as autonomous driving, medical imaging, cultural heritage preservation, media and entertainments, to name but a few. The capability of completing panoramic content from masked views also allows for efficient compression and transmission of large-volume panoramic images, one of the key requirements to achieve meta-universe. Moreover, the newly proposed techniques, such as the spherical deformable convolution (SDC) operation and spherical noise, may also contribute to other panoramic-related tasks, including panoramic generation (as we have newly evaluated in **Rebuttal 5 to Reviewer vstk**) and content understanding. We believe that this direction is worth investigating with rich juicy unrevealed, in which our method paves a possible way to the field.\n\n> ### **2 Other Comments Or Suggestions: Typos**\n\nMany thanks! We will modify ''gird'' to ''grid'' and also carefully polish our wordings, by correcting grammatical errors and typos.\n\n> ### **3 Questions For Authors: Clarification on Equation (9)**\n\nMany thanks for the careful considerations. Yes, the outputs of our spherical net are integrated into the primary U-Net architecture, as depicted in Fig. 3 of our manuscript. Therefore, Equation (9) was misleading and is corrected by:\n$$\n\\epsilon\\_{\\boldsymbol{\\psi}} = f\\_\\text{SD}\\left[\\mathbf{Z}\\_{t}, \\mathbf{T}, f\\_\\text{PDN}\\left[\\mathbf{F}\\_\\text{pri}, \\mathbf{T}\\right]\\right]\n$$\nwhere $f\\_{\\text{SD}}$ denotes the pre-trained diffusion model and $f\\_{\\text{PDN}}$ denotes the spherical net, which is embedded into the diffusion model. Moreover, $\\mathbf{T}$ is the embedding of the input prompt and $\\epsilon\\_{\\boldsymbol{\\psi}}$ is the output of our SpND model. In this way, the spherical net can extract panoramic features that guide the pre-trained diffusion model for panorama outpainting.", "claims_and_evidence": "The claim that panoramic outpainting requires components that can naturally deal with spherical patterns, is plausible and convincing. Using tailored modules rather than soft constraints can obviously give better results.", "methods_and_evaluation_criteria": "The proposed modules or changes (e.g., spherical noise, spherical deformable convolution, and circular mask) are quite intuitive and well-designed. The paper provides most of the standard evaluation metrics for popular benchmark datasets.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "As mentioned above, experimental designs are sound.  Experimental results show that the proposed method outperforms existing methods, supporting the effectiveness of the proposed modules.", "relation_to_broader_scientific_literature": "I believe this paper proposes a well-designed method tailored to a particular problem (panoramic outpainting). For broader fields, the impact is somewhat limited. That being said, the problem being dealt with in this paper has some practical values for various applications (VR, AR, etc.). Overall, my opinion is that this is a good paper focusing on a specific problem.", "essential_references_not_discussed": "I believe the bibliography is thorough enough.", "other_strengths_and_weaknesses": "As I mentioned above, this is a good paper focusing on a specific problem. The proposed method is well designed and the performance is great, so I have no doubt in this part. I think the main question boils down to its broader impact. This method is somewhat limited to this particular problem, and the novelties in the method are not like \"groundbreaking.\" But at the same time, the problem itself has many valuable real-life applications. At the moment, I'm slightly leaning towards the positive side.", "other_comments_or_suggestions": "There are some typos, e.g., \"gird\" -> \"grid.\"", "questions_for_authors": "(9) is somewhat puzzling. Seeing Fig. 3, it seems the outputs of Spherical Net are \"embedded\" in the main U-Net. This seems slightly contradictory to (9). Could the authors clarify this point?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9hpQeAjTN3", "reviewer": "Reviewer_vstk", "summary": "This work proposes SpND, a novel pipeline for panoramic image outpainting based on diffusion model. The authors tackle the limitation of previous works in which the spherical nature of panoramic image is injected using soft regularization techniques, often failing to fully enforce the spherical constraint. To address this, this work aims to intrisically enforce the sphercial constraint during the generation through a sphercial-nested diffusion model, including the sphereical noise technique and a novel spherical deformable convolution (SDC) specifically designed to handle the spherical grid of panoramic images. SpND shows the state-of-the-art performance on panoramic image outpainting, surpassing previous works on image quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the positive comments and insightful suggestions. \n- **Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef**\n- **Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253**\n\n> ### **1 Claims and Evidence 4: Elaboration on One-to-Many Mapping**\n\nThe one-to-many mapping naturally arises from the ERP format of the spherical content. Pixels near the polar regions exhibit strong correlations in the ERP domain, reflecting spatial coherence in this projection. The consistent denoising procedure ensures coherent style and content by propagating the entire image across different views. Controlled by $\\zeta$, the one-to-many correspondence reaches its optimum when the correlations between noise and feature/image are similar, as explained in Supplementary-A. When $\\zeta$ is too small, blurry artifacts occur due to excessive pixel correspondence. Conversely, when $\\zeta$ is too large, spatial coherence decreases. This was verified through two new ablation studies, which is detailed in **Rebuttal 2 to Reviewer QS7j**.\n\n> ### **2 Methods and Evaluation Criteria 2: Applicability of RePaint to Panoramic Generation Methods**\n\nOutpainting can be achieved through text-to-panorama and diffusion-based generative methods, with each inference step integrated by the RePaint operation for denoising the full panoramic image. However, MVDiffusion generates $8$ perspective views via a multi-view diffusion model, which are then aggregated into a full panorama using Rodrigues rotation. This makes it infeasible to access each inference step for denoising, preventing the straightforward application of RePaint. In contrast, PanFusion accesses the full latent panoramic features at each inference step, enabling outpainting via RePaint. We modified PanFusion by adding RePaint and present new results in Fig. 3 in the link. As shown, PanFusion's outpainted panoramas exhibit repetitive patterns and poor quality due to latent feature rotation to preserve spherical continuity. In comparison, our SpND uses a masked image as input and consistently achieves the superior panoramic outpainting. \n\n> ### **3 Methods and Evaluation Criteria 3: Pixel-space Discrepancy by RePaint**\n\nYes. The discrepancy may exist in our SpND method. Most diffusion-based outpainting methods, particularly for panoramic images, operate within the latent space and suffer accuracy degradation in the input mask. For methods using RePaint, we calculated the PSNR$\\_\\text{cent}$ for the repainted area against the original image, with results reported in Tab. 3 in link. As shown, our SpND method achieves the highest accuracy and quality, which naturally results from our intrinsic SDC operation and spherical noise.\n\n> ### **4 Experimental Designs or Analyses 3: Clarifying FID and FID$_\\text{hori}$, with Enhanced Metrics**\n\nThe FID metric is widely used to evaluate generated image quality, and we preliminarily applied it to assess the performance of outpainted panoramic images. The FID$_\\text{hori}$ metric evaluates the FID values of 8 perspective view images of size $512 \\times 512$, following MVDiffusion. We also incorporated the panorama-specific FAED metric to evaluate the quality of panoramic images. Notably, FAED requires depth information, which is infeasible for outpainting. Therefore, we followed the PanFusion method to compute FAED without explicit depth input. The results, reported in Tab. 3, demonstrate that our SpND method outperforms state-of-the-art outpainting techniques.\n\n> ### **5 Relation to Broader Scientific Literature: Applicability to Panoramic Generation Tasks**\n\nYes, we believe injecting non-*i.i.d.* noise into existing panoramic generation methods could further improve detail retention and the quality of generated panoramic images, including text-to-image methods such as  MVDiffusion and PanFusion. Our SpND method can also generalize to generate images from text descriptions using a fully masked input. We evaluated this during the rebuttal and we achieved an FID score of 29.65 by retraining our method without other modifications. Subjective results are shown in Fig. 4. \n\n\n> ### **6 Essential References not Discussed: PanFusion and AOG-Net**\n\nYes, we have analyzed PanFusion by using the RePaint strategy, by **Rebuttal 2 above**. AOG-Net uses an autoregressive pipeline for 360° outpainting, utilizing feature remapping for panoramic content. We conducted a comparative analysis in Fig. 5 and Tab. 3 against AOG-Net using the Matterport3D dataset. From these, we conclude that our SpND method, by enforcing panoramic deformation through intrinsic convolution and spherical noise, consistently outperforms others in generating high-quality outpainted panoramic images.\n\n> ### **7 Other Comments or Suggestions: Training and Inference Procedures** \n\nYes. During inference, we applied iterative RePaint steps based on the trained SpND model, following other outpainting methods such as PanoDiffusion.", "claims_and_evidence": "1. The paper is well written and easy follow. The authors tackle a practically meaningful application of panoramic outpainting.\n\n2. The two main technical contributions - spherical noise and SDC - both have clear motivations and seem to be effective based on the qualitative/quantitative comparisons and ablations studies. \n\n3. The analysis on the spherical property of the ERP format well delivers why it is critical to sample the noise in the spherical spaces instead of directly from the planar space. \n\n4. Yet, it would be better if the authors provide a clear elaboration on the claim that \"one-to-many mapping inherently introduces spatial coherence in the EPR domain\". While these overlaps between multiple views can enforce the same content to be generated in the corresponding pixels, it is unclear how this can also lead to a globally coherent image. Is it because the coherent content/style is somehow propagated across different views during denoising? Moreover, have the authors observed any downsides of this one-to-many mapping, such as blurry/over-smoothed image near the overlapping regions?", "methods_and_evaluation_criteria": "1. The introduction of SDC seems to be quite novel and an approach for handling the spherical nature of panoramic images. Adopting the idea of deformable convolution specifically for the spherical grid seems to be a reasonable method that takes into account the key characteristic of panoramic images.\n\n2. While this work aims for panoramic \"outpainting\", the design choice of SpND would be better justified if the authors had provided discussions that compare with the panoramic generation works. For instance, MVDiffusion [1] and PanFusion [2] both suggest novel architectures for text-to-panorama generation. Can we also apply RePaint to such models to obtain panoramic image outpainting? If it is not possible to apply the same approach to these works, can the authors please explain which aspect of SpND allows this in contrast?\n\n3. Moreover, based on the pretrained SpND model, the RePaint technique is utilized to perform seamless outpainting based on the input mask. Since the original RePaint is based on \"pixel-space\" diffusion models, it doesn't introduce any discrepancy between the input mask and the space in which the denoising is performed. In constrast, since this work uses a latent diffusion model, I am concerned if the discrepancy between the input mask (pixel-space) and the actual inpainting operations (latent-space) could lead to quality degradation. If this is not the case, it would be nice if the authors can explain why this discrepancy does not necessarily lead to degradation.\n\n[1] MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion, Tang et al., NeurIPS 2023\n\n[2] Taming Stable Diffusion for Text to 360◦ Panorama Image Generation, Zhang et al., CVPR 2024", "theoretical_claims": "This work does not provide theoretical claims and instead focuses on empirical evidence for the effectiveness of SpND.", "experimental_designs_or_analyses": "1. The paper provides detailed explanations on both the evaluation settings and implementation details of SpND.\n\n2. Both quanitative and qualitative comparisons are presented clearly, showing that SpND outperforms previous methods on panoramic outpainting.\n\n3. I am confused by the choice of using FID and FID_hori for image quality evaluation. To measure the quality and realism of the panoramic image, it seems more intuitive to either measure the FID of the projected perspective views or use a panorama-specific metric such as FAED [1].\n\n4. The paper provides a thorough ablation study of its core components, which shows that the design choice for the pipeline works according to their intentions.\n\n[1] Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning, Oh et al., ECCV 2022", "relation_to_broader_scientific_literature": "It would be interesting to see whether the proposed idea of inject a structural prior by sampling from spherical noise could be extended to pure generation of panoramic images. For instance, can it lead to more accurate panoramic images (or videos) generated just from text prompts? Applying a similar idea to text-to-panorama methods like MVDiffusion [1] and PanFusion [2] would be an interesting future work.\n\n[1] MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion, Tang et al., NeurIPS 2023\n\n[2] Taming Stable Diffusion for Text to 360◦ Panorama Image Generation, Zhang et al., CVPR 2024", "essential_references_not_discussed": "As mentioned above, an important related work PanFusion [1] is missing. Since this is not an outpainting method but rather a pure text-based generation method, it doesn't seem necessary to make quantitative comparions. But it would be nice to discuss why this previous approach is not sufficient for panoramic outpainting.\n\nMoreover, AOG-Net [2] seems to be a more directly related work, and possibly a valid baseline.\n\n[1] Taming Stable Diffusion for Text to 360◦ Panorama Image Generation, Zhang et al., CVPR 2024\n\n[2] Autoregressive Omni-Aware Outpainting for Open-Vocabulary 360-Degree Image Generation, Lu et al., AAAI 2024", "other_strengths_and_weaknesses": "The strengths and weaknesses are discussed in the above sections.", "other_comments_or_suggestions": "When first reading the paper, it was a bit confusing whether the whole pipeline is trained end-to-end and the inference is done in a simple feed-forward manner. However, as far as I understand, the inference for outpainting is done by applying iterative RePaint steps using the pretrained diffusion model. It would have been better if that was emphasized.", "questions_for_authors": "Some questions are already included in the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EpunTeNooU", "reviewer": "Reviewer_RXh3", "summary": "This paper presents a new diffusion-based panoramic image outpainting control net model that incorporates  1) spherical noise as structural prior into the equirectangular projected image for better handling the ERP distortion and 2) spherical deformable convolution layers to handle the varying CNN reception field on the ERP image. The experimental results demonstrate the promising panorama outpainting quality compared to prior works.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your positive opinion and valuable comments! \n- **Tab.1 - 4 are in https://anonymous.4open.science/api/repo/tabl/file/T.pdf?v=5327f2ef**\n- **Fig.1 - 9 are in https://anonymous.4open.science/api/repo/figu/file/F.pdf?v=2d1be253**\n\n> ### **1 Supplementary Material: Further Analysis on Fig. A. 3**\n\nThe randomness in our SpND model consists of two components: the structural prior $\\mathbf{P}\\_\\text{ERP}$ from spherical noise and the diffusion noise $\\mathbf{Z}$. Fig. A. 3 in Supplementary-A illustrates the impact of these two components, in which spherical noise minimally affects the semantic content of generated images, maintaining structural consistency. In contrast, diffusion noise drastically changes the output, creating distinct visual features. This indicates spherical noise serves as a structural prior, preserving geometric constraints, while diffusion noise controls semantic content generation. These observations support our formulation of $\\mathbf{P}\\_\\text{ERP}$ as a mechanism to regularize the latent space without disturbing semantic attributes.\n> ### **2 Relation to Broader Scientific Literature: Relation to Non-Isotropic Diffusion Models**\n\nThe noise in our SpND method follows non-isotropic Gaussian distributions, with the non-isotropic property arising from the ERP operation applied to spherical noise. In contrast, Ref. [1] introduces a critically-damped Langevin diffusion process to enhance network convergence by adding an auxiliary velocity variable, while Ref. [2] generalizes score-based denoising diffusion models with non-isotropic Gaussian Free Field (GFF) noise, improving generation quality on CIFAR10. However, these methods, with their particular non-isotropic settings, are not designed for panoramic image outpainting. For instance, GFF noise in Ref. [2] defines the covariance between sampled values $f(\\theta, \\phi)$ and $f(\\theta', \\phi')$ as:  $$\\left(\\text{Cov}(f(\\theta,\\phi), f(\\theta',\\phi')) \\approx -\\log\\|(\\theta,\\phi)-(\\theta',\\phi')\\| \\right) + C,$$where $C$ is a constant. In contrast, for our spherical noise, when $\\phi=\\frac{\\pi}{2}$, the covariance simplifies to: $$\\text{Cov}(f(\\theta,\\frac{\\pi}{2}), f(\\theta',\\frac{\\pi}{2})) = \\sigma^2.$$Thus, the constant covariance $\\sigma^2$ contradicts the logarithmic dependence required by Ref. [2], demonstrating that the non-isotropic noise in our SpND method cannot be generalized by existing models. This highlights the novelty and contribution of our approach in panoramic outpainting.\n> ### **3 Weakness 1: More Visual Results**\n\nYes, outpainting tasks heavily rely on perceptual evaluations. We have added more visual examples for all comparisons discussed in our experiments, including the main comparisons, ablation study and view-image outpainting experiments, in Figs. 5-8 in the link. The results contain ground truth, masked input, all baseline methods, and our SpND method for both the Matterport3D and Structured3D datasets, which consistently demonstrate the superior performance of our SpND method.\n> ### **4 Weakness 2: Evaluations on Distorted Masks**\n\nTo further evaluate our method on different masks, we conducted additional experiments on masks near the poles, where spherical distortion becomes evident. Specifically, we projected a $256 \\times 256$ image onto the ERP format at $\\theta=90^\\circ, \\phi=60^\\circ$ to create a highly-distorted mask near the pole. We then retrained our SpND method, along with the second and third best baseline methods (PanoDiff and PanoDiffusion). Results reported in Tab. 4 and Fig. 9, confirm the superior performance of our SpND method, highlighting the advantages of spherical noise and the SDC operation in accommodating panoramic deformation.\n> ### **5 Questions For Authors 1: Mask Location and Evaluation Angle**\n\nYes, in our original experiments, the images were masked only around the center during training, corresponding to Fig. 4-(a) in our manuscript. For the masked input, the region containing image content spans an elevation angle range of approximately $[-45^\\circ, 45^\\circ]$, within a full image that has an elevation angle range of $[-90^\\circ, 90^\\circ]$. We further evaluated our SpND method's ability to generalize to various non-centered masks, as detailed in **Rebuttal 6 below**.\n> ### **6 Questions For Authors 2: Generalization to Varying Masks**\n\nYes, our SpND model can generalize to varying mask formats. We re-trained our SpND method with the entire input masked and show the subjective results in Fig. 4. While performance was reduced, our SpND method achieved an FID of 29.65 and successfully performed panoramic generation, confirming its generalization ability. Additionally, our SpND method is highly flexible with multiple input views and we refer to our **Rebuttal 4 above** and our **Rebuttal 3 to Reviewer Qs7j**. \n> ### **7 Questions For Authors 3: More Possible Applications**\n\nMany thanks! For more possible applications, we refer to our **Rebuttal 1 to Reviewer GNkK**.", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, this submission largely follows the evaluations from the prior work.", "theoretical_claims": "There’s no proof for theoretical claims.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound to me.", "relation_to_broader_scientific_literature": "The proposed method of injecting spherical structural noise to the original diffusion noise might be relevant to the non-isotropic diffusion model, for example:\n* “Score-based generative modeling with critically-damped langevin diffusion”\n* “Score-based denoising diffusion with non-isotropic gaussian noise models”", "essential_references_not_discussed": "References are sufficient.", "other_strengths_and_weaknesses": "Strengths:\n* The introduced spherical noise as a structural prior for panorama diffusion denoising is interesting and novel.\n* The design of spherical deformable convolution is reasonable and effective.\n\nWeaknesses:\n* Since the outpainting tasks largely rely on perceptual evaluations, the author should provide more visual results to assist readers’ perceptual comparison. Besides, there’s no visual result in the ablation study. Providing the reference ground truth would also be helpful.\n* Since the evaluated mask region is located at the center of the ERP panorama, input image patches do not contain too much distortion. This evaluation setting may not show the advantage of spherical noise and SDC processed on meaningful pixels.", "other_comments_or_suggestions": "None", "questions_for_authors": "* How are the panorama images masked during training? Are they only masked around the center? What about the range of elevation angle? \n* How does this model generalize to varying masks? If the entire input is masked, will this model work properly? \n* It would be better if the authors could include more applications of the proposed method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a well-motivated and technically sound contribution to panoramic image generation that gradually becomes a popular research topic. Specifically, the proposed spherical noise and deformable convolution can benefit the panoramic images' ERP distortions. Through rebuttal efforts, all the reviewers positively commented on the submission and suggested that the authors clarify and emphasize distinctions from prior works in the final revision.

After reading the paper and rebuttal, the AC votes for weak acceptance of its novel architecture design, extensive experimental results, and impact on the specific area. 

### Method 

By introducing spherical noise and a spherical deformable convolution (SDC) layer, the model intrinsically captures the spherical geometry of panoramic images. Reviewers appreciated the sound motivation and architectural clarity.

### Experiment

SpND outperforms state-of-the-art baselines such as PanoDiffusion and AOG-Net, demonstrating significant improvements in FID and FAED across multiple datasets. Both qualitative and quantitative results are convincing.

###  Rebuttal Period

Authors addressed all reviewer concerns with additional experiments, ablation studies, visualizations, and theoretical clarifications, leading to improved reviewer ratings.
Decision: Accept (poster)