PanoDiffusion: 360-degree Panorama Outpainting via Diffusion
Authors: Tianhao Wu
Chuanxia Zheng
Tat-Jen Cham
Venue: ICLR 2024 poster (2024)
Keywords: Panorama Outpainting
Latent Diffusion
Multi-modal Generation
Primary area: generative models
TL;DR: A bi-modal latent diffusion structure using RGB-D panoramic for training and creatively introduce progressive camera rotations during each denoising step to achieving panorama wraparound consistency.

Abstract: Generating complete 360\textdegree{} panoramas from narrow field of view images is ongoing research as omnidirectional RGB data is not readily available. Existing GAN-based approaches face some barriers to achieving higher quality output, and have poor generalization performance over different mask types. In this paper, we present our 360\textdegree{} indoor RGB panorama outpainting model using latent diffusion models (LDM), called PanoDiffusion. We introduce a new bi-modal latent diffusion structure that utilizes both RGB and depth panoramic data during training, which works surprisingly well to outpaint depth-free RGB images during inference. We further propose a novel technique of introducing progressive camera rotations during each diffusion denoising step, which leads to substantial improvement in achieving panorama wraparound consistency. Results show that our PanoDiffusion not only significantly outperforms state-of-the-art methods on RGB panorama outpainting by producing diverse well-structured results for different types of masks, but can also synthesize high-quality depth panoramas to provide realistic 3D indoor models.

Published as a conference paper at ICLR 2024
PANODIFFUSION: 360-DEGREE PANORAMA OUT- PAINTING VIA DIFFUSION
Tianhao Wult, Chuanxia Zheng² & Tat-Jen Cham
LS-Lab, 'Nayyang Technological University
tianhao001 .e..ee.......................... ntu. edu. sg, jshanhannt edu. University of Oxford
exzhengerobot ox accuk
ABSTRACT
Generating complete 360 panoramas from narrow field of view images is on- going research as omnidirectional RGB data not readily available. Existing GAN-based approaches face some barriers to achieving higher quality output, and have poor generalization performance over different mask types. this paper, we present our 360 indoor RGB-D panorama outpainting model using latent dif- fusion models (LDM), called PanoDiffusion. We introduce a new bi-modal la- tent diffusion structure that utilizes both RGB and depth panoramic data during training, which works surprisingly well to outpaint depth-free RGB images dur- ing inference. We further propose novel technique of introducing progressive camera rotations during each diffusion denoising step, which leads to substan- tial improvement achieving panorama wraparound consistency. Results show that our PanoDiffusion not only significantly outperforms state-of-the-arr methods on RGB-D panorama outpainting by producing diverse well-structured results for different types of masks, but can also synthesize high-qualit, depth panoramas to provide realistic 3D indoor models.
INTRODUCTION
Omnidirectional 360 panoramas serve as invaluable assets in various applications, such as light- ing estimation (Gardner et al.) 2017; 2019) [Song & Funkhouser] 2019) and new scene synthesis (Somanath & Kurz 2021) in the Augmented and Virtual Reality (AR VR). However, an obvi- ous limitation is that capturing, collecting, and restoring dataset with 360 images is a high-effort and high-cos! undertaking (Akimoto et al. 2019 2022), while manually creating 3D space from scratch can be demanding task (Lee et al. 2017 Choi et al. 2015; Newcombe et al. 2011).
(a) Masked input
(b) Ours PanoDiffusion
(c) BIPS
(d) OmniDreamer
Figure 1: Example results of 360 Panorama Outpainting various masks. Compared to BIPS (Oh et al. 2022) and OmniDreamer (Akimoto al. 2022), our model not only effectively generates semantically meaningful content and plausible appearances with many objects, such as beds, sofas and TV's, but also provides multiple and diverse solutions for this ill-posed problem. (Masked regions shown in blue for better visualization. Zoom in to the details.)

---

Published as a conference paper at ICLR 2024
To mitigate this dataset issue, the latest learning methods (Akimoto et al. 2019 Somanath & Kurz, 2021 Akimoto et al. 2022; Oh et al. 2022) have been proposed. with a specific focus on generating onnidirectional RGB panoramas from narrow field of view (NFoV) images. These methods are typically built upon Generative Adversarial Networks (GANs) (Goodfellow et al., 2014), which have shown remarkable success in creating new content. However, GAN architectures face some notable problems, including 1) mode collapse (seen in Fig. [[)),, 2) unstable training (Salimans al. 2016), and 3) difficulty in generating multiple structurally reasonable objects (Epstein et al. These limitations lead to obvious artifacts in synthesizing complex scenes (Fig.
2022
The recent endeavors of (Lugmayr al.) 2022 Li al. 2022; Xie al., 2023 Wang et al. 2023)
directly adopt the latest latent diffusion models (LDMs) (Rombach et al. 2022) in image impainting tasks, which achieve a stable training of generative models and spatially consistent images. How- ever, specifically for 360 panorama outpainting scenario, these inpainting works usually lead to grossly distorted results. This because: 1) the missing (masked) regions in 360 panorama out- painting is generally much larger than masks in normal inpainting and 2) it necessitates generating semantically reasonable objects within a given scene, as opposed to merely filling in generic back- ground textures in an empty room (as shown in Fig. (c)). To achieve this, we propose alternative method for 360 indoor panorama outpainting via the latest latent diffusion models (LDMs) (Rom-) bach al. 2022), termed as PanoDiffusion. Unlike existing diffusion-based inpainting methods, we introduce depth information through a novel bi-modal latent diffusion structure during the training, which is also significantly different from the latest concurrent works (Tang et al., [023] Lu et al.) 2023) that aims for text- ext-ggided 360 panorama image generation. Our key motivation for doing so that the depth information is crucial for helping the network understand the physical structure of objects and the layout of the scene (Ren et al. 2012). It is worth noting that our model only uses par- tially visible RGB images as input during inference, without requirement for any depth information, yet achieving significant improvement on both RGB and depth synthesis (Tables I and 2
Another distinctive challenge in this task stems from the unique characteristic of panorama images: 3) both ends of the image must be aligned to ensure the integrity and wraparound consistency of the entire space, given that the indoor space lacks a definitive starting and ending point. To enhance this property in the generated results, we introduce two strategies: First, during the training process, camera-rone -rotatinrotation approach is employed to randomly crop and stitch the images for data augmenta- tion. It encourages the networks to capture information from different views in a 360 panorama. Second, a two-end alignment mechanism is applied at each step of the inference denoising process (Fig. 4. which explicitly enforces the two ends of an image be wraparound-ccosiitenn.
We evaluate the proposed method on the Structured3I dataset (Zheng et al. 2020). Experimental results demonstrate that our PanoDiffusion not only significantly outperforms previous state-of-the- art methods, but is also able to provide multiple and diverse well-structured results for different types of masks (Fig. In summary, our main contributions are as follows:
A new bi-modal latent diffusion structure that utilizes both RGB and depth panoramic data to better learn spatial layouts and patterns during training. but works surprisingly well to outpaint normal RGB-D panoramas during inference, even without depth inpul;
A novel technique of introducing progressive camera rotations during each diffusion de- noising step, which leads to substantial improvement in achieving panorama wraparound consistency:
With either partially fully visible RGB inputs, our PanoDiffusion can synthesize high- quality indoor RGB-D panoramas simultancously to provide realistic 3D indoor models.
2 RELATED WORK
2.1 IMAGE INPAINTING/OUTPAITTINN
Driven by advances in various generative models, such as VAEs (Kingma & Welling 2014) and
GANs (Goodfellow et al. 2014), a series of learning- approaches (Pathak et al. [2016] lizuka
et al. 2017 Yu et al. 2018 Zheng et al. 2019 Zhao et al. 22200 Wan et al. |021 [Zeng al. 2022)
have been proposed to generate semantically meaningful content from a partially visible image.
More recently, ofate-of-the-art methods (Lugmayr et al. 2022 Li et al. 2022 Xie et al.
2023
2

---

Published as a conference paper at ICLR 2024
Training Stage
Refine Stage
Proses
Zegh
Argb
D
82 AA
E2
Arght
Rrgh
%
(b) Inforence Stage
SR
Aryb
Zrgb
Zrghh
2 38 Denoiiang Chipaisting
Argba
Output
Figure 2: The overall pipeline of proposed PanoDiffusion method. (a) During training, the model is optimized for RGB-D panorama synthesis, without the mask. (b) During inference, however, the depth information is longer needed for masked panorama outpainting. (c) Finally, super-resolution model is implemented to further enhance the high-resolution outpainting. We only show the input/output of each stage and omit the details of circular shift and adding noise. Note that the VQ-based encoder-decoders are pre-trained in advance, and fixed in the rest of our framework.
Wang et al., 2023) directly adopt the popular diffusion models (Rombach et 2022 for image inpainting. achieving high-quality completed images with consistent structure and diverse content. However, these diffusion-basedt models either focus on background inpainting, or require input text as guidance to produce plausible objects within the missing regions. This points to existing gap in achieving comprehensive and contextually rich impainting/otppainingg results across wider spectrum of scenarios, especially in the large scale 360 Panorama scenes.
2.2 360 PANORAMA OUTPAINTING
Unlike NFoV images, 360 panorama images are subjected to nonlinear perspective distortion, such equirectangulat projection. Consequently, objects and layouts within these images appear sub- stantially distorted, particularly those closer to the top and bottom poles. The image completion has not only preserve the distorted structure but also ensure visual plausibility, with the additional at both ends. Previous endeavors (Akimoto et al. 2019 Somanath Kurz 2021) mainly focused on deterministic completion of 360 RGB images, with BIPS (Oh et al. 2002) further extending this to RGB-D panorama synthesis. In order to generate diverse results (Zheng et al. 2019 2021), various strategies have been employed. For instance, SIG-SS (Hara et al. (222)) leverages a symmetry-informed CVAE, while OnniDreamer (Akimoto sampling. In contrast, our PanoDiffusion built upon DDPM, wherein each reverse diffusion step inherently introduces stochastic. naturally resulting in multiple and diverse results. Concurrently with our work, MVDiffusion (Tang et al., 2023) gen- erates panorama images by sampling consistent multi-view images, and AOGNet (Lu et al. 2023)
requirement of saappurrunndcoosisstern
al. 2022) employs transformer-base
does 360 outpainting through an autoregressive process. Compared to the concurrent models, our PanoDiffusion excels in generating semantically multi-objects for large masked regions, without the need of text prompts. More importantly, PanoDiffusion capable of simultaneously generating the corresponding RGB-D output, using only partially visible RGB images as input during the inference.
METHODS
Given a 360 image E HxxxC degraded by number of missing pixels to become a masked image Fm, our main goal is to infer semantically meaningful content with reasonable geometry for the missing regions, while simultaneously generating visually realistic appearances. While this task conceptually similar to conventional learning- image inpainting, it presents greater chal- lenges due to the following differences: 1) our output is 360" RGB-D panorama that requires wraparound consistency; 2) the masked/missing areas are generally much larger than the masks in traditional inpainting; our goal is to generate multiple appropriate objects within scene. instead of simply filling in with the generic background; 4) the completed results should be structurally plausible, which be reflected by reasonable depth map.

---

Published as a conference paper at ICLR 2024
rotated
Z
ZT
Random
Diffusion
Data Loader
Denoising
Training Stage
Mask
2t
pisible
2t
mixed
Inference Stage
90'
Z
denoise
RGB channel
ZE
Musk Inv.
Ze invisible
Depth channel
Next Iteration
Camera rotation
Zt-1
Figure 3: Our LDM outpainting structure with camera rotation mechanism. During training (a), we randomly select a rotation angle to generate new panorama for data augmentation. During inference (b), sample the visible region from the encoded features (above) and the invisible part from the denoising output (below). The depth map is not needed, and is set to random noise. At each denoising step, crop equiivallll area of the intermediate result from the right and stitch it to the left, denoted by the circle following z ziized this strongly improves wraparound consistency.
To tackle these challenges, we propose a novel diffusion-based framework for 360 panoramic out- painting, called PanoDiffusion. The training process, as illustrated in Fig. _[a), starts with two branches dedicated to RGB æ and depth d, information. Within each branch, following (Rombach) et al., 2022), the input data is first embedded into the latent space using the corresponding pre- trained VQ model. These representations are then concatenated to yield Erghd. which subsequently undergoes the forward diffusion step obtain 27. The resulting 2T is then subjected to inverse denoising, facilitated by a trained UNet, ultimately returning to the original latent domain. Finally, the pre-trained decoder is employed to rebuild the completed RGB-D results.
During inference, our system takes masked RGB image as input and conducts panoramic outpaint- ing. It is noteworthy that proposed model does not inherently require harder-to-acquue depth maps as input, relying solely on a partial RGB image (Fig. bb)). The output further super-resolved into the final image refinement stage (Fig. ((c)..
3.1 PRELIMINARIES
Latent Diffusion. Our PanoDiffusion builds upon the latest Latent Diffusion Model (LDM) (Rom-) bach et al. 2022), which executes the denoising process in the latent space of autoencoder. This design choice yields a twofold advantage: it reduces computational costs while maintaining high level of visual quality by storing the domain information in the encoder E() and decoder D(-). During the training, the given image Zo is initially embedded to yield the latent representation 20 (xo). which is then perturbed by adding the noise a Markovian manner:
q(z 21- = Nzz /1 3, 3yd),
where t [1. T is the number of steps in the forward process. The hyperparameters B, denote the noise level at each step For the denoising process, the network in LDM is trained to predict the noise as proposed in DDPM et al. 2020), where the training objective be expressed
t) = Diffusion Outpainting. The existing pixel-level diffusion inpainting methods (Lugmayr et al.
L B.e,,,e.....
(2)
2022 Horita et al. 2022 are conceptually similar to that used for image generation, except Tt
incorporates partially visible information, rather than purely sampling from Gaussian distribution during the inference. In particular, let Tig denote the original image in step O, while x vitisible = m@ and mi mosaibbe (1 m) @ z5 contain the visible and missing pixels, respectively. Then, shown
in Fig. the reverse denoising sampling process unfolds as follows:
Itt ossible ~ g( 11)-))
(3)
x invisible ~ po (zt |xtt)
(4)
Ft-1 visible + (1 m) 2 x
x
invisible
(5)
4

---

Published as a conference paper at ICLR 2024
Rotate 90
90
Deooising Process
Figure 4: An example of our two-end alignment mechanism. During inference, we rotate the scene for 90" in each denoising step. Within a total of 200 sampling steps, our PanoDiffusion will effectively achieve wraparound consistency.
Here, 9 is the forward distribution in the diffusion process and pe is the inverse distribution. After T iterations, Zo is restored to the original image space.
Relation to Prior Work. contrast to these inpainting methods at pixel-level, our PifoDiffusion builds upon the LDM. Despite the fact that the original LDM provided the ability to inpainting images, such inpainting focuses on removing objects from the image, rather than generating a variety of meaningful objects in panoramic outpainting. In short, the Hy is embedded into the latent space, yielding = (xx), while the subsequent sampling process follows the equations (3)-(5). The key motivation behind this is perform our task higher resolution 512x 1024 panoramas. More importantly, we opt to go beyond RGB outpainting, and to deal with RGB-D synthesis (Sec. 3.3
which is useful for downstream tasks in 3D reconstruction. Additionally, existing approaches can not ensure the wraparound consistency during completion, while our proposed rotational outpainting mechanism in Sec. 3.2 significantly improves such a wraparound consistency.
3.2
WRAPAROUND CONSISTENCY MECHANISM
Camera Rotated Data Augmentation. It is expected that the two ends of any 360 panorama should be seamlessly aligned, creating a consistent transition from one end to the other. This is especially crucial in applications where smooth visual experience is required, such as 3D recon- struction and rendering. To promote this property, we implement a circular shift data augmentation, termed camera-rotation, to train our PanoDiffusion. As shown in Fig. 3(a), we randomly select a rotation angle, which is subsequently employed to crop and reassemble image patches, generating new panorama for training purposes.
Two-Ends Alignment Sampling. While the above camera-rotation technique can improve the model's implicit grasp of the wraparound consistency using the augmentation of data examples, may not impose strong enough constraints on wraparound aligmment of the results. Therefore. in the inference process, we introduce novel two-end aligument mechanism that can be seamlessly integrated into our latent diffusion outpainting process. In particular, the reverse denoising process within the DDPM is characterized by multiple iterations, rather than single step. During each iter- ation, we apply the camera-rotation operation, entailing 90º rotation of both the latent vectors and mask, before performing a denoising outpainting step. This procedure more effectively connects the two ends of the panorama from the previous step, resulting in significant improvement in visual results (as shown in Fig. 8). Without changing the size of the images, generating overlapping con- tent, introducing extra loss functions, provide 'hints' to the model by rotating the panorama horizontally, thus enhancing the effect of alignment at both ends (examples shown in Fig.
3.3 BI-MODAL LATENT DIFFUSION MODEL
In order deal with RGB-D synthesis, one straightforward idea could be to use Depth as an explicit condition during training and inference, where the depth information may be compressed into latent space and then introduced into the denoising process of the RGB images via concatenation or cross- attention. However, (as shown in Fig. I.). Alternatively, using two parallel LDMs to reconstruct Depth and RGB images separately, together with a joint loss, may also appear be an intuitive solution. Nonetheless, this idea presents significant implementation challenges due to the computational resources required for multiple LDMs.
found that such approach often leads to blurry results in our experiments
Therefore, we devised a bi-modal latent diffusion structure to introduce depth information while generating high-quality RGB output. It is important to note that this depth information is solely
5

---

Published as a conference paper at ICLR 2024
(a) NFoV
(b) Camera
(c) Random
(d) Layout
Figure 5: Examples of various mask types. See text for details.
necessary during the training phase. Specifically, we trained two VAE models independently for RGB and depth images, and then concatenate Frgh E Rxxxxx with zdepth E [gxxxx at the latent level to get Erghd E <xxwxx The training of VAEs is exactly the same as in (Rombach et al. 2022) with downsampling factor f=4. Then we follow the standard process to train an unconditional DDPM with Erghd via variant of the original LDM loss:
RGB MOBB-D
E P 1), llle (2,, cllE), 2rghut & (x) 2 (d.,
Reconstructed RGB-D DD images can be obtained by decoupling Erghd and decoding. It is important to note that during training, we use the full RGB-D image input, without masks. Conversely, during the inference stage, the model can perform outpainting of the masked RGB image directly without any depth input, with the fourth channel of 2rghd replaced by random noise.
3.4 REFINENET
Although mapping images to a smaller latent space via an autoencoder prior to diffusion can save training space and thus allow larger size inputs, the panorama size 512x 1024 is still a heavy burden for LDM (Rombach et al. 2022). Therefore, we adopt two-stage approach to complete the outpainting task. Initially, the orgginal input downscaled to 256x512 as the input to the LDM. Correspondingly, the image size of the LDM output is also 256x512. Therefore, additional mod- ule is needed to upscale the output image size to 512x 1024. Since panorama images are distorted and the objects and layouts do not follow the regular image patterns, we trained a super-resolution GAN model for panoramas to produce visually plausible results at a higher resolution.
EXPERIMENTS
4.1 EXPERIMENTAL DETAILS
Dataset. We estimated our model on the Structured3D dataset (Zheng et al. 2020), which provides 360 indoor RGB-D data following equirectangular projection with a x12x 1024 resolution. We split the dataset into 16930 train, 2116 validation, and 21 17 test instances.
Metries. For RGB outpainting, due to large masks, we should not require the completed image to be exactly the same the original image, since there many plausible solutions (e.g. new furniture and ornaments, and their placement). Therefore, we mainly report the following dataset- level metrics: 1) Fréchet Inception Distance (FID) (Heusel et al. (017). 2) Spatial FID (sFID) (Nash) et al. 2021). 3) density and coverage (Naeem et al.) 2020). FID compares the distance between distributions of generated and original images in a deep feature domain, while sFID variant of FID that uses spatial features rather than the standard pooled features. Additionally, density reflects how accurate the generated data to the real data stream, while coverage reflects how well the generated data generalizes the real data stream. For depth synthesis, we use RMSE, MAE, AbsREL, and Deltal ..55 implemented in (Cheng al.) 2018 Zheng al.) 2018). which are commonly used to measure the accuracy of depth estimates. Implementation details can be found in section
Mask Types. Most works focused on generating omnidirectional images from NFoV images (Fig. 5ja)). However, partial observability may also occur due sensor damage in 360 cameras. Such masks can be roughly simulated by randomly sampling a number of NFoV camera views within the panorama (Fig. (())). We also experimented with other types of masks, such as randomly generated regular masks (Fig- (()). Finally, the regions with floors and ceilings in panoramic images are often less interesting than the central regions. Hence, we also generated layout masks that muffle all areas except floors and ceilings, to more incisively test the model's generative power (Fig, ((d)..

---

Published as a conference paper at ICLR 2024
(a) Ground Truth
(b) Masked Input
(c) PanoDiffusion (RGB) (d) PanoDiffusion (RGB-)
(e) BIPSECCV
(1) Omniireamerccpp
(g) TFillcVPR 2022
(h) Inpaint Anythingarxiv 2023
Figure 6: Qualitative comparison for RGB panorama outpainting. Our PanoDiffusion generated more objects with appropriate layout, and with better visual quality. For BIPS and OmniDreamer, despite the seemingly reasonable results, the outpainted areas tend to fill the walls and lack diverse items. As for TFill, generates blurry results for large invisible areas. For Inpaint anything. it gen- erates multiple objects but they appear to be structurally and semantically implausible. Compared to them, PanoDiffusion generates more reasonable details in the masked region, such as pillows, paintings the wall, windows, and views outside. More comparisons are provided in Appendix.
Table 1: Quantitative results for RGB outpainting. All models were re-trained and evaluated using the same standardized dataset. Note that, we tested all models without the depth input.
Methods
FID
Canera Mask
.FID
D7
CT
F1D
.F15
NEvV
Mask
DT
F10
Layout Mask
2110
D7
C1
F1D
Randen Box Mask
2213
D²
RIPS
31.70
25.80
0.760
0.660
$7.49
44.68
0.205
0.277
32.25
24.66
0.645
0.579
25.35
22.60
0.666
0.798
Chnaieeamner
65:47
37.04
0.143
0.175
62.56
36.34
0.125
0.184
82.71
28:40
0.123
0.130
45.10
24.12
0.329
0.576
LaMe
115.92
137.69
0.034
0.082
125.77
136.32
0.002
0,006
129.77
35.23
0.018
0.043
45.25
24.11
0.429
0.701
TRII
$3.84
61.40
0.075
0.086
93.62
76.13
0.037
0.027
97.99
43.40
0.046
0.052
46.84
30.72
0.368
0.574
Inpaisting Anytting
97.38
54.73
0.076
0.135
105.75
59,70
0.054
0.035
92.15
32.00
0.116
0.085
46.30
26.71
0.372
0.632
RePain
$2.54
$4.39
0.996
0.105
95.38
K2.35
0,0039
0.079
09:14
31.63
0.294
0.263
55.47
38.78
0.433
0.311
PanolXaffucon
21.58
26.95
0.867
0,708
21.41
27.80
0.790
0.609
23,06
22.39
1.000
0.737
16.13
20.39
1.000
0,883
Baseline Models.
For RGB panorama outpainting, we mainly compared with the following state-
of-the-art art methods:
including image inpainting models LaMa (Suvorov et al. 2022 WACV'2022 and
TFill (Zheng al.
2022)cvPR 2022, panorama outpainting models BIPS (Oh al., 2022 20222 2022
and OmniDreamer
Akimoto et al. 2022)
)022)cVPR220) 2V22 diffusion-based image inpainting models Re-
paint (Lugmayr et al. 2022)cVPR CVPR 2022 and Inpaint Anything (Yu et al.] [023 arXiv'2023. To evaluate the quality of depth panorama, we compare our method with three image- image-guided depth synthesis methods including BIPS (Oh et al. 2022), NLSPN (Park al.] [0000, and CSPN (Cheng et al,) 2018). All models are retrained on the Structured3D dataset using their publicly available codes.
4.2
MAIN RESULTS
Following prior works, we report the quantitative results for RGB panorama outpainting with camera masks in Table IT All instantiations of our model significantly outperform all state-of-the-art models. Specifically, the FID score is substantially better (relative 67.0% improvement).
It is imperative to note that our model is trained unconditionally, with masks only employed during the inference phase. Therefore, it is expected to handle broader spectrum of mask types. To vali- date this assertion, we further evaluated our model with the baseline models across all four different mask types (displayed in Fig. 5). The results in Table [howw that PanoDiffusion consistently outper- forms the baseline models on all types of masks. Conversely, baseline models' performance displays significant variability in the type of mask used. Although the visible regions of the layout masks are always larger than the camera masks, the performances of baseline models on camera masks are significantly better. This is likely because the masks in the training process closer to the NFoV distribution. contrast, PifoDifuusion has a more robust performance, producing high-quality and diverse output images for all mask distributions.
The qualitative results are visualized in Fig. Here we show an example of outpainting on a layout mask (more comparisons in Appendix). Besides the fact that PanoDiffusion generates more visually realistic results than baseline models, comparing the RGB (trained without depth) and RGB- in Fig. ((C), some unrealistic structures generated the center
D versions of our

---

Published as a conference paper at ICLR 2024
On
(a) RGB Input
(b) Depth GT
(c) Ours PanoDiftusion
(d) CSPN
(e) BIPS
(f) NLSPN
Figure 7: Qualitative comparison for depth panorama synthesis.
Table Depth map ablations. All models are trained and evaluated on the Structured3D dataset.
Camera Mask s
NFoV Mask
Layout Mask
Random Box Mask
Noise Level
FiD.
D.
CT
FID
D2
CT
TID.
sFIl 4.
CT
FiD
<FID.
D
CT
depth
24.33
29.00
0.667
0.635
24.01
30.00
0.639
0.617
25.37
22.92
0.785
0.677
17.88
21.21
0.913
0.857
50%.
21.65
28.12
0.678
0.660
21.99
29.37
0.678
0.561
24.24
23.05
0.855
0.724
17.02
21.22
0.919
0.837
30%
21.78
27.96
0.714
0..674
21.78
29.39
0.643
006658
24.11
23.00
0.919
0.724
16.87
21.25
0.937
0.855
10%.
21.68
27.79
0.721
0.658
21.49
29.74
0.558
0.620
24.02
22.68
0.938
0.741
16.60
21.02
0.932
0.853
depth
21.55
0.867
0.708
21.41
27.80
0.790
0.669
23.06
22.39
1,000
0.737
16.13
20.39
1,000
(a) Usage depth maps (training). We use different sparsity levels depth for training and the results (more intense color means better performance) verify the effectiveness of depth for RGB outpainting. also proves that the model can accept sparse depth as input.
Methods
Input Depth
FID
sHD J
Methods
Impui Depth
RMSE
MAE
AbsREL
Delial.25
BIPS
PannlDifflssion
fully visible
29.74
21.90
26.78
30.59
0.829
0.931
0.693
0.721
BIPS
CSPN
NI SPN
fully masked
284
183
0.1692
0.2273
0.8544
partial visible
PanoDiffusion
276
193
0.1355
0,9060
PanoDiffussio
22.34
26.74
0.856
0.686
BIPS
247
136
0.1098
0.9032
BIPS
fully masked
68.79
42.62
0.306
0.412
pantall
PanoDDifuusion
219
123
0.1127
0.9278
(b) Usage depth maps (inference). BIPS heav- ily relies on the availability of input depth during inference, while our model is minimally affected.
(c) Depth panorama synthesis. Our model outperforms baseline models in most of the metrics.
wall, and when we look closely the curtains generated by the RGB model, the physical structure of the edges is not quite real. In contrast, the same region of RGB-D result () appears more structurally appropriate. Such improvement proves the advantages of jointly learning to synthesize depth data along with RGB images, even when depth is not used during test time, suggesting the depth information significant for assisting the RGB completion.
4.3 ABLATION EXPERIMENTS
We ran a number of ablations to analyze the effectiveness each core component in our PanoDif- fusion. Results shown in tables 2 3nd figs. and and discussed in detail next.
Depth Maps. practice applications, depth data may exhibit sparsity due the hardware limita- tions (Park et al. 2020). To ascertain the model's proficiency in accommodating sparse depth maps as input, we undertook training process using depth maps with different degrees of sparsity (i.e., randomized depth value will be set to 0). The result is reported in Table 2(a). The denser colors in the table represent better performance. As the sparsity of the depth input decreases, the performance of RGB outpainting constantly improves. Even if use 50% sparse depth for training, the result is overall better than the original LDM.
We then evaluated the importance of depth maps during inference. and compared it with the otate-of- the-art BIPS (Oh et al. 2022), which is also trained with RGB DD images. The quantitative results are reported in Table b). As can be seen, BIPS's performance appears deteriorate significantly when
8

---

Published as a conference paper at ICLR 2024
(a) PanoDiffusion w/ TA (b) PanoDiffusion w/o TA (c) OmmiDreamer
(d) BIPS
Figure 8: Examples of stitched ends of the outpainted images. For each image, the left half was unmasked (i.e. ground truth), while the right half was masked and synthesized. The results generated with rotation are more naturally connected both ends (a).
Table 3: Two-end alignment ablations. Using rotational outpainting, we achieve optimal consis- tency at both ends of the PanoDiffusion output.
Methods Mask Type
Camera
NioV
Layout
Random Box
Methods Musk Type
Camera
NioV
Layout
Random Box
DitsSDiTo (w/ rotation)
90.41
89.74
88.01
85.04
PanoDiifusiion (wlo rocation)
125.82
128.3
128 10
128.19
BIPS LaMa
117.59
96.82
132.15
148.78
OraniDreamer
115.6
109.00
1/6.37
136.68
119.51
119.39
133.54
136.35
TFill
155.16
157.60
136,91
122,96
the input depth visual area is reduced. Conversely, our PanoDiffusion is not sensitive to these depth maps, indicating that the generic model has successfully handled the modality. Interestingly, we noticed that having fully visible depth at test time did not improve the performance of PanoDiffusion, reasonable explanation is that during the training
and in fact, the result deteriorated slightly. process, the hignal-tonnoss ratios (SNR) of RGB and depth pixels are roughly the same within each iteration since no masks were used. However, during outpainting, the SNR balance will be disrupted when RGB input is masked and depth input is fully visible. Therefore, the results are degraded, but only slightly because PanoDiffusion has effectively learned the distribution of spatial visual patterns across all modalities, without being overly reliant on depth. This also explains why our model is more robust to depth inputs with different degrees of visibility.
Finally, we evaluated the depth synthesis ability of PanoDiffusion, seen in Table and Fig. The results show that our model achieves the best performance on most of the metrics and the qualitative results also show that PanoDiffusion is able to accurately estimate the depth map. This not only indicates that PanoDiffusion can be used for depth synthesis and estimation but also proves that it has learned the spatial patterns of panorama images.
Two-end Alignment. Currently, there no metric to evaluate the performance of aligning the two ends of an image. To make a reasonable comparison, we make one side of the input image fully visible, and the other side fully masked and then compare the two ends of output. Based on the Legh-Right Consistency Error (LRCE) (Shen et al., 2022) which is used to evaluate the consistency of two ends of the depth maps, we designed a new RGB-LRCE to calculate the difference between the two ends of the image: LRCE = N Σm first
Piatl and reported results in table
The qualitative results are shown fig. X To compare as many results, we only show the end regions that are stitched together to highlight the contrast, They show that the consistency of the two ends of the results is improved after the use of rotational outpainting, especially the texture of the walls and the alignment of the layout. Still, differences be found with rotated outpainting. We believe is mainly due to the fact that rotational denoising is based on the latent level, which may introduce extra errors during decoding.
CONCLUSION
In this paper, we show that our proposed method, the two-stage RGB-D PanoDiffusion, achieves state-of-the--ar performance for indoor RGB-D panorama outpainting. The introduction of depth information via our bi-modal LDM structure significantly improves the performance of the model. Such improvement illustrates the effectiveness of using depth during training RGB panorama generation. In addition, we show that the alignment mechanism we employ at each step of the denoising process of the diffusion model enhances the wraparound consistency of the results. With the use of these novel mechanisms, our two-stage structure is capable of generating high-quality RGB-D panoramas at 512x 1024 resolution.
aid to guide

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
REFERENCES
Naofumi Akimoto, Seito Kasai, Masaki Hayashi, and Yoshimitsu Aoki. 360-degree image comple- tion by two-stage conditional gans. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 4704-4708. IEEE, 2019.
Naofumi Akimoto, Yuhi Matsuo, and Yoshimitsu Aoki. Diverse plausible 360-degree image out- painting for efficient 3deg background creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11441-11450 1450,, 2022.
Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convo- lutional spatial propagation network. In Proceedings of the European conference on computer vision (ECCV), pp. 103-119, 2018.
Sungjoon Choi, Qian-Yt Yi Zhou, and Vladlen Koltun. Robust reconstruction of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5556- 5565, 2015.
Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros. Blobgan: Spatially disentangled scene representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 616-635. Springer, 2022.
Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Chris- tian Gagné, and Jean-Frangee Lalonde. Learning to predict indoor illumination from single image. ACM Transactions on Graphics (TOG), 36(6):1-14, 2017.
Marc-André Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, and Jean- François Lalonde. Deep parametric indoor lighting estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7175-7183, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Takayuki Hara, Yusuke Mukuta, and Tatsuya Harada. Spherical image generation from a single image by considering scene symmetry. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1513-1521, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 6851, 2020.
Daichi Horita, Jiaolong Yang, Dong Chen, Yuki Koyama, and Kiyoharu Aizawa. A structure-guided diffusion model for harge-hole diverse image completion. arXiv preprint arXiv:2211.10437 2022.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4): 2017.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In editor (ed.), Proceedings of the International Conference Learning Representations (ICLR), 2014.
Jeong- Kyun Lee, Jaewon Yea, Min-Gyu Park, and Kuk-Jin Yoon. Joint layout estimation and global vieti-view registration for indoor reconstruction. In Proceedings of the IEEE international con- ference computer vision, pp. 162-171, 2017.
10

---

Published as a conference paper at ICLR 2024
Wenbo Li, Xin Yu, Kun Zhou, Yibing Song, Zhe Lin, and Jiaya Jia. Sdm: Spatial diffusion model for large hole image inpainting. arXiv preprint arXiv:2212.02963, 2022.
Zhuqiang Lu, Kun Hu, Chaoyue Wang, Lei Bai, and Zhiyong Wang. Autoregressive omni-aware
outpainting for ppen-vocabulary 360-degree image generation. arXiv preprint (tX.Xx::330903467. 2023.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint:
Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 11461-11477 2022.
Muhammad Ferjad Naeem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning. pp. 7176-7185 PMLR, 2020.
Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W Battaglia. Generating images with sparse representations. arXiv preprint (03:::2003.084441 2021.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE international symposium on mixed and augmented reality, pp. 127-136. leee, 2011.
Changgyoon Oh, Wonjune Cho, Yujeong Chae, Daehee Park, Lin Wang, and Kuk-Jin Yoon. Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings, Part XVI, pp. 352-371. Springer, 2022.
Jinsun Park, Kyungdon Joo, Zhe Hu, Chi-Kuei Liu, and In So Kweon. Non-local spatial propagation network for depth completion. Computer RisionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 120-136. Springer, 2020.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Xiaofeng Ren, Liefeng Bo, and Dieter Fox. Rgb-(d) scene labeling: Features and algorithms. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2759-2766. IEEE, 2012.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panoformer: Panorama transformer for indoor 360-degree depth estimation. In European Conference on Com- puter Vision, pp. 195-211. Springer, 2022.
Gowri Somanath and Daniel Kurz. Hdr environment map estimation for real-time augmented reality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11298 1306, 2021.
Shuran Song and Thomas Funkhouser. Neural illumination: Lighting prediction for indoor environ- ments. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6918-6926, 2019.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robut large mask inpainting with fourier convolutions. In Proceedings of the IEFF/CVF winter conference on applications of computer vision, pp. 2159-2159, 2022.
11

---

Published as a conference paper at ICLR 2024
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: En- abling holistic multi-view image generation with correspondence-awee diffusion. arXiv preprint arXiv:2307.01097 2023.
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4692-4701, October 2021.
Su Wang. Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 18359-18369) 2023.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 222437,,,, 2023.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint Irxiv: 03365, 2015.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5505-5514, 2018.
Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng, and Zhibo Chen. Inpaint anything: Segment anything meets image impainting. arXiv preprint arXiv:2304.0677 3040006900 2023.
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric, Chao Chang, and Yan Xu.
Large scale image completion via co- no-oodulated generative adversarial networks. In International Conference on Learning Representations, 2020.
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. T2net: Synthetic-to- ro-aalistic translation for solving single-image depth estimation tasks. In Proceedings of the European conference computer vision (ECCV), pp. 767-783, 2018.
Chuanxia Zheng. Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1438-1447 2019.
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic free-form image completion. Interna- tional Journal of Computer Vision, 129:2786-2805, 2021.
Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai, and Dinh Phung. Bridging global context interactions for high-fidelity image completion. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11512-11522, 2022.
Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pp. 519-535. Springer, 2020.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
The supplementary materials are organized as follows:
1. A video is added to illuminate the work with more results.
2. The reproducible code is included.
3. An additional PDF for implementation, training, metrics details, well quantita- tive and qualitative results.
IMPLEMENT DETAILS
TRAINING OF VQ MODELS FOR RGB AND DEPTH PANORAMA
The reason why LDMs (Rombach et al. 2022 be trained on larger image scales is that per- ceptual image compression is used so that the diffusion process can be conducted in latent space, with the decoder D used for returning the latent vector = Rxxwxe to high-resolution image E "xxx' xc, During this process, the inherent spatial structure of the image does not change but downscaled. Such spatial constancy is critical for our work as the partially visible regions will not be changed during outpainting, shown in Fig.
(a) Masked Image
(b) Masked Latent Image
(c) Masked Image
(d) Masked Latent Image
Figure 9: Masking at different levels. (a)(c) are pixel-level masked images, while (b)(d) are corre-
sponding latent-level masked images. They are aligned well, except for the different mask values.
Two VQ models, VQrgh and VQdepth with downsampling factor f = are trained using RGB and depth data respectively. The output channel numbers VQrgt and VQdegtth are set and 1. For the training of V VQ we finetuned on the pre-trained VQ-f4 model provided by Rombach et al. (Rombach et al. 2022). Due to the lack of pre-trained model, VQ is trained from scratch. trained for 30 epochs and select the models that perform best
Both VQryb and VQdepth on the validation data for the training of bi-modal LDM. Throughout the process, the datasets used during training, validation, and testing exactly matched for VQrgb and V VQdepth-
TRAINING OF LATENT DIFFUSION MODEL
The training of PanoDiffusion initially loaded the LSUN-Bedrooms (Yu et al., 2015) pre-trained model provided by official LDM (Rombach et al. 2022). Despite the fact that the images are in normal view, where the objects and the layout are subjected to equirectangular projection, we believe that it can provide useful priori knowledge of item semantics, texture, etc. for the outpainting of the indoor panoramas. At the same time, pre-trained VQrgb and VQdepth are loaded and fixed during the training of our bi-modal LDM.
CAMERA-ROTATION DIRECTION
During the training stage, restrict random angle rotation solely the horizontal direction. As panorama images typically follow cquirectangular projection, the distortion increases non-uniformly towards the top and bottom poles. Introducing random angle rotation in the vertical direction would lead to substantial changes the projection results, and require more complex preprocessing. which will increase training costs. Conversely, distortion is uniform horizontally. image manipulation here only involves horizontal cropping and splicing, without the need for reprojection.
Here show two examples where the camera has 90-degree vertical rotation as Fig T0
13

---

Published as a conference paper at ICLR 2024
Horizontal Rotation
Original View
Vertical Rotation
Figure 10: Examples of vertical and horizontal camera rotation. No matter how many degrees the panorama is rotated horizontally, the distortion of each component remains unchanged. However, when the panorama is rotated vertically, even by small angle, the distortion of the entire scene changes significantly, which can be hindrance for PanoDiffusion learn.
REFINENET IMPLEMENTATION
For the second stage of structure, generate visually plausible 512x 1024 results, trained GAN for super-resolution. consists of a generator and discriminator For training G, we use weighted sum of the pixel-wise L1 loss and adversarial loss. The pixel-wise L1 loss is denoted as Ippinet measuring the difference between the GT and the output panorama.
Lpinet L (g G(gttr))
Lado [[((G(())+)) 1)
(8)
LG ALgivel + Ladv-
The training data is randomly downscaled from 512x 1024 GT images 128 256x512 and upscaled back to 512x x 1024 using the traditional interpolation method, which will crase details from GT images. Then they are used as the input of our super-resolution GAN, denoted as ghr,. Here the value of À is set to 20 during the training.
QUANTITIVE METRICS
In this section, we will describe how the quantitative metrics used in this paper are implemented.
Fréchet inception distance (FID) FID (Heusel al. 2017) used capture the similarity of generated images to real ones. We used the official Py Torch implementation of FID to evaluate the similarity between the final average pooling features of GT images and model outputs.
Spatial FID (sFID) Spatial FID (Nash et al. 2021 is a variant of FID, using spatial features rather than the standard pooled features. As standard pool_3 33 features compress spatial information to a large extent, making it less sensitive to spatial variability, mixed 6/conv features can provide sense of spatial distributional similarity between models. sFID is calculated using the first channels from the intermediate mixed 6/conv feature maps in order to obtain feature space of size 177 which is comparable to the final average pooling features of size 2048.
Density and Coverage Density and coverage metrics are proposed by Nacem et al. (Naeem al., 2020), who argue that even the latest version of the precision and recall metrics are still not reliable. Therefore, they proposed density and coverage, which can provide more interpretable and reliable signals. In this paper, used their official implementation with nearest neighbor & = 3 to calculate the density and coverage of the final average pooling features of the GT panoramas and the generated output.
14

---

Published as a conference paper at ICLR 2024
Depth Estimation Metrics Given ground truth depth Dgt {dgt} and predicted depth D ppred (dpred): ). we use the following metrics to evaluate the quality of our depth estimates.
RMSE
l1dtt dpreall
(10)
D
1
MAE
Σ cdtt dpreals
(11)
[D|
bdgt dpreal dge
AbsREL ID]
(12)
Uprod dgt
Deltal.25 % of dpred in Dreed s.t. max det Apred
1.25
(13)
ADDITIONAL RESULTS AND EXAMPLES
QUANTITATIVE COMPARISON FOR DEPTH-CONDITION LDM
As described in the main paper, introduce depth information aid RGB generation, an intuitive idea would be to use depth information as an explicit condition during training and inference. By compressing depth information into latent space 2depth. it can introduced into the denoising process of the RGB images via cross-attention, denoted as depth However, we have found that such an approach often leads to blurry results, as the image examples we have shown in the main paper (more examples are shown in Fig. 1T.
LDM (DC LDM).
Figure 11: Outpainting with basic
LDM. This leads to blurry results.
Since DC LDM is also trained with RGB-D images, here we report its performance on depth inputs with different degrees of visibility together with BIPS and our RGB-D PanoDiffusion, shown in Table 4
Table 4: Full quantitative results for RGB outpainting with different depth input at test time.
Methods
Inpet Depth
FI1 J
sFID>
Density
Coverage
BIPS
29.74
30.59
0.931
0.721
fully visible
77.75
RGD D PunoDifiusion
21.90
26.87
0.829
0.693
HIPS
31.70
28.89
0.769
0.660
DC LDM
partially visible
77.77
44.44
0.054
0,080
RGB-D PanoDiffusio
22.34
26.74
0.856
0.686
DC LDM
fully masked
78.15
44.29
0.048
0.073
RGB PanoDiffusion
21.55
26.95
0.867
0.708
The results show that the DC LDM does not perform well in terms of both visual and quantitative results. Even complete depth information provided as a condition, the improvement in results is very marginal. This indicates that the conditional LDM strueture cannot make good use of depth information to assist RGB panorama outpainting, which proves the effectiveness of our bi-modal LDM structure.
FUL.I QUANTITATIVE COMPARISON FOR REFINENET
As we only show the results of our RefineNet on camera and NFoV mask in the main paper, here we report the full quantitative results on all types of masks with both RGB and RGB-D PanoDiffusion
15

---

Published as a conference paper at ICLR 2024
(a) Classical Interpolation Output
(b) Super Resolution Output
Figure 12: Examples upscaled results using classical interpolation method and out super-
resolution GAN.
visualized comparison example is shown in Fig. 12 The results show that super- resolution GAN improves the quality of PanoDiffusion output comprehensively, except for slight degradation of the sFID of RGB PanoDiffusion on the NFoV mask.
(Table 5)
Table 5: Full quantitative results for RefineNet. (-): classical interpolation. (+): super-resolution.
Mask Type
Version
FID
sFID
Density
Coverage
Mask Type
Version
SR
FID 1
sFID 1
Density |
Coverage
RGB-D
24.29
26.75
29.89
28.05
0,581
0.867
0.805
0.570
0.663
0.708
Layout
RGB-D
28.18
23,06
26.16
21.30
22.39
24.21
0.920
0.751
0,689
0.737
0.650
RGB
24.33
29.00
0.667
0.635
RGB
25.37
22.92
0.785
0.677
23.96
28.19
0.775
0.645
RGB-D
20.05
22.77
0.996
0.836
NFoV
RGB
24.01
26.72
29.93
30.00
0.648
0,639
0,790
0.595
0.617
Random Box
RGB
17.88
21:77
23.72
21.21
0.913
0.833
0,857
0.800
ABLATION STUDY ON CAMERA-ROTATION ANGLES
We additionally explored the effect of different rotation angles, including 180%, 90" (chosen for our final result), and 45", on the outpainting results, seen Table (6 The results show that the wrap- around consistency of outpainting results improved across all settings. Compared to 180%, 90% leads to better consistency. However, diminishing the angle further to 45" did not lead to additional improvements. We believe this is reasonable, as the model is expected to generate coherent content when the two ends are in contact for enough denoising steps. Therefore, smaller rotation angles than 90" and longer connections do not necessarily lead to more consistent results.
Table Camera-rotation angles ablations.
Methods Mask Type
Camera
NioV
Layout
Random Box
End
PanoDiffusioní w/o rotation)
125.82
128.33
128. 10
128.19
132.69
PanoDifuusion(188
95.11
96.57
90.93
85.23
119.60
PanoDiffusion(00"
90.41
89.74
88.01
85.04
116.77
PanoDiffusioo(455))
90.67
90.25
87.65
86.50
112.47
ADDITIONAL VISUALIZATION EXAMPLES OF RGB PANORAMA OUTPAINTING
Due to page limitations, we only provide one group of comparative results for RGB outpainting in the main paper. Here we will provide more visualization examples, shown in Fig. [3 Same as in the main paper, we compare PanoDiffusion with LaMa (Suvorov et al. 2022). TFill (Zheng et al.) 2022), OmniDreamer (Akimolo et al. 2022). BIPS (Ohet al. 2022). Repaint (Lggmayr et al. 2022). and Inpaint Anything YYu et al., 2023) on different types of masks. It can be seen that our method outperforms the baseline models by generating various objects with appropriate layout, and with better visual quality. Besides, to prove that our PifoDiffusion can perform diverse and plausible completions given input, provide two different outpainting results for each example.
16

---

Published as a conference paper at ICLR 2024
ADDITIONAL VISUALIZATION EXAMPLES OF DEPTH PANORAMA SYNTHESIS
Due to page limitations, we only provide one group of comparative results for Depth synthesis in the main paper. Here we will provide more visualization examples, shown in Fig. 14 Same in the main paper, we compare PanoDiffusion with BIPS (Oh et al. 2022), NLSPN (Park et al.) 2020), and CSPN (Cheng al. 2018). It can be seen that our method outperforms the baseline models by accurately estimating the depth map.
GT RGB Image
Masked RGB
Image (Input)
Ours
Ours 2
LaMa
TFill
BTPS
OmniDreamee
Inpaint Anything
RePaint
17

---

Published as a conference paper at ICLR 2024
GT RGB
Image
Masked RGB Image (Input)
Ours
Ours
LaMa
TFill
BIPS
Oremmrr
Inpaint Anything
RePaint
Figure 13: Additional qualitative comparisons for RGB panorama outpainting. Our PanoDif- fusion generated more objects with appropriate layout, and with better visual quality. Please zoom to see the details.
QUALITATIVE RESULTS OF ZERO-SHOT TEST ON MATTERPORT3D DATASET.
To test the generalization capability of PanoDiffusion, we conducted additional tests using a set of panorama images from the Matterport3D dataset. Here provide six groups of examples from the outpainting results, which show that our model can have a decent outpainting effect on the real panorama dataset as well.
18

---

Published as a conference paper at ICLR 2024
RGB Input
Depth GT
Ours
BIPS
NLSPN
CSPN
Figure 14: Additional qualitative comparisons for Depth panorama synthesis. Our PanoDiffu- sion achieves most accurate estimation. Please zoom in to see the details.
Figure 15: Results of zero-shot test on Matterport3D dataset. Zoom in to see the details.
QUALITATIVE RESULTS DISCRETE MASK ABLATION.
To explicitly assess our model's 's performance with discrete masks, we flipped the camera mask swapping the originally visible and invisible parts to simulate this situation. This type of mask is equivalent to randomly sampling several NFoV masks and making them invisible. Here we provide examples from the outpainting results as Fig T6
19

---

Published as a conference paper at ICLR 2024
Figure 16: Outpainting Results on discrete masks.
Masked Image (Input)
Generated RGB
Generated Depth
Figure 17: Synthesized RGB-D Panorama Outpainting Results.
QUALITATIVE RESULTS OF SYNTHESIZED RGB-D PANORAMA RESULTS.
In Fig. we provide some synthesized RGB-D panorama examples where RGB is partially visible and depth is fully masked. The results show that our PanoDiffusion can outpainting plausible and consistent RGB-D panoramas simultaneously.
COMPLEXITY ANALYSIS
Table 7: Training and inference time comparison
Method
Type
Depth
Training (mins/epoch)
Inference (sec/image)
PanoDiffusion
bi-modal LDM GAN
82
BIPS
131
RePaint
Diffusion model
LDM
LDM
72
OmniDreamer
Transformer VQGAN
158
61
20

---

Published as a conference paper at ICLR 2024
For training, compared average training time (minutes) for epoch of PanoDiffusion against baseline models on the same devices, using the same batch size 4 and the same training dataset. For inference, we compared the time (seconds) required to infer single image.
The results show that while our model is not the fastest, it remains within a reasonable and acceptable range. It's also noteworthy that, compared to the original LDM framework, bi-modal structure achieves a significant improvement in the quality of outpainting. This improvement comes without a proportionate increase in resource consumption observed only modest increase of 13.8% in training time and 25% in inference time.
21


Reviews:
[{"review_id": "r5eWG5kDMB", "reviewer": "Reviewer_SqKp", "summary": "This paper proposes a new diffusion based method to tackle panorama image generation task. The proposed method utilizes the latent diffusion models to outpaint the area that is not originally taken by the camera. Relying on the powerful generation capability of diffusion models, the proposed method proposes to progressively apply camera rotations during the image generation process in order to enhance the generalizability. Extensive experiments show that the proposed method is able to outperforms the baseline methods significantly.", "questions": "The authors are suggested to address the concerns raised in the weaknesses section during the rebuttal period.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is generally well written with strong motivation and well-organized writing. It clearly demonstrates the problem of  current pano generation and implies the proposed method is tackling the problems stated. \n2. The proposed method significantly outperforms the baseline methods on the selected benchmarks.", "weaknesses": "1. During the training stage, does the random angle rotation only apply to horizontal direction?\n2. Depth map actually provides rich information indicating the scales of the objects in the images. In order to test the generalization capability, can you provide more results of running the proposed method on other datasets?\n3. This paper claims the camera rotations as one of the contributions, please provide an ablation study on how the camera rotation actually works effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "msuGy8LJJ4", "reviewer": "Reviewer_inST", "summary": "The authors propose a way to outpaint a near field-of-view image (i.e. a normal image from a normal camera) to a panorama, which they represent as a equirectangular projected image. They use latent diffusion models to do so. Their latent model is trained on RGB-D panoramic data, but works on just RGB inputs. \n\nOne problem with outpainting panoramic images is that the left and right side of the equirectangular project image need to map to each other. The authors introduce a novel technique to do this. It works by rotating the image by 90 degrees in each denoising step in the diffusion model.\n\nThe authors compare to a variety of other techniques and show that their method is better than competing methods.", "questions": "There are many ways to represent 360 degree panorama images. The authors should clarify that using the equirectangular projection is a choice they are making.\n\nDo the authors only rotate the camera in the horizontal direction? Or are vertical rotations allowed as well?\n\nDuring inference, did the authors try 180 degree shifts instead of 90 degree shifts? Or some other rotation?\n\nWhat’s the difference between Fig. 2 and Fig. 3? They seem like they are both figures about how training and inference are done, but they seem to be different. Specifically, Fig. 2b does not add noise and does not reference the circular shift, while Fig. 3b does. Is Fig. 2b incorrect?\n\nIn the supplemental video, do the shown 3D Scenes at 00:12 use the generated depth maps?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and the results seem pretty good. The author’s idea to rotate the image 90 degrees in each denoising step seems very novel. \n\nThere are lots of comparisons to many other methods and the proposed method is better based on a variety of techniques. \n\nI appreciate that the authors include the code as supplementary material!", "weaknesses": "It is somewhat hard to evaluate the generated panoramic images based on looking at images or fixed rotations. I would be interested in seeing the panoramic images of both the proposed method and competing method in a viewer like threejs. See https://threejs.org/examples/webgl_panorama_equirectangular.html", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rctI0Aksf1", "reviewer": "Reviewer_MEmh", "summary": "This paper proposed a latent diffusion model (LDM) for indoor RGB panaroma inpainting and depth map generation. During training stage, the input to the bi-modal LDM structure are RGB images and corresponding depth maps, which improve the performance of panaroma inpainting. At each stage of the denoising process in the diffusion model, the proposed alignment mechanism enhances the wraparound consistency of the results. The results indicate that the proposed PanoDiffusion excels not only by achieving a substantial performance advantage over state-of-the-art techniques in RGB-D panorama outpainting, yielding diverse and well-structured results for various mask types, but also by demonstrating the capability to generate high-quality depth panoramas.", "questions": "- How about the training and inference time compared to other frameworks? Since LDM requires a step-by-setp mechanism to generate the final results, the efficiency may be a problem.\n- There are only 4 types of maskes. Each type of mask covers different portion of the whole panaroma. I recommand the authors to add a chart that describes the performance changing along the percentage of the mask. According to my understanding, with the increase of the percentage of the mask, the performance will drop.\n- The mask are all continuous presented in this paper, I am curious when there are several separate masks in one panaroma, what will the performance become?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- During inference process, there is no need to input the depth map as a guidance, only training process requires depth maps as input. It is significantly different from the previous approaches.\n- This paper proposed a noval approach that involves gradually introducing camera rotations at each stage of the diffusion denoising process, resulting in a notable enhancement in achieving seamless panorama wraparound consistency.\n- With the clear description of each module and step, this approach ensures that the framework is not only effective but also easily comprehensible and straightforward to follow.", "weaknesses": "- There is not many noval changes to the LDM framework. The authors only add a pretrained depth map encoder as a guidance.\n- There are only one visual result (Figure 6) in the paper, which is not convincing. The authors should add more visual results for comparison in the supplementary materials.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6sgtseAPeN", "reviewer": "Reviewer_dEjQ", "summary": "The paper presents a diffusion model for panoramic image generation. A two-stage RGB-D PanoDiffusion model is proposed for indoor RGB-D panorama outpainting. The model taks depth information as input and processed through a bi-modal LDM structure. As the results shown, the use of depth information enhances the generation of RGB panoramas, and the alignment mechanism ensures wraparound consistency in the results. The method can be used to generate RGB-D panoramas at 512×1024 resolution.", "questions": "Apart from the visualization results, how about the evaluation results of using concatenation or cross-attention in the so-called depth-conditional diffusion model? \n\nHow is the cross-attention operation used in the depth-conditional method? Whether the authors try other advanced multimodal fusion methods to better combine the features from RGB and depth? \n\nIt would be better to ablate the camera-rotation data augmentation method. \n\nIn the depth panoramic synthesis, how about the comparison to some RGB-based depth estimation methods? For example, in the case of using fully masked input. \n\nHow about the runtime or complexity analysis of the proposed method, since the parallel LDM are used for RGB and depth separately.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The RGB-D fusion and combination mechanism is brought to the field of panoramic image generation via using the latent diffusion models. A RGB-D panoramic outpainting model is proposed to perform indoor 360-degree image generation. \n\nA bi-modal latent diffusion structure is proposed to combine RGB and depth information during training the diffusion model for panoramic image generation. \n\nA camera-rotation method is proposed to perform a stronger data augmentation. The two-end method can be used to crop a 90-degree equivalent area and stitch to the opposite sides to perform additional data augmentation to improve the wraparound consistency.", "weaknesses": "As compared to the previous methods, the depth information is additionally added to train the diffusion model. Even though the depth information is not needed for inference. \n\nThe proposed RGB-D framework is constructed by using two parallel LDM to reconstruct the depth and RGB images separately. This structure might result in a larger and more complex model architecture than using a shared or depth-conditional LDM.  \n\nAn additional module is needed to refine and upscale the low-resolution image output to a high-resolution image. However, a pre-trained super-resolution GAN model is needed to perform such a refinement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)