DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion
Authors: Weicai Ye
Chenhao Ji
Zheng Chen
Junyao Gao
Xiaoshui Huang
Song-Hai Zhang
Wanli Ouyang
Tong He
Cairong Zhao
Guofeng Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Spherical Epipolar-Aware Diffusion
Text to Multi-View Panoramas Generation
Primary area: diffusion_based_models
TL;DR: We propose a novel text-driven panorama generation framework that leverages multi-view consistency between generated images to achieve scalable, consistent, and diverse panoramic scene generation.

Abstract: Diffusion-based methods have achieved remarkable achievements in 2D image or 3D object generation, however, the generation of 3D scenes and even $360^{\circ}$ images remains constrained, due to the limited number of scene datasets, the complexity of 3D scenes themselves, and the difficulty of generating consistent multi-view images. To address these issues, we first establish a large-scale panoramic video-text dataset containing millions of consecutive panoramic keyframes with corresponding panoramic depths, camera poses, and text descriptions. Then, we propose a novel text-driven panoramic generation framework, termed DiffPano, to achieve scalable, consistent, and diverse panoramic scene generation. Specifically, benefiting from the powerful generative capabilities of stable diffusion, we fine-tune a single-view text-to-panorama diffusion model with LoRA on the established panoramic video-text dataset. We further design a spherical epipolar-aware multi-view diffusion model to ensure the multi-view consistency of the generated panoramic images. Extensive experiments demonstrate that DiffPano can generate scalable, consistent, and diverse panoramic images with given unseen text descriptions and camera poses.

DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion
Weicai Y.
Chenhao Ji
Zheng Chen
Junyao Gao? Xiaoshui Huang
Song-Hai Hai Zhang*
Wanli Ouyang
Tong H
Cairong Zhao
Guofeng Zhang
State Key Lab of CAD&CG, Zhejiang University Tongji University
Shhnngaii AT Laboratory 'Tsinghua Univerisity
malkeyeneiicai@mail.ccm
jichenhao@tongja iew.gn.c.i..eeecmmoommmom edu.cn
zhangguofengdzju.co edu.
kilchan black
well-malstained
leads celling
farnishd dining
leading upen
DiffPano allows scalable and consistent panorama generation (i.e. room switching) with given unseen text descriptions and camera poses. Each column represents the generated multi-view panoramas, switching from one room another.
Figure 1:
Abstract
Diffusion-basee methods have achieved remarkable achievements in 2D image 3D object generation, however, the generation of 3D scenes and even 360 images remains constrained, due to the limited number of scene datasets, the complexity of 3D scenes themselves, and the difficulty of generating consistent multi-viee viel im- ages. To address these issues, we first establish large-scale panoramic video-text dataset containing millions of consecutive panoramic keyframes with corresponding panoramic depths, camera poses, and text descriptions. Then, we propose novel text-driven panoramic generation framework, termed DifiPano, to achieve scalable, consistent, and diverse panoramic scene generation. Specifically, benefiting from the powerful generative capabilities of stable diffusion, we fine-tune single-view text-to-pan to-panorama diffusion model with LoRA the established panoramic video- text dataset. further design a spherical epipolar- multi-view diffusion model to ensure the multi-view consistency of the generated panoramic images.
Equal Contribution. 8: Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Extensive experiments demonstrate that DiffPano can generate scalable, consistent, and diverse panoramic images with given unseen text deseriptions and camera poses. Code and dataset are available at
gith io Difffano
Introduction
Generating scenarios from the text descriptions that meet one's expectations is an imaginative and
marvelous journey, which has many potential applications, such as VR roaming [62] for metaverse [67]
68 69 72.. physical world simulation 2 6 14), embodied agents in scene navigation and manipula-
tion 75 etc. With the advent of the AIGC era, several works [8] 14 20 [9] 42
43 48
[1] 52 on
object generation even scene generation has emerged. However, they generally only generate a series of perspective images, making it impossible to comprehensively simulate the entire environment for
scene understanding [2 19 66 and reconstruction [
5
65
64
30
26
12
49
try to generate panoramas to solve these problems by taking advantage of the inherent characteristics of panoramic images, which can capture the surrounding environment with single shot.
Given this, some methods 70 9
These methods can be roughly divided into four categories: 1) Directly single-view equirectangular projection (ERP) panorama generation (70]. However, its camera is immovable, making it incapable
of scene exploration: 2) Multiple perspective views generation methods [20
43
48
40)
without
considering multi-view panorama generation. 3) The inpainting solutions are based on the infinite expansion of a single perspective view, which lacks 3D awareness. Single scene optimization methods 53 with inpainting have no generalization ability and cost too much time for optimization.
4) Directly extending the multiple perspective views generation methods 43 to the ERP panorama, which is difficult to converge and results in poor multi-view consistency (see Fig. 5Dp
This paper aims to generate scalable and multi-view consistent panoramic images from text descrip- tions and camera poses (see Fig. [ with many potential applications such as immersive VR roaming with unlimited scapes and preview for interior home design. However, achieving this goal is not trivial. To the best of our knowledge. there is currently a lack of rich and diverse panoramic datasets to meet the task of toxt-to-multi-iewy ERP panorama generation. To this end, we propose novel panorama
panoramic video-text dataset and a generation framework suitable for the generation task, advancing the development of this field. Specifically, we first establish a large-scale panoramic video-text dataset using Habitat Simulator [41] (see Sec. (3), which contains millions of panoramic keyframes and corresponding panoramic depths, camera poses, and text descriptions. Next, built upon the proposed dataset, we propose a generation framework for consistent multi-view ERP panorama generation (see Sec. 4 termed DiffPano. The DiffPano framework consists of a single-view rext-to-panorama diffusion model (see Sec. 1. and spherical epipolar-aware multi- -a--ooampa diffusion model obtained by fine-tuning the stable diffusion model [38 of perspective images using LoRA [18]. Considering that the single-view pano-based diffusion model cannot guarantee the consistency of generated multi- view panoramas with different camera poses, we derive spherical epipolar constraint applicable to panoramic images, inspired by the perspective epipolar constraint. We then incorporated it as a spherical epipolar-aware attention module (see Sec. 4.2) into the multi-view panoramic diffusion model to ensure the multi-view consistency of the generated ERP panoramic images.
view diffusion model (see Sec. 4.2) The single-view
Since there are no related methods for comparison, we try to extend the MVDream method 43 to generate multi-view ERP panoramas. Trained and tested on the proposed panoramic video-text dataset, extensive experiments demonstrated that compared to the modified MVDream, our proposed mueti-view panorama generation based on spherical cpipolar-awarg attention can generate more scalable and consistent panoramic images. Our method also demonstrates the generalization ability of the original diffusion model to generate satisfactory multi-view ERP panoramas with given unseen text descriptions and camera poses.
Our contributions can be summarized as follows: 1) To the best of our knowledge, we are the first to propose a scalable and consistent multi-view panorama generation task from text descriptions and camera poses. 2) We established a large-scale diverse and rich panoramic video-text dataset, which fosters the research of cext-to-panorann panoramic generation framework with spherical epipolar attention module, allowing scalable and consistent panorama generation with unseen text descriptions and camera poses.
video generation. 3) We propose novel text-driven

---

2 Related work
2.1 Single- View Panorama Generation
Recently, latent diffusion model (LDM) methods have attracted widespread attention, and many
62) have emerged, achieving simultaneously generates eight diffusion model and stitches them together to produce a panorama. However, it cannot support the generation of top and bottom views, and the generated panorama resembles wide-angle images with an extensive field of view rather than true 360 images. Some methods [55] [3] solve this problem using equirectangular projection (ERP) and try to facilitate the interaction between the left and right sides during the panorama generation process to enhance the left-right continuity property inherent in ERP images. To address the domain gap between panorama and perspective images, PanFusion 70] proposed a novel dual-branch diffusion model that mitigates the distortion of perspective images projected on panoramas while providing global layout guidance. However, its more complex model architecture incurs longer inference times for panorama generation. In addition, PanFusion cannot be expanded as an effective pre-trained model to the multi-view panorama generation task due to its excessive network parameters. To strike a balance between computational complexity and ensuring left-right continuity of panoramas, our proposed single-view panorama- based stable diffusion model only requires fine-tuning with LoRA [18] to learn panoramic styles and achieve good edge continuity while maintaining higher generation speed and simpler architecture.
single-view panorama generation works 770 48 74 24
60
56
33
[5
remarkably impressive results. Among them, MVDiffusion
48
fixed-veeppoint perspective images through a multi-view Correspondence-Awe
The existing single-view panorama generation methods cannot achieve scalable panorama generation. The core of our paper lies the generation of multi-view consistent panoramic images, which we will introduce in Section 2.2 More importantly, the single-view panoramic image generated by previous methods mainly supports 3DoF roaming, while our method can generate multi-view panoramic images for 6DoF roaming, which can serve as the inputs for 360 Gaussian Splatting [23] or 360 NeRF [T0 I. Our method also has a great potential value in 360 relightable novel view synthesis with the combination of 360 multi-view inverse rendering method 25
2.2 Multi-View Image Generation
To the best of our knowledge, there is no work focusing multi-view panorama generation. We review the existing works about multi-view generation for perspective images in this part.
Zero 123 [29] laid the foundation for 3D object generation based on multi-view generation, while the pose-guided diffusion model [1 40 explored consistent view synthesis of scenes. However, iteratively applying the diffusion model to generate individual views in the multi-view generation task may lead to poor multi-view consistency of generated images due to accumulated errors. To generate high-quality multi-view images simultaneously, some methods [3] 43 57 32 modify the
UNets in the diffusion model into multi-branch form and achieve the effect of generating consistent multi-view images through the interaction between different branch.
Currently, most multi view generation tasks focus on generating multi- view perspective images of single objects [43 61 63 28 46, 45) or scenes 15 while minimal research has been conducted on multi-view panorama generation. Narrow FoV (field-of-view) drawbacks of perspective images lead to the fact that the existing generation methods only generate very local region of the scene at a time. Our work focuses on the task of exploring the generation of 360 images from multiple different viewpoints. Due to the camera projection difference between panoramic and perspective images, achieving consistency in multi-view panoramas is challenging. It is impossible to directly apply the existing epipolar attention module [51] 20 to multi-view panoramas. We strive derive the spherical epipolar line formula for panoramic images and propose a spherical epipolar attention module to ensure the multi-view consistency of the generated panoramas.
2.3 Panoramic Dataset
Great progress text to single-view panorama generation has been witnessed. However, text-to-mulde mollm view panorama generation is still a blank slate. One of the main limitations of this task is the lack of suitable datasets. The common panoramic datasets used in single-view panorama generation consist of indoor HDR dataset [16), outdoor HDR dataset (71), HDR360-UHD dataset [9] Structured3D (73.
3

---

Habitat Simulator
Trajectory
Depth CubeMap
View : The ceiling bathroom.
Prompt: six prompts describe
appcarance a Your task summarize
View empty
these six given text prompts into single caption
with painting on the wall.
that deseribe what the looks like.
View A bedroom with beige walls and white bed.
BLIP2
LLM
View bodroom tan walls and carpeted floor.
A well-farnished bodroom with beige walls, white
BLIP2
bed, and white wardrobe. The floor carpeted.
View S: An empty with white doors.
The room also features a painting on the wall and a
cciling and carpet
carpet bedroom.
and comfortable.
Stitched Paanorama and Depth
Panorama Caption
CubeMap Caption
Figure 2: Panoramic Video Construction and Caption Pipeline.
Standford 2D-3D-S I. and Matterport3D dataset 4., etc. Most of these datasets relatively small in scale and only have single-view panoramas, which cannot support multi-view panorama generation, except Matterport3D 4 In addition, the sky box images in Matterport3D [ contain only sparse views. Although HM3D 37 provides the textured mesh of 1000 scenes, it lacks the corresponding text description for each view. To generate multi-view panoramas, maps at each viewpoint in the 3D meshes HM3D, using the Habitat Simulator [41], and stitch them into panoramas. We generate complete text descriptions corresponding the panoramas by using Blip2 [22] to create text descriptions for each face of the cube map separately, and then summarizing them using Llama2 [50]. this way, obtain a panoramic video-text dataset that includes camera poses, corresponding panoramas, and text descriptions of the panoramas, which facilitates subsequent multi-view panorama generation tasks.
render cube
3 Panoramic Video-Text Dataset
Due to lack of high-quality panorama-text datasets, most text-to-panorant generation tasks require researchers to construct their own datasets. The dataset constructed in PanFusion (70) suffers from
blurriness at the top and bottom of the panoramic images, and the corresponding text descriptions not precise enough. To address these issues, utilize the Habitat Simulator [41] to randomly
select positions within the scenes of the Habitat Matterport 3D (HM3D) [37] dataset and render the six-far six-face cube maps. These cube maps then interpolated and stitched together form panoramas so we can obtain panoramas with clear tops and bottoms. To generate more precise text descriptions for the panoramas, obtained cube map, and then employ Llama2 [50] to summarize and receive accurate and complete render images based camera trajectories within the HM3D scenes, enabling the creation of a dataset that simultaneously includes camera poses, panoramas, and corresponding text descriptions. This dataset will be utilized in the multi-view panorama generation (see Sec. 4.2
first use BLIP2 22
generate corresponding text descriptions for each
text descriptions. Furthermore, the Habitat Simulator allows
Proposed Method: DiffPano
DiffPano is capable of generating multi-view consistent panoramas conditioned on camera viewpoints and textual descriptions, as illustrated in Fig. [ In this section, we first introduce our single-view panorama stable diffusion in Sec. 4.1 We then elaborate how to extend single-view panorama generation multi-view consistent panorama generation by leveraging the spherical epipolar attention module in Sec. 4.2

---

Text Prompis P.
Camera Poses R.T,,N
Text Procupts: A
butdoved arge duse and bus buclttch deck bodk lelvess Soon anca
Nosse
Text
Prompt
Noise n
Xi:N
Stahle Diffission
Denoising UNet
LoRAA
Text to Panorama
Epipolar Line
Computation
Generated Panorama
Target
Source View
Spherical Ep Apwpolar-Aware Module
Text to MV Paneramas
Single-View
Spherical Epipolar-Aware Multi- View DilTusion
Applications
Figure 3: DifIPano Framework. The DiftPano framework consists of a single-view text-to- texI-to-panorama diffusion model and spherical epipolar-aware multi-view diffusion model. It can support to single-view panorama or multi-view panorama generation.
4.1 Single- Vieg Panorama-Based Stable Diffusion
straightforward way to generate a single-view panorama from text is to train a text-to text-to-panorama diffusion model from scratch with large number of text-panorama pairs, which is both time- consuming and computationally expensive. However, stable diffusion [38] leverages a vast amount of perspective images and their corresponding textual descriptions training data, endowing it with excellent prior knowledge of perspective images and strong text understanding capabilities. An economical and effective way for panorama generation is to fine-tune the trained perspective diffusion model with a few text-panorama pairs. To this end, panorama generation from text can be regarded as style transfer of images generated by stable diffusion, converting them from perspective style to panoramic style, and requiring them to satisfy the left-right continuity property of panoramas.
LoRA-based fine-tuning Diffusion models for text-to text-to-image generations possess excellent prior knowledge of 2D images and strong text comprehension capabilities. We aim to preserve these abilities of the model while fine-tuning the Low-Rank Adaptation (LoRA) A) [8] fine-tuning method, which has been previously used in large language models. Compared full fine-tuning. LoRA is faster and requires fewer computational resources, In our approach, we freeze all the parameters of the original Stable Diffusion model and add trainable layers to the UNet component using the LoRA fine-tuning method. We then train the model using our custom-creater panorama-tex dataset. To improve the left-right continuity of the generated images, we perform data augmentation on the panorama training dataset by randomly concatenating a portion of the right side of the panorama to the left side. Experiments demonstrate that the panoramas generated using this method exhibit satisfactory left-right continuity.
generate images in the style of panoramas. We employ
4.2 Spherical Epipolar-Aware Multi- View Diffusion
Built upon our proposed single-view panorama stable diffusion in Sec. 4.1 diffusion model multi-view diffusion model with a spherical apipolar-aware attention module to generate multi-view scalable and consistent panoramas.
extend the viel
Spherical Epipolar-Aware Attention Module Epipolar attention was proposed in 20 51 to ensure consistency between generated multi-view perspective images. However, due to the differences imaging methods between perspective and panoramic views, existing epipolar attention cannot be directly used for panoramic views. To overcome this challenge, we derived the epipolar line for panoramic images the equirectangular projection (ERP), and the specific proof process provided in Appendix A Equation 14 shows the mathematical form of the spherical epipolar line in ERP images. The spherical epipolar line visualized
spherical epipolar-aware attention module of Fig. 3 We extend the principle of epipolar attention panoramic images to implement the spherical epipolar-aware attention module. Given pixel p the target view, we calculate corresponding spherical coordinates Paphere based on the spherical projection process:
5

---

0 = (0.5
piz
2m
W
Upiz
Ø = (0.5
H
where ppiz and ypiz are the pixel coordinates of p. and 6 are its corresponding spherical coordinates and W and H are the resolutions of panorama. We then transform the spherical coordinate system to the Cartesian coordinate system to obtain the camera coordinates Pcamera corresponding to p
Team cos( (i)
Yeam sin(Ø)
Zeam cos( cos
The camera coordinates of Poom are converted to world coordinates Pwcrtd through the camera's pose matrix. This allows us to compute the ray from the camera center to Pwoord in the world coordinate system and construct the spherical epipolar attention module.
Given N feature maps F' Fi < S N corresponding to N panoramic images and their respective camera pose matrices, we implement cross-attention between different views through the spherical epipolar-aware module. During the generation process, each feature map F can considered as the target view, and the K nearest views are selected from the remaining features as reference views.
For each feature point in the target view feature map, uniformly sample S points the ray between the feature point and the camera. All sampled points are reprojected onto the feature maps of the K reference views, and the corresponding feature values obtained through interpolation. We denote the features the target view as q, and the features of all sampled points in the reference views as k and The cross-altention is then constructed using these features.
Let p3 be a feature point in the target view feature map Fr, and Pi.l1 S S S be the uniformly sampled points the ray between p; and the camera center. We reproject these points onto the K reference view feature maps Fr.|1 [1 < k < to obtain the corresponding feature values fa,j, S j S, 5.15k k < K. The query qi, key k; and value v; for the cross- attento mechanism are defined as follows:
4 = F((p), == S s s
8= =
(3)
The cross-attention output o; for the feature point p; is computed
k
= Attention(q, ks, v.) = softmam
14
(4)
where d is the dimension of the query and key vectors.
Positional Encoding To enhance the model's understanding of 3D spatial relationships between different views, we follow EpiDiff (20 to employ the positional encoding method from Light Field Networks (LFN) 44. In the world coordinate system, let p; be pixel the target view, and o and d, be the origin and direction of the ray between 24 and the camera center, respectively. The Plücker coordinates r; of the ray are computed as:
(o. x di, d.).
(5)
For each sampled point pi.j on the ray, its corresponding spherical depth žj,j is transformed using harmonic transformation to get 7 (z1,). Similarly, the Plücker coordinates vi transformed as yr (T:).Thh positionally encoded features (rx) and 7=(2),) are then concatenated to obtain the combined positional encoding (r,, 2.
((1,215)) = (rx(),,, (z1,)))
(6)
The combined positional encoding ((72,21,)) is then concatenated with the feature maps F; and Fr, 1 < k < K to obtain the enhanced feature representations Ft and Frx 1 < k < K:
Ft(p.) (F.)),), 7/,, 2,)),, =

---

where [ denotes concatenation. These enhanced feature representations are then used in the cross-attention mechanism improve the model's understanding of 3D spatial relationships between different views.
Two-Stage Training The main difference between panoramic images and perspective images is that panoramic images contain 360 content of the surroundings, while perspective images only contain content from a given viewpoint. Therefore, when the camera only rotates or translates by small amount, the corresponding content in the panoramic image hardly changes. To make the generated multi-view panoramic images better match each corresponding text, we divide the training into two stages. In the first stage, we use the selected dataset with almost no change in image content (small camera movement) for training, which enhances the effect of the spherical epipolar-aware attention module. In the second stage, we increase the camera movement distance between each viewpoint and train with images that generate new content, improving the model's ability to understand text based on changes in perceived spatial location while ensuring multi-view consistency and enhancing scalability.
Experiment
Dataset We leverage the Habitat Simulator [4] to render panoramic video dataset based on the Habitat Matterport 3D (HM3D) dataset
[37]. The pipeline of dataset rendering and captioning is shown in Sec. After post post-processing such as dataset filtering, we constructed 8,508 panorama-texx pairs training sets for single-view panorama generation. For multi-view panorama generation, constructed 19,739 multi-view panorama-text pairs with nearly identical image content and 18,704 multi-view panorama-text pairs with different image contents training sets. For specific details regarding the dataset, please refer to Appendix B
Implementation Details In the multi-view panorama generation, we simultaneously generate N panoramas from different viewpoints. Within the spherical epipolar-aware attention module, consider the nearest views to the target view reference views, i.e., K 3, and sample = 10 points along each ray. We conducted separate training for 100 epochs on datasets with almost identical image content and datasets with varying image content. Please refer Appendix C for further implementation details.
Evaluation Metrics
To evaluate the performance of our proposed single-view panorama- based stable diffusion model, we employ three commonly used metrics: Fréchet Inception Distance (FID) [17). Inception Score (IS) (39] and CLIP Score (CS) 36). FID measures the similarity between the distribution of generated panoramas and the distribution of real images. IS assesses the quality and diversity of the generated panoramas. CS is utilized to evaluate the consistency between the input text and the generated panoramas. To further evaluate the consistency of multi-view panorama generation, we leverage the Peak hignal-toNoiis Ratio (PSNR) and Structural Similarity Index Measure (SSIM) [59] metrics. PSNR and SSIM quantify the pixel-level differences and structural similarity between the generated views respectively.
5.1 Single- View Panorama Generation
We evaluate the performance of our proposed method by comparing it with the following baseline approaches for text-to-panorama generation:
Baselines
Ligt2Light [iis two-stage approach that first generates a low-resolution panorama based on the input text, and then expands ultra- Altra-high resolution.
PanFusion [7] dual-branch text-to- to-poor-pa model that aims to mitigate the distortion caused by projecting perspective images onto a panoramic canvas while providing global layout guidance.
Since the panoramic images generated by MVDiffusion 48 do not include top and bottom viewpoints, they are not complete panoramas. Therefore, we do not compare method with MVDiffusion.
7

---

Text2Light
PanFusion
Pano-SD (Oars)
"A living room with hardwood Blors, fir fireplace, and large windows."
"A easy stody with built-in booksbelves leather
"A beddroom with celling fan, gray walls, hardwaed floor, bod, and TV on
eaey buili-in daylied with throw pillows and blanket,"
spacious corridor with har dwood floors."
Figure 4: Text to Panorama Comparison between TextLight, PanFusion, and Ours.
Table 1: Quantitative Panorama Comparisons with Baseline Methods
Method
FID4
ISt
CSt
Inference time(s)
Text2Light
76.50
3.60
27.48
50.4
PanFusion
47.62
4.49
28.73
27.6
Pano-SD((Ours)
48.52
3.30
28.98
5.1
Quantitative Results Table presents the quantitative results. The FID, IS, and CS values of Text2Light are from the PanFusion [70]. We trained our proposed Pano-SD using the same dataset as PanFusion [70] and re-evaluated the performance of PanFusion. From the results, it can be observed that our method slightly outperforms others in terms of CS value and achieves a significantly lower FID value compared to Text2Light, which is close to PanFusion [70]. Moreover, due to the simplicity and efficiency of our model architecture, our method has a substantial advantage in inference time, which can significantly improve the efficiency of subsequent multi view generation tasks.
Qualitative Results The qualitative comparison results are shown in Figure 4 We compare our method with the two models trained their respective datasets. The panoramas generated by Text2Light exhibit poor left-right consistency. On the other hand, the panoramas generated by PanFusion suffer from blurriness the bottom and top regions, which affects the overall integrity of the panoramas. In contrast, our model is capable generating panoramas with clear bottom and top regions and better left-right continuity. However, due to the quality of the dataset, the image quality may be slightly inferior.
5.2 Multi- Muew Panorama Generation
To the best of our knowledge, there is no method for multi-view panorama generation, the existing SOTA method MVDream for perspective images cannot be directly applied to multi-view panorama generation. To verify the validity of proposed spherical epipolar-aware attention module, adapted MVDream to the panorama generation task as a comparative baseline.
Specifically, we remove the spherical epipolar-aware attention module from our method and load the pre-trained LoRA layers from Pano-SD. We then convert the self-attention in the UNet to the form used in MVDream 43). Additionally, transform the camera pose matrix into camera

---

Table 2: User Study of Text Multi-view Panoramas
Method
Image qualityt
Image-text consistencyf
Multi-view consistencyl
MVDream
3.6
3.7
3.8
PanFusion
3.8
4.0
3.0
DiffPano(Ours)
4.0
4.2
4.3
Table 3: Ablation Study on the Number of Sampling Points and Reference Frames
FID
Ist
CS1
PSNRT
SSIM
Inference time(s)
1, $ 10
73.30
3,40
22.14
29,99
0.69
30,06
= 10
66.02
3.34
22.92
32.04
0.79
33.12
==3, 10
69.89
3.58
22.76
32.29
0.81
35.79
K= 6
68.39
3.57
22.74
33.32
0.86
26.61
= =
67.30
3.54
22.66
33.00
0.84
32.23
K 4, S 10
65.98
3.37
22.59
33.39
0.87
37.91
= $ = 12
62.79
3.26
22.65
32.89
0.83
43.72
embeddings through a 2-layer MLP and add it as a residual to the time embeddings. The qualitative comparison of the experiment is shown Fig Under the same training iteration, DiffPano significantly outperforms MVDream 43 in terms of multi-view consistency. Our method can maintain consistency in the details of multi-view images, while MVDream can only achieve certain level of similarity in the overall images. Even compared to MVDream with twice the number of training iterations, our method still performs better in of consistency.
User Study collected 20 text prompts and recruited nearly 50 volunteers evaluate text-1o- multi-view panoramic image generation. Evaluation metrics of multi-view ERP panorama generation include the quality of multi-view panoramic images, multi-view consistency, and the consistency between the text and multi-view panoramas. Experimental results on Tab, 2 show that method can generate multi-view panoramic images with better quality, higher text and image similarity, and more consistent multi-view images, compared with MVDream 43 and PanFusion [70]. See more qualitative results in Fig. to Fig. 10
5.3 Ablation Study
MVDream
MVDream x2
Diffaano (Ours)
peaceful unnoomm with bod, plants Baur-lo-ciling
Figure Comparisons with MVDream. DifiPano can generate more consistent multi-view panoramas. "MVDreamx2" denotes MVDream is trained with twice iteration number relative to method.

---

Table 4: Ablation Study of One-Stage vs Two-Stage Training
FID4
ISt
LPIPS,,
PSNR1
SSIM
One-stage 82.92
4.52
0.0454
31.64
0.74
Two-stage 74.08
3.13
0.0610
31.54
0.73
Number of Reference Views and Sample Points To explore the influence of varying numbers of reference frames and sampling points on model performance, we generate 5-frame multi-view panoramas simultaneously, and compute the FID, IS, and CS metrics for the first frame of each group to assess the quality of the generated panorama under different quantities of reference frames and sampling points. Furthermore, we set the same camera pose for the first and last frames of each generated panorama group, and calculate the PSNR and SSIM values between these two frames to evaluate the model's s multi-view consistency. As shown in Table. Bl frames and sampling points can improve the quality of generated panoramas to a certain extent, but the changes in image diversity and consistency with text remain marginal. With increasing number of reference frames and sampling points, the model's consistency exhibits improvement, however, when the number of sampling points becomes excessive (S=12), the multi-view consistency of the model diminishes.
increasing the number of reference
One-Stage Two-Stage We conduct ablation experiments one-stage and two-stage training. Table. | shows that the two-stage method will obtain the higher FID values. The IS of the two-stage training method lower, and the diversity is reduced to certain extent, which is slightly worse than the one-stage training. However, it should be noted that the images after one-stage training will have ghosting, but the two-stage will not.
6 Conclusion
We have proposed the panoramic video-text dataset and panorama generation framework with spheri- cal epipolar-aware attention for sox-----inglle-ieew or multi-view panorama generation. Extensive experiments demonstrate that our method can achieve scalable, consistent, and diverse multi-view panoramas.
Limitation and Future Work Although our method demonstrates the ability to generate consistent multi-view panoramas under the same setting the training phase, it is important to note that the number of frames increases during inference, the model tends to hallucinate content.
Exploring the use of video diffusion models to improve the consistency of generated multi-view panoramas is a promising direction. Longer panoramic videos are expected to be realized based on the generated panoramas conditions.
7
Acknowledgements
This work was partially supported by the National Key Research and Development Program of China (No. 2023Y and National Natural Science Foundation of China (Nos. 61932003, 62076184 and 62473286).
References
[1] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01 105, 2017.
[2] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing. David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. https://opemai..om//eesse wttp://///sseeeeeeeeeeeeeeeeeeee
ne-world-simulatee 2024.
[3] Weiwei Cai, Weicai Ye, Peng Ye, Tong He, and Tao Chen. Dynasurfgs: Dynamic surface reconstruction with planar-based gaussian splatting. arXiv preprint 24972 2024.
10

---

[4] Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng. and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 2017, pages 667-676. IEEE Computer Society, 2017.
[5] Danpeng Chen, Hai Li, Weicai Ye, Yifan Wang. Weijian Xie, Shangjin Zhai, Nan Wang, Haomin Liu, Hujun Bao, and Guofeng Zhang. PGSR: Planar- Gaussian Splatring for Efficient and High-Fidelity Surface Reconstruction. arXiv preprint arXiv:2406. 06521, 2024.
[6] Junyi Chen, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Where am i and what will
i see: An regresseee model for spatial localization and view prediction. arXiv preprint arXiv:2410. 18962, 2024.
[7] Junyi Chen, Weicai Ye, Yifan Wang, Danpeng Chen, Di Huang, Wanli Ouyang, Guofeng Zhang. Yu Qiao, and Tong He. Gigags: Scaling up glanar-based 3d gaussians for large scene surface reconstruction. arXiv preprint arXiv:2409. 06680, 2024.
[8] Yiwen Chen, Tong He, Di Huang, Weicai Ye, Sijin Chen, Jiaxiang Tang, Xin Chen, Zhongang Cai, Lei Yang, Gang Yu, el. Meshanything: Artist-created mesh generation with autoregressive transformers. arXiv preprini urXiv:2406. 10163, 2024.
[9] Zhaoxi Chen, Guangcong Wang. and Ziwei Liu. Text2light: Zero-shot text-driven HDR panorama generation. ACM Trans. Graph., 4166):19:: 2022.
[10] Zheng Chen. Yan-Pei Cao, Yuan-Chen Guo, Chen Wang, Ying Shan, and Song-Has Hai Zhang. Panogrf: Generalizable spherical radiance fields for wide-baseline panoramas. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Infor- mation Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023.
[11] Zheng Chen, Chen Wang, Yuan-Chen Guo, and Song-Hai Zhang. Structnerf: Neural radi- ance fields for indoor scenes with structural hints. IEEE Trans. Pattern Anal. Mach. Intell., 45(12)155944155055 2023.
[12] Xiao Cui, Weicai Ye, Yifan Wang, Guofeng Zhang, Wengang Zhou, Tong He, and Houqiang Li. Streetsurfgs: Scalable urban street surface reconstruction with planar-based gaussian splatting. arXiv preprint 4rXiv:2410.04354, 2024.
[13] Mengyang Feng, Jinlin Liu, Miaomiao Cui, and Xuansong Xie. Diffusion360: Seamless 360 degree panoramic image generation based on diffusion models. CoRR, abs/2311.13311 2023.
[14] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et al. Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusson transformers. arXiv preprint 0555v:2405.05945, 2024.
[15] Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualle Pratul Srinivasan, Jonathan T Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view diffusion models. arXiv preprint arXiv:2405.100
2024.
[16] Marc-André Gardner, Kalyan Sunkavalli, Ersin Yumer, Xiaohui Shen, Emiliano Gambaretto, Christian Gagné, and -rançooss Lalonde. Learning to predict indoor illumination from single image. arXiv preprint 1rX4vv1704.000
2017.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 10.6:22/00609685, 2021.
[19] Chenxi Huang, Yuenan Hou, Weicai Ye, Di Huang, Xiaoshui Huang, Binbin Lin, Deng Cai, and Wanli Ouyang. Nerf-det++: Incorporating semantic cues and perspective-aware depth supervision for indoor multi-view 3d detection. arXiv preprint arXiv:2402. /4464, 2024.
[20] Zehuan Huang, Hao Wen, Junting Dong, Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai, and Lu Sheng. Epidiff: Enhaneing multi-view synthesis via localized constant diffusion. CoRR, abs/2312.06722 2023.
11

---

[21] Hai Li, Weicai Ye, Guofeng Zhang, Sanyuan Zhang, and Hujun Bao. Saliency guided subdivision for single-view mesh reconstruction. In 2020 International Conference on 3D Vision (3DV). pages 1198-1107, IEEE, 2020.
[22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 2023.
[23] Longwei Li, Huajian Huang, Sai-Kit Yeung, and Hui Cheng. Omnigs: Omnidirectional gaussian splatting for fast radiance field reconstruction using omnidirectional images. CoRR, abs/2404.03202 2024.
[24] Renjie Li. Panwang Pan, Bangbang Yang, Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, and Zhiwen Fan. 4k4dgen: Panoramic 4d generation at 4k resolution. arXiv preprint arXiv:2406. 13527, 2024.
[25] Zhen Li, Lingli Wang, Mofang Cheng. Cihui Pan, and Jiaqi Yang. Milti-view inverse rendering for large-scale
indoor scenes. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canuda, June 17-24, 2023, pages 12499- 12509, IEEE, 2023.
[26] Jiuming Liu, Guangming Wang, Weicai Ye, Chaokang Jiang, Jinru Han, Zhe Liu, Guofeng Zhang, Dalong Du, and Hesheng Wang. Difflow3d: Toward robust incertainty-awan scene flow estimation with iterative diffusion-based refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15109-15119, 2024.
[27] Jiuming Liu, Ruiji Yu, Yian Wang, Yu Zheng, Tianchen Deng, Weicai Ye, and Hesheng Wang. Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy. arXiv preprint arXiv:2403.0mm 06467, 2024.
[28] Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng. Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d difffuson. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10072-10083 2024.
[29] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-33 Zero-shot one image to 3d object, 2023.
[30] Xiangyu Liu, Weicai Ye, Chaoran Tian, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang Coxgraph: multi-robot collaborative, globally consistent, online dense reconstruction system. In 2021 IEEE/RS. International Conference on Intelligent Robots and Systems (IROS), pages 8722 8728. IEEE, 2021.
[31] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-connisee images from a single-vies view image. CoRR, abs/2309.034 .04453,,.333,..... 2023.
[32] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, and Wenping Wang. Wonder3d: Single image to 3d using cross-domain diffusion. CoRR, abs/2310. 15008, 2023.
[33] Zhuqiang Lu, Kun Hu, Chaoyue Wang, Lei Bai, and Zhiyong Wang. Autoregressive omni-awar outpainting for open- vocabulary 360-degree image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 14211-14219, 2024.
[34] Yuhang Ming, Minyang Xu, Xingrui Yang, Weicai Ye. Weihan Wang, Yong Peng, Weichen Dai, and Wanzeng Kong. Viper: Visual incremental place recognition with adaptive mining and lifelong learning, arXiv preprint arXiv:2407.21416 2024.
[35] Yuhang Ming, Weicai Ye, and Andrew Calway. idf-slam: End-to-end rgb-d slam with neural implicit mapping and deep feature tracking. arXiv preprint arXiv:2209 (tt.:22299.7999,, 2022.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, PMLR, 2021.
pages
12

---

[37] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexan- der Clegg. John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat- 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AT. In Thirty-fisth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685. IEEE, 2022.
[39] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[40] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. Zeronvs: Zero-shot 360-degree view synthesis from a single real image. CoRR, abs/2310. 17994,, 2023.
[41] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[42] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero 1oool1+++++ a single image to consistent multi-view diffusion base model. CoRR, abs/2310.15110,, 2023.
[43] Yichun Shi, Peng Wang. Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi- view diffusion for 3d generation. CoRR, abs/2308. 16512, 2023.
[44] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Frédo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In NeurlPS, pages 19325- 2021.
[45] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1 1-18. Springer, 2025.
[46] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.1663 2023.
[47] Shengji Tang, Weicai Ye, Peng Ye, Weihao Lin, Yang Zhou, Tao Chen, and Wanli Ouyang. Hisplat: Hierarchical 3d gaussian splatting for generalizable sparse-view reconstruction. arXiv preprint arXiv:2410.062. 2024.
[48] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mydiffu- sion: Enabling holistic multi-view image generation with correspondence-awarc diffusion. In NeurIPS, 2023.
[49] Ziyu Tang, Weicai Ye, Yifan Wang, Di Huang, Hujun Bao, Tong He, and Guofeng Zhang. Nd-sdf: Learning normal deflection fields for high-fidelity indoor reconstruction. arXiv preprint arXiv:2408.1 12598, 2024.
[50] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, 0is//2377.0928.. 2023.
13

---

[51] Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Consistent view synthesis with pose-guided diffusion models. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 16773-16783. IEEE, 2023.
[52]
Chen Wang, Jiatao Gu, Xiaoxiao Long, Yuan Liu, and Lingjie Liu, GECO: generative image-to- 3d within second. CoRR, abs/2405 2ttp/22405.20277, 2024.
[53] Guangeong Wang, Peng Wang, Zhaoxi Chen, Wenping Wang. Chen Change Loy, and Ziwei Liu. Perf: Panoramic neural radiance field from a single panorama. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2024.
[54] Hai Wang, Xiaoyu Xiang, Yuchen Fan, and Jing-Hao Xue. Customizing 360-degree panoramas through text-to-image diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1933-4943 2024.
[55] Hai Wang, Xiaoyu Xiang, Yuchen Fan, and Jing-Hao Hao Xue. Customizing 360-degree panoramas through text-to-image diffusion models. In WACV, pages
IEEE, 2024.
[56] Jionghao Wang, Ziyu Chen, Jun Ling, Rong Xie, and Li Song. 360-degree panorama generation from few unregistered nfov images. arXiv preprint arXiv:2308. 14686, 2023.
[57] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. CoRR, abs/2312 12.02201, 2023.
[58] Yifan Wang, Di Huang, Weicai Ye, Guofeng Zhang, Wanli Ouyang, and Tong He. Neu- rodin: A two-stage framework for high-fidelity neural surface reconstruction. arXiv preprint arXiv:2408. 10178, 2024.
[59] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[60] Tianhao Wu, Chuanxia Zheng, and Tat-Jen Cham. Ipo-Idm: Depth-aided 360-degree indoor rgb panorama outpainting via latent diffusion model. arXiv preprint (rXivv2307.03177 2023.
[61] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujum Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403.14621, 2024.
[62] Bangbang Yang, Wenqi Dong, Lin Ma, Wenbo Hu, Xiao Liu, Zhaopeng Cui, and Yuewen Ma. Dreamspace: Dreaming your room space with text-driven panoramic texture propagation. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 650-660. IEEE, 2024.
[63] Jiayu Yang, Ziang Cheng, Yunfei Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7079-7088, 2024.
[64] Weicai Ye, Shuo Chen, Chong Bao, Hujun Bao, Mare Pollefeys. Zhaopeng Cui, and Guofeng Zhang. IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2023.
[65] Weicai Ye. Xinyue Lan, Shuo Chen, Yuhang Ming, Xingyuan Yu, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Pvo: Panoptic visual odometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9579-9599 June 2023.
[66] Weicai Ye, Xinyue Lan, Ge Su, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang. Hybrid tracker with pixel and instance for video panoptic segmentation. arXiv preprint urXiv:2203.012111 2022.
[67] Weicai Ye, Hai Li, Tianxiang Zhang, Xiaowei Zhou, Hujun Bao, and Guofeng Zhang, Super- plane: 3d plane detection and description from a single image. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 207-215. IEEE, 2021.
[68] Hailin Yu, Youji Feng, Weicai Ye, Mingxuan Jiang, Hujun Bao, and Guofeng Zhang. Improving feature-based visual localization by geodetry-aided matching. arXiv preprint Nix::22110087/11 2022.
14

---

[69] Hailin Yu, Weicai Ye, Youji Feng, Hujun Bao, and Guofeng Zhang, Learning bipartite graph matching for robust visual localization. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 146-155. IEFF, 2020.
[70] Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, and Jianfei Cai. Taming stable diffusion for text to 360o panorama image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[71] Jinsong Zhang and Jean- -aan-François Lalonde. Learning high dynamic range from outdoor panora- mas. In Proceedings of the IEEE International Conference on Computer Vision, pages 4519- 4528, 2017.
[72] Tianxiang Zhang, Chong Bao, Hongjia Zhai, Jiazhen Xia, Weicai Ye, and Guofeng Zhang. Arcargo: Multi-device integrated cargo loading management system with augmented reality. In 2021 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Inil Conf on Cyber Science and Technology Congress
pages 341-348.
IEEE, 2021.
[73] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Parl IX, volume 12354 of Lecture Notes in Computer Science, pages 519-535. Springer, 2020.
[74] Haiyang Zhou, Xinhua Cheng. Wangbo Yu, Yonghong Tian, and Li Yuan. Holodreamer: Holistic 3d panoramic world generation from text descriptions. arXiv preprint arXiv:2407.15187, 2024.
[75] Haoyi Zhu, Yating Wang, Di Huang, Weicai Ye, Wanli Ouyang, and Tong He. Point cloud matters: Rethinking the impact of different observation spaces robot learning. arXiv preprint arXiv:2402.0222 02500, 2024.
15

---

A Spherical Epipolar Line Computation
Given the pixel coordinates of point in the panorama of the target view i, the corresponding epipolar line in the panorama of the source view j can be calculated. First, according to Eq. (T. the camera
coordinates p in the target view are computed. Based on the relative pose {R i T ). p' the source view are calculated:
p' = R p T
(8)
Simultaneously, the coordinates o' of the camera origin 0,00 in the target view projected onto the source view are computed:
O' = R
o T
(9)
Then, need find the plane I. containing the three points p', and o' in the camera coordinate system of the source view j. The intersection of this plane with the spherical surface is the desired epipolar line. The equation corresponding plane is:
AX BY CZ + =
(10)
Substituting the coordinates of the three points into the equation yields the coefficients A, B, C, and D:
A 2o' Upp
2p' Yo'
C
Zp' Yo Io' 1/p'
B Fo' Tp' 2p' Fio C
(11)
Up To Yo' Tpp'
D 0.
To simplify the representation of the equation for plane L, we introduce new coefficients and dz!
a1 Zo' Yp 2p' Yoo
Tp' Ho' Upp
(12)
2o' Tp' Zp Eo'
02 U/ Vo' Yo' Tip'
the equation plane can be simplified to:
a1 X + 0g 2
(13)
According to Eq and
and Eq
2 the epipolar line equation in the source view can be obtained:
a1 sin
2x
cos
2 2pir W
Upin H
arctan
/ + 0.5
(14)
where Tpix and ypiz are the corresponding pixel coordinates.
B Experiment Details
Dataset Processing In single-view panorama generation, we select 100 scenes from HM3D B7 and render cube maps from random viewpoints within each scene, which are then stitched into panoramas through interpolation. However, due to the imperfect quality of the corresponding scene meshes, which may have missing parts, we filter out images with high proportion of zero-depth values based on their corresponding depth maps, ultimately creating dataset of 8,508 panoramas.
16

---

Table 5: Quantitative Perspective Images Comparisons with Baseline Methods
Method
FIDI
1St
MVDiffusion
188.16
1.82
PanFusion
213.19
2.61
Pano-SD(Ours)
164.75
1.96
In the multi-view panorama generation task, we render images based on camera trajectories across scenes. To accommodate the first stage of training, we select camera trajectories with minimal changes. In contrast, for the second stage, the dataset comprises panoramas with more significant camera displacements between consecutive frames. Each training dataset consists of multiple panoramas, their corresponding pose matrices, and text descriptions. To construct a multí-view panorama dataset with nearly identical image content, we select the first few frames from each camera trajectory, as the camera movement between these frames is minimal and they are captured within the same scene. For datasets where the image content changes, the construction needs to be performed across the entire camera trajectory. By projecting the panorama from the subsequent frame onto the viewpoint of the previous frame using spherical projection, we compare the pixel value differences between the two. If more than 40% of the pixel values differ, we consider new content to have been generated between the two frames. After filtering, there are 19,739 data sets with nearly identical image content and 18,704 data sets with differing image content.
Compare with Baseline Methods
In the quantitative analysis, the FID, IS, and CS values referenced from the results reported in PanFusion[Z0] [70) which were obtained by testing on the same dataset after training. For the qualitative analysis, we directly employ their jointly trained model on both indoor and outdoor datasets to generate LDR (low dynamic range) panoramas based on text descriptions.
Tegt2Light 9]
PanPusion 770 : To compare the quality of the generated panoramas, also train the model using panoramas stitched from MP3D skybox images. We employ the same dataset split and text descriptions as in [70]. which includes 9,820 panoramas for training and 1,092 for evaluation. In the qualitative analysis, we use our own dataset constructed from HM3D37). The panoramas in our dataset do not suffer from blurriness at the bottom and top regions, but the overall image quality may be slightly lower.
Compare with MVDream We adopt the method from MVDream 43 by formatting the B x N H W x C tensor input to the self-attention module as B x NHW C. This allows the model to simultaneously integrate features from all viewpoints during the self-attention computation, thereby improving the consistency among the generated multi-view panoramas. However, since MVDream designed for multi-view image generation objects, the camera pose matrices used in their method only contain rotation information without translation. In contrast, when generating camera embeddings, we use pose matrices that include both rotation and translation information, which increases the learning difficulty for the model. This may be one of the reasons why using this method does not yield particularly ideal results.
Transform Panoramas to Perspective Views We converted the generated panoramas into perspec- tive images and conducted quantitative comparisons, shown in Tab. 5] Experiments show that our method achieves the lowest FID, while our method higher than MVDiffusion in IS and slightly lower than PanFusion. It should be noted that MVDiffusion directly generates perspective images and then stitches them into panoramas. It is not the ERP format and does not have the top and bottom parts. Our panorama generation speed is faster.
17

---

B.1
Applications
B.1.1
Text to Vingle-View Panorama
B.1.2 Text to Multi-View Panoramas
DiffPano can generate panoramic video frames with large camera pose spans and multi-view consis- tency based on diverse textual descriptions, thus achieving the effect of text-to-panoramic video, as shown in the Fig
PanFusinn
MVDream
DiffPano (Ours)
beightt kitches with white cablncts, farmshous slnk, harge Idaad."
Figure 6: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PunFusion.
Pan FunFusion
MY Dream
DitPaao (Ours)
"A rustic living with stone firplice, leather safa, and mondes coffee rable."
Figure 7: Qualitative Comparisons of Text to Panoramic Videos. Ours MV Dream PanFusion.
C Network Architecture and Training Details
Network Architecture Our LoRA-based single-view panorama generation model produces panora- mas with resolution 512x 1024. the multi-view panorama generation approach, we generate continuous frames of panoramas with a resolution of 256x512. Generating multiple frames of 512x 1024 resolution panoramas simultaneously would consume a significant amount of compu- tational resources. Moreover, our experiments reveal that generating multi-frame high-resolution panoramas requires an exceptionally long training time to improve the quality of the generated images. The network architecture of our multi-view panorama generation model is shown in Table 0 and Table7
Training Details We fine-tuned the Stable Diffusion v1.5 model using the LoRA method for single-view pano-based synthesis. The training was conducted on A100 GPUs with 80GB memory for 100 epochs (approximately 6.5 hours), with learning rate of le-4 and batch size of 6. For
18

---

Table 6: Network architecture of DiffPano-1
Layer
Output
Additional Inputs
Latent Map
x 32 x 64
Conv.
320 x 32 64
CrossAttnDowwnBlock
ResBlock
320 x 32 x 64
Time emb.
AttnBlock
320 x 32 64
Prompt emb.
ResBlock
320 x 32 x 64
Time emb.
AttnBlock
320 32 64
Prompt emb.
DownSampler
320 x 16 X 32
CrossAtrnDowwnBloekk2
ResBlock
640 16 32
Time emb.
(9)
AutnBlock
640 x 16 x 32
Prompt emb.
(10) ResBlock
640 16 32
Time emb.
(11)
AttnBlock
640 x 16 x 32
Prompt emb.
(12)
DownSampler
640 x 8 16
CrossAttnDownBllok3
(13) ResBlock
1280 x 8 x 16
Time emb.
(14) AttnBlock
1280 x 16
Prompt emb.
(15) ResBlock
1280 x 8 x 16
Time emb.
(16)
AttnBlock
1280 X 16
Prompt emb.
(17)
DownSampler
1280 X 4 x 8
DownBlock
(18)
ResBlock
1280 8
Time emb.
(19)
ResBlock
1280 X 4 x 8
Time emb.
MidBlock
(20) ResBlock
1280 8
Time emb.
(21)
EAModule
1280 X 4 x 8
(22)
AttnBlock
1280
8
Prompt emb.
(23)
ResBlock
1280 x 4 x 8
Time emb.
UpBlock
(24) ResBlock
1280 8
(19), Time emb.
(25) ResBlock
1280 X 4 x 8
(18), Time emb.
(26) ResBlock
1280
8
(17), Time emb.
(27) EAModule
1280 X 4 x 8
(28) UpSampler
1280 x 16
CrossAttnUpBlockl
(29) ResBlock
1280 x 8 x 16
(16), Time emb. Prompt emb.
(30) AttnBlock
1280 x 16
(31) ResBlock
1280 x 8 x 16
(14), Time emb.
(32)
AunBlock
1280 x 16
Prompt emb.
(33) ResBlock
1280 x 8 x 16
(12), Time emb.
(34) AttnBlock
1280 x 16
Prompt emb.
(35)
EAModule
1280 X 8 x 16
(36) UpSampler
1280 X 16 x 32
19

---

Pan PanFusion
MV MVDreea
DIIIPARO (Ours)
diylish patto with dining set, an unbrella, and pertod plats."
Figure 8: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PanFusion.
PunFosion
MVDream
DiffPano (Ours)
welceming mudroam cobbies, books, and beach"
Figure 9: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream vs PanFusion.
multi-view panoramas generation, we conducted training on 8 80G A A100 GPUs with batch size of and learning rate of Te-5. Each GPU utilized approximately 50% of memory. The two-stage training process involved 100 epochs for each stage, with total training time of approximately days.
D Societal Impact
Since method achieve scalable, consistent, and diverse multi-view panoramas, it has many potential applications, such as unlimited room roaming in VR, interior design preview, embodied intelligent robot exploration, etc.
20

---

Pan
MVDream
DifiPano (Ours)
"A finetional laundry with washer, builbin board."
Figure 10: Qualitative Comparisons of Text to Panoramic Videos. Ours vs MVDream PanFu- sion.
with warden fostuns stairnoy, and muliiple duars leading variaus
"A kltchen while bardword floers, black cabineti and stainles applianco. kinchen with black cabinets, hardword florr, and stavs, connected nnother reom with wood Boors."
spaciass
hedmomm with banlwand floees reiling bed. the wall, festering white and bowna floar. two-turry hous with small stairwyy, hardwosd Boors. us daseway kealing bedreom."
rouch, IV, and desk. A well- furnshee roums with doer, featuring couch, desk, TV the wall with carpet."
cusy wandon the
hoase with stairway leading second floor surroanded by wood floors and aboraed with picture wall."
Figure 11: Qualitative Results of Text to Panoramic Videos. DiffPano can generate scalable and consistent panorama videos.
21

---

remsdeeed garage with liems the walls, wall-it hallwy with a deer and various furaiture."
walls
worden
light
deor, furiished with
kitchenetto, The furuishedd with musden fumitune. A diaing cabde and chairs puinting
ltt
cuuches and moanied the wall.
spacioss with hardwood floees, kitchen bedroam with warden floors anda
wall, featuring thing and dining consiiatiio, and white csuch."
and dining and old Sathisned teuck." bed. The rosm foraisbed with rug."
Figure 12: Qualitative Results of Text to Panoramic Videos. DiffPano generate scalable and
consistent panorama videos.
"A hume office with barge desk, hookshelves, hardwoud Blsirs and cunfortable chair. hallway with hardwaed fisor, kaading to bonne office with a larns dook and boskabelves.
"A hodraam with a berl and dreerr, featoring windure with curtaias and binds and a TV on top of the drvwsr.. cucs bedreom with bed, dreswe, and TV. surrondedd by a beowa carpei."
"A rury living room with purple conch, a firaplars, and rag, The coiling lace light un it. living room with white stairease. weoden flores, couch, and an opea door that keads to smother recm."
"A small roem with mueden doer and bed. frotaring whitu wall with picturs.
A spacious room with corpeted floe,, hallway leading to bedreoms with woodce doees and confectable bod."
Figure 13: Qualitative Results Text to Panoramic Videos. DiffPano can generate scalable and
consistent panorama videos.
22

---

Table 7: Network architecture of DiffPano-2
Layer
Output
Additional Inputs
CrossAttnUBloock?
(37)
ResBlock
640 x 16 x 32
(11), Time emb.
(38)
AttnBlock
640 x x 32
Prompt emb.
(39)
ResBlock
640 x 16 x 32
(9), Time emb.
(40)
AttnBlock
640 x 16 x 32
Prompt emb.
(41)
ResBlock
640 16 x 32
(7), Time emb.
(42)
AttnBlock
640 x 16 x 32
Prompt emb.
(43)
EAModule
640 16 x 32
(44)
UpSampler
640 x 32 x 64
CrossAttnUBBlock3
(45)
ResBlock
320 x 32 x 64
(6), Time emb. Prompt emb.
(46) AttBBlock
320 x 32 x 64
(47)
ResBlock
320 32 x 64
(4), Time emb.
(48)
AttnBlock
320 x 32 x 64
Prompt emb.
(49) ResBlock
320 32 x 64
(2), Time emb.
(50)
AttnBlock
320 x 32 x 64
Prompt emb.
(51)
EAModule
320
64
(52)
GroupNorm
320 x 32 x
(53) SiLU
320 x 32 x 64
(54)
Conv.
4 32 x 64
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims stated in the abstract and introduction accurately reflect the contributions and scope of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all the information for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We will open source the code and data after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
cooooooollyy ll for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipps.cc/pulli///uiii wwwiiicgguie///eeeeeee CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe all the implementation details in the Experiment chapter.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our method requires large amount of graphics cards and resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the graphic card used and training time in Experiment chapter.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We confirm that our research adheres to all aspects of the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We do not generate any pretrained models image generators that may be misused.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have ensured that all assets used in our research are properly credited to their original creators.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "0WPCVOdjUK", "reviewer": "Reviewer_2LuU", "summary": "The paper presents a novel framework called DiffPano for scalable and consistent text-to-panorama generation. The authors first build a panoramic video-text dataset, then propose a spherical epipolar-aware diffusion model to generate multi-view consistent panoramic images, addressing the limitations of existing single-view panorama generation methods. Extensive experiments demonstrate that the proposed DiffPano framework can generate scalable, consistent, and diverse panoramic images from text descriptions.", "questions": "What is the use case for this work? \n\nWhat kinds of downstream applications need multi-view panorama images?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper is well-written and easy to follow.\n- This is the first work that tackles the research direction of text to multi-view panorama generation.\n- The authors have created a large-scale panoramic video-text dataset, which is a valuable resource for the community.\n- The proposed spherical epipolar-aware diffusion model is a novel and technically sound approach to address the challenge of generating multi-view consistent panoramic images from text.\n- The experimental results show the effectiveness of the DiffPano framework in generating scalable, consistent, and diverse panoramic images.", "weaknesses": "- It seems like this work can only do in-distribution generation (indoor scene generation only) and cannot generalize to arbitrary unbounded outdoor scenes.\n\n- This paper does not address why generating multi-view ERP panoramas is meaningful. Is the multi-view output format more helpful for later 3D reconstruction? And why?\n \n- Typos: \n\nL34: see Sec. ??", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer 2LuU for his constructive suggestions and \nappreciate comments like \"well-written\", \"easy to follow\", \n\"first work for text to mv panorama\", \"valuable panoramic video-text dataset\", \n\"novel and technically sound approach of spherical epipolar-aware diffusion\", \n\"effectiveness in generating scalable, consistent, and diverse panoramic images\".\nWe will gladly incorporate all the feedback in the revised version. \n\n1) The main concern of reviewer 2LuU is the generalizability of DiffPano and whether it can generate outdoor scenes. \nTo our knowledge, there is no outdoor multi-view panoramic video dataset. Although DiffPano is currently only trained on the indoor multi-view panoramic dataset, as suggested by the reviewer, we try to generate outdoor panoramic scenes. \nIn the global response, we generated some outdoor multi-view panoramas of DiffPano, which show that DiffPano has good generalization. Of course, it is not perfect. We can explore the 3D reconstruction of outdoor scenes and put the 3D scenes \ninto the panoramic video-text data pipeline to create a large-scale outdoor panorama dataset to improve the generalization of DiffPano further. We leave this as future work.\n\n2) Reviewer 2LuU is also concerned about “why generating multi-view ERP panoramas is meaningful”. \nA single-view panorama can only represent local areas, missing many parts of the scene. \nA multi-view panorama is controllably generated with camera poses, which can generate the missing parts of a single view. \nWe visualized the difference between multi-view and single-view in the global response. \n\n3) As for the use cases of this work, we support single-view panorama generation or multi-view panorama generation \nwith text prompts and camera poses. We additionally provide a generation framework with image conditions, \nwhich can further achieve the effect of panoramic video generation, as shown in the global response. We will add it to the revised version.\n\n4) Reviewer 2LuU would like to know about the downstream applications of multi-view panoramas. \nMulti-view panoramas have many applications, such as room roaming in VR (reviewer Yygs recognized this application), \ninterior design preview, embodied intelligent robot exploration, etc. For example, in VR applications, \nexisting single panorama generation can only obtain local information about scenes, leading to limited roaming scapes, \nwhile multi-view panorama generation can achieve unlimited roaming from one room to another.\n\n5) Typos: We will polish the writing and proofread the revised version, including the typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tA2kbeHn7E", "reviewer": "Reviewer_7HQE", "summary": "This work proposes a text-driven panorama generation framework to achieve scalable, consistent, and diverse panoramic scene generation. In particular, a spherical epipolar attention module with relative poses is designed to ensure multi-view consistency. Moreover, a comprehensive panoramic video-text dataset is constructed, which contains millions of consecutive panoramic frames with corresponding depths, camera poses, and text descriptions. Extensive experiments demonstrate the superiority of the proposed method beyond previous works.", "questions": "Can the proposed framework be extended to panoramic video generation? More insightful discussions are expected to be provided.", "limitations": "This work lacks some customized metrics for evaluating the property of panoramic images, which should be different from those designed for the perspective images.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This work is a pioneer in exploring the scalable multi-view panorama generation task from text descriptions.\n+ A diverse and rich panoramic video-text dataset, which shows promising potential to promote the community of panoramic generation.\n+ The proposed framework looks compact and practical to implement.\n+ Overall, I am in favor of the presentation of this work, it is well-structured and easy to follow. The generated panoramic results also look plausible and visually pleasant.", "weaknesses": "- For the motivation of constructing the panoramic video-text dataset, the authors argue that the corresponding text descriptions in previous panoramic datasets are not precise enough. However, such a conclusion needs to be supported by some quantitative metrics or data statistics.\n- The seamless content is an important property in the generated panoramas. Namely, the panorama can be stitched from left to right boundaries without a noticeable edge effect. This work performs data augmentation to improve such a left-right continuity. Have the authors considered the cylinder convolutions like previous panoramic vision works? For example, \"Cylin-Painting: Seamless 360° Panoramic Image Outpainting and Beyond\" and \"Spherical Image Generation From a Few Normal-Field-of-View Images by Considering Scene Symmetry\" use a cylinder-like convolution or circular padding to ensure the seamless content generation of panoramas. The brief reviews and discussions of these works are also expected to be presented in this work.\n- Some previous works propose specific metrics to evaluate the panorama model. For example, \"PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation\" proposes the Left-Right Consistency Error (LRCE) to quantitatively measure the consistency of the left-right boundaries by calculating the horizontal gradient. The authors are suggested to add this metric for generated panorama evaluations in experiments.\n- Minor points: There are some typos in the manuscript. For example, in line 34: \"(see Sec. ??)\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 7HQE for his constructive suggestions and appreciate comments like “pioneer work in scalable multi-view panorama generation\", \"diverse and rich panoramic video-text dataset promote the community of panoramic generation\", \"compact and practical framework\", \"well-structured and easy to follow\", \"plausible and visually pleasant panoramic results\". \nWe will gladly incorporate all the feedback in the revised version. \n\n1) Text descriptions between previous panoramic datasets (PanFusion) with our proposed panoramic video-text dataset: \nPanFusion uses BLIP2 to directly generate text descriptions for panoramas. It only has four or five words and is very concise. \nThe CLIP Score (CS) cannot reflect the accuracy of the text description, and the PanFusion dataset has the problem of blurry top and bottom. In contrast, our panoramic video dataset construction pipeline first generates text descriptions for perspective images using BLIP2, then uses LLM to summarize, which can obtain more detailed text descriptions. At the same time, the top and bottom of our panoramas are clear, and the dataset is larger (millions of panoramic keyframes ). We also provide the camera pose of each panorama, the corresponding panoramic depth map, etc. We show the difference between the PanFusion dataset with our dataset in the global response.\n\n2) Consider cylinder convolutions or padding to maintain left-right continuity: \nAs suggested, we compare cylinder padding without data augmentation and the data augmentation method used in this paper to improve the left-right consistency. Experiments of 1092 samples show that our simple data augmentation scheme can obtain higher CLIP Score (CS) and Inception Score (IS), as well as lower Fréchet Inception Distance (FID). To further verify the consistency, we propose a new left-right consistency evaluation method. First, we simply cut the generated panorama into left and right images, and then use the image-matching algorithm to calculate the correspondence. The number of correspondences is called as Left-Right Matching Pairs (LRMP) metric. Experiments show that our method can obtain more correspondence.\n| Methods | IS↑ | FID↓ | CS↑ | LRMP↑ |\n|:-------:|:----:|:----:|:-------:|:-------:|\n| Circular Padding    |   3.81   |   68.31  | 23.51 | 143   |\n| Data Augmentation   |   4.07   |   61.43   | 23.58   | 149   |\n\n3) Specific metrics for panorama, such as Left-Right Consistency Error (LRCE):\nThe Left-Right Consistency Error (LRCE) metric proposed by PanoFormer evaluates the accuracy of panoramic depth estimation \nby comparing the gap between the leftmost and rightmost horizontal gradients of the predicted panoramic depth map \nand the ground-truth panoramic map. In the panorama generation task of this paper, we do not have the ground-truth panoramic depth, \nand this method does not generate panoramic depth. The LRCE metric is not applicable to the work of this paper. \nInstead, we propose a Left-Right Matching Pairs (LRMP) metric to calculate the left-right consistency by using matching algorithms. \nWe follow Text2Light to report Fr´echet´s Inception Distance (FID) and Inception Score (IS) on panoramas to measure realism and diversity. Additionally, the CLIP Score (CS) is used to evaluate the text-image consistency. While FID is widely used for image\ngeneration, it relies on an Inception network trained on perspective images, thus less applicable for panorama images. \nWe adopt a variant of FID customized for panorama, Fr´echet Auto-Encoder Distance (FAED) in PanFusion, which is used to better compare realism. The quantitative results are as follows:\n| Methods | FID↓ | IS↑ | CS↑ | \tFAED↓ | Inference time↓ | LRMP↑ |\n|:-------:|:----:|:----:|:-------:|:-------:|:-------:|:-------:|\n| Text2Light |  76.5   |  3.60   | 27.48   | 97.24  | 80.6s | 127 |\n| PanFusion | 47.62 | 4.49 | 28.73 | 6.36 | 15.1s | 133 |\n| Ours\t| 48.52| \t3.30| \t28.98| \t10.70| 5.1s| 149\n\nExperiments show that our method achieves the best CLIP Score, the best left-right consistency, and the fastest inference speed, which makes scalable panorama generation possible. Although our method is slightly weaker than PanFusion in FID, IS, and FAED metrics, it cannot obtain clear images due to the blur at the top and bottom, and the complex network makes its inference speed too slow.\n\n4) Extended to panoramic video generation? Yes, our method can be extended to panoramic video generation. \nThis paper mainly studies the method of text-to-multi-view panorama generation. \nWe have recently studied the method of image-to-multi-view panorama generation. \nTo generate panoramic videos, we can first generate multi-view images of different rooms with large pose changes conditioned on the text, and then run the image-to-multi-view panorama generation conditioned on the generated multi-view panoramas, \nso that it can be expanded to generate longer panoramic videos. We will add this part to the revised version. \nWe show some panoramic video effects in the pdf of the global response. It is recommended to open it with Adobe reader software so that you can see these videos.\n\n5) typos: We will polish the writing and proofread the revised version, including the typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eiImne4zk7", "reviewer": "Reviewer_K8Nd", "summary": "The paper focusing on the simulation of panoramic images and the annotation of text descriptions. The primary contribution is the introduction of a LoRA-based fine-tuning technique, aimed at enhancing the performance of benchmark datasets. The authors propose a pipeline that integrates simulated panoramic data with annotated textual descriptions to improve the accuracy and robustness of 3D models.", "questions": "Can the authors clarify the experimental setup and parameters used for LoRA-based fine-tuning? More details on the hyperparameters and training process would be helpful.\n\nAre there any specific challenges or limitations encountered when simulating panoramic images (especially the difficulty for outdoor/synthetic simulation)? How were these addressed in the proposed pipeline?", "limitations": "Although this paper introduce an interesting dataset and problem, it is unclear how was the dataset quality, as the supplementary materials are empty. Could the author clarify the date for releasing the datasets? The scope of the paper is more close to the benchmark and dataset track where the datasets can be used to insight the 3D computer vision researches.\nBesides, the authors should discuss the scalability of their approach to larger and more complex scenarios (indoor, outdoor, synthetic, cartoon). Additionally, they should address any potential biases introduced by the simulated data and how these might affect the generalizability of the results.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of simulating panoramic images combined with text annotation is interesting, potentially offering new ways to enhance 3D vision using panorama as representation.\nThe paper is well-structured, with a clear explanation of the proposed method and the experimental setup.", "weaknesses": "Lack of Technical Contribution: The paper primarily focuses on the application of existing techniques (panoramic simulation, text annotation and LoRA-based fine-tuning) rather than introducing new technical innovations. This limits its contribution to the field.\n\nTypos and Clarity Issues: There are several typos and unclear sentences throughout the paper (e.g., \"??\" in line 34), which detract from its overall readability and professionalism.\n\nThe scope of this paper lies in the indoor scene panorama generation, and it is suggested to highlight this keyword in both title and abstract.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer K8Nd for his constructive suggestions and appreciate comments like \"interesting idea\", \"new way to enhance 3D Vision\", \"well-structured\", and \"clear explanation\". However, the reviewer seems to have misunderstood the core content of this paper, \nthe problem it solves, and the core innovation. Here, we point out the factual error. \n\nThis paper aims to perform scalable multi-view panorama generation with text (which can be expanded from one room to another). As far as we know, we are the first to propose text-to-multi-view panorama generation. Previous methods are limited to single-view panorama generation. The core reason is the lack of large-scale multi-view panorama datasets, and single-view panorama generation cannot guarantee the multi-view consistency of the generated panorama. A new framework suitable for multi-view panorama generation is needed. \nTo this end, we first construct a large-scale panoramic video-text dataset. Then, we design a multi-view panorama generation framework. First, we propose a diffusion model for text-to-single-view panorama generation (fine-tuned on the newly proposed panorama dataset using LoRA). To achieve multi-view consistency, we derive a spherical epipolar constraint suitable for panoramas and propose the spherical epipolar multi-view panorama generation framework, which can achieve diverse and consistent scalable panorama generation results. We will gladly incorporate all the feedback in the revised version. \n\n1) Summary: As mentioned above, the reviewer seems to have misunderstood the core mission, problem, solution, and core innovation of this paper. Other reviewers (2LuU, 7HQE, Yygs) can understand the content of this paper relatively well. \nWe suggest that the reviewer re-review this paper to make a reasonable evaluation. \nWe are glad to resolve any doubts of the reviewer. This paper first introduces a novel task for text to multi-view panorama generation. \nSo we need to reconstruct a large-scale panoramic video-text dataset for this task and design a novel multi-view panorama generation framework with novel spherical epipolar attention. This method can generate diverse and consistent scalable panorama generation. \n\n2) Lack of Technical Contribution: The paper focuses on text-to-multi-view panorama generation, rather than single-view panorama generation. It is the essential difference between this paper and the existing single-view panorama generation method. \nWe propose a novel panorama video construction and caption pipeline and create a large-scale panorama video-text dataset. \nThen we design a novel multi-view panorama generation framework with novel spherical epipolar attention.\n\n3) Indoor scene panorama generation: Although this paper focuses on the multi-view panoramic generation of indoor scenes, \nwe also show the results of some outdoor scene generation of this method in the global response, which shows a certain generalization. \nIn the future, we will explore outdoor scene reconstruction to construct large-scale outdoor panoramic video datasets \nto further improve the generalization of multi-view panoramic generation.\n\n4) LoRA-based fine-tuning: It should be emphasized that Lora-based fine-tuning is a method for text to single-view panorama generation \n(which is detailed introduced in Section 4.1), and this paper focuses on text-to-multi-view panorama generation \n(which is detailed introduced in Sec. 4 and 5 and Appendix C). \nHere we add the details of Lora-based fine-tuning: We first fix the stable diffusion model 1.5 and use Lora for fine-tuning. \nThe fine-tuning dataset is selected from our newly proposed video panoramic-text dataset to form 8508 high-quality text-panorama pairs. \nThe learning rate is 0.0001, the batch size is 32, and 8 A100 80GB are used to train 100 epochs.\n\n5) Any specific challenges or limitations encountered when simulating panoramic images? \nOur synthetic panoramic video-text dataset is based on Matterport3D, which is reconstructed from the captured real images.\nBased on this dataset, there will be no domain gap problem at least when generating indoor panoramas. \nWe also try to perform outdoor panora generation, which is shown in the global response.\nExperiments show that our method has a certain degree of generalization in outdoor panorama generation. \nOf course, if we can add a more high-quality outdoor dataset, it may make our model more generalizable. We leave it as future work.\n\n6) Dataset quality of the proposed panoramic video-text dataset? releasing date? \nIn the global response, we show the comparison between our dataset and PanFusion dataset. \nCompared with the PanFusion dataset, our dataset has a clear top and bottom, and the text description is more detailed. \nSince the complete dataset is very large, we plan to release the train/test dataset and \nopen source a complete panoramic video construction and caption pipelines upon acceptance, \nso that the community can create large-scale panoramic video-text datasets according to their needs. \nWe will also release the complete framework upon acceptance so that the community can use and follow our work. \n\n\n7) Closer to benchmark and dataset track? No, as mentioned above, we propose a novel task of text to multi-view panorama generation and propose novel panoramic video construction and caption pipelines to create a large-scale panoramic video-text dataset. Then we design a novel panorama generation framework, consisting of a single-view panorama generation module and a novel spherical epipolar multi-view panorama generation module. \nIt's not just a dataset; the dataset is just one of our core contributions. This paper is suitable for the main paper track.\n\n8) Scalability and generalizability: We demonstrate panoramic video generation and outdoor panorama generation, which shows that our method has good scalability and generalization ability.\n\n9) Typos: We will polish the writing and proofread the revised version, including the typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RwwrLJVKo9", "reviewer": "Reviewer_Yygs", "summary": "This paper introduces the DiffPano framework, designed to generate consistent panoramic images from multiple viewpoints based on a text description of a scene. The authors first create a panoramic video-text dataset from 3D scenes using Habitat Simulator, BLIP2, and an LLM. Using this dataset, they fine-tune the original Stable Diffusion model with LoRA to develop a “single-view panorama-based Stable Diffusion.” This single-view model is then extended to a “multi-view consistent panorama Stable Diffusion” by incorporating a spherical epipolar attention module into the U-Net. Experiments show that DiffPano produces more consistent and higher-quality multi-view panoramic images compared to baseline methods.", "questions": "Missing citation: The data collection process for “Panorama Video Construction and Caption” (Fig. 2) appears to be quite related to the data captioning method proposed in Cap3D [1]. Can you elaborate on how it differs from or is related to their approach? If it is based on their approach, proper citation for the dataset generation pipeline is necessary.\n\n[1] Scalable 3D Captioning with Pretrained Models, Luo et al., NeurIPS 2023", "limitations": "The paper discusses the limitation of the proposed method (hallucination of content).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper defines an interesting task of generating consistent panoramic images from multiple viewpoints, which could be useful for practical applications such as VR environments.\n2. Providing a new dataset specifically designed for multi-view panoramic generation would be quite meaningful, facilitating further studies on panorama generation.\n3. While epipolar-aware attention has been employed in prior works for perspective images [1], this paper effectively adapts a similar concept to panoramic images through a mathematical derivation for the spherical epipolar line.\n\n[1] EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusion, Huang et al., CVPR 2024", "weaknesses": "1. In Sec. 4.1, the authors describe panorama generation as a “style transformation” from the original perspective generation. However, generating a panoramic image requires more than a style change, as a 360-degree panorama must adhere to certain geometric constraints (i.e., when a portion of the panorama is projected into a perspective image, it should appear realistic). To address this, other than the user study, there need to be clear quantitative or qualitative evaluations of the realism of the generated multi-view panoramas. For instance, FAED [1] could be used to compare the realism of the panoramas themselves. Without such evidence, it is concerning whether PanoDiff is indeed generating more geometrically “realistic” panoramic images compared to the MVDream-based baselines for multi-view generation.\n2. This work does not provide any qualitative examples of the perspective projections of the generated panoramas. In previous works on 360 panorama generation, the realism of the perspective view has been a crucial source for assessing the realism of the generated panoramas [2, 3]. Including perspective views would enhance the proper evaluation of the generation quality of the proposed method.\n3. While the paper claims that DiffPano shows high generalizability, there are concerns regarding its performance on diverse text prompts. The single-view model is initially fine-tuned on panoramic images using LoRA, and the multi-view model is subsequently trained based on this fine-tuned model. This two-stage fine-tuning process on panoramic data might hinder the original generation diversity of Stable Diffusion, which is primarily trained on perspective images.\n\n[1] Bips: Bi-modal indoor panorama synthesis via residual depth-aided adversarial learning, Oh et al., ECCV 2022\n\n[2] MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion, Tang et al., NeurIPS 2023\n\n[3] Taming Stable Diffusion for Text to 360 Panorama Image Generation, Zhang et al., CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Yygs for his constructive suggestions and appreciate comments like \"interesting task\", \"useful for practical applications such as VR environments\", \"new dataset quite meaningful\", and \"effectively adapts to panoramic images\".\nWe will gladly incorporate all the feedback in the revised version. \n\n1) The main concern of reviewer Yygs is the quantitative or qualitative evaluations of the realism of the generated multi-view panoramas. We adopt the FAED metric to evaluate the realism. Experiments show that \nour method achieves lower FAED than MVDream and Text2Light, but slightly higher than PanFusion. \n\n| Methods | FID↓ | IS↑ | CS↑ | \tFAED↓ | Inference time↓ | LRMP↑ |\n|:-------:|:----:|:----:|:-------:|:-------:|:-------:|:-------:|\n| Text2Light |  76.5   |  3.60   | 27.48   | 97.24  | 80.6s | 127 |\n| PanFusion | 47.62 | 4.49 | 28.73 | 6.36 | 15.1s | 133 |\n| MVDream | 57.78 | 2.87 | 27.69 |  11. 63| 5.1s | 138 |\n| Ours\t| 48.52| \t3.30| \t28.98| \t10.70| 5.1s| 149\n\nHowever, our CLIP Score achieves the highest results. Note that our method is significantly faster than existing methods, making scalable panoramas possible. \nThe reason is that PanFusion adopts several complex modules in single-view panorama generation, which can improve FAED to a certain extent, but the speed is too slow. A fast pre-training module makes fine-tuning faster and better during multi-view generative network training.\n\n2) Transform the panorama to the perspective views: As suggested, we converted the generated panorama into perspective images and conducted qualitative (shown in global response) and quantitative comparisons. Experiments show that our method achieves the lowest FID, while our method is higher than MVDiffusion in IS and slightly lower than PanFusion. It should be noted that MVDiffusion directly generates perspective images and then stitches them into panoramas. It is not in the ERP format and does not have the top and bottom parts. Our panorama generation speed is faster. \n\n| Methods | FID↓ | IS↑ | \n|:-------:|:----:|:----:|\n| MVDiffusion |  188.16   |  1.82   | \n| PanFusion | 213.19 |2.61| \n| Ours\t| 164.75| 1.96| \t\t\n\n3) Diversity of DiffPano: We show some qualitative panorama results in the global response, which shows our method has good diversity. We also show some outdoor panoramic generation results to demonstrate generalization. Generalization and panorama effects are a trade-off. When we pursue better geometric consistency in panorama generation, generalization may indeed decrease. \nThis requires a balance. A potential solution is to use more panoramic images to train native text-to-panorama generation diffusion models, rather than using LoRA to fine-tune from SD, which can be a future work. However, training such a model from scratch requires lots of GPU and dataset. Using LoRA to fine-tune the SD model may be a more practical and cost-effective approach.\n\n4) One-stage vs Two-stage: \nWe conducted ablation experiments on one-stage and two-stage training. Experiments show that due to the introduction of the epi module, the multi-view consistency of the two training methods is relatively good. Comparing the consistency is not enough to measure the difference between the two methods. The IS of the two-stage training method is lower, and the diversity is reduced to a certain extent, which is slightly worse than the one-stage training. However, it should be noted that the images after one-stage training will have ghosting, but the two-stage will not. We show the qualitative comparison results in the global response (the FID value of the two-stage method will also be higher). \n\n\n| Methods | IS↑ | FID↓\t| LPIPS↓| \tPSNR↑ | SSIM↑| \n|:-------:|:----:|:----:|:-------:|:-------:|:-------:|\n| One-stage |  4.52   |  82.92   | 0.0454   | 31.64  | 0.74 | \n| Two-stage | 3.13 | 74.08 | 0.0610 | 31.54 | 0.73 | \n\t\n5) Missing citation of Cap3D: First of all, the objects of text annotation are different. Cap3D annotates objects, while we annotate scenes. We first annotate the perspective view of the cube map. When integrating different perspective views, the instructions input to LLM are specially processed for the scene to prevent redundancy and other problems. We show the difference in text description before and after LLM integration in Figure 2 of this article. Our LLM text prompt is as follows:\n\nYou will be given six text prompts that describe the appearance of a room. Your task is to summarize these six given text prompts into a single text prompt that describes what the room looks like. This single summary text prompt in your response should strictly begin with <begin> and end with </end>. Simplify the room types if there are multiple room types in the prompts. The final text prompt should be less than 77 tokens and do not omit any furniture or any appearance details in the given prompts. \n\nAs suggested, we will cite Cap3D in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive suggestions and appreciate comments like 
“well-written, well-structured and easy to follow, clear" (R2LuU, R7HQE, RK8Nd), "first work, pioneer work, novel, technically sound, interesting" (R2LuU, R7HQE, RK8Nd, RYygs), "valuable, promote, enhance, useful, meaningful, facilitating" (R2LuU, R7HQE, RK8Nd, RYygs), "effectiveness, looks compact and practical to implement" (R2LuU, R7HQE, RYygs), "plausible results and visually pleasant" (R7HQE).

We hope after reading our comprehensive and detailed rebuttal, the reviewers can eliminate RK8Nd's misunderstanding about our work, change his mind, and give us a higher score. We will gladly incorporate all the feedback in the revised version. 

1) The main concerns of the reviewers are the generalization (R2LuU, RK8Nd, RYygs), diversity (RYygs), and scalability (RK8Nd) of DiffPano, the left-right consistency (R7HQE) and realism (RYygs) of the generated panoramas, and the realism (RYygs) of converting the generated panoramas into perspective images. We show some generated outdoor panoramas in the PDF to verify the generalization, generate diverse panoramas with the same text description to show the diversity, and generate longer panoramic videos with image-based panorama generation to demonstrate scalability (we recommend reviewers open the video with Adobe software to see the video). 

As for the left-right consistency of the generated panoramas, we propose to split the panorama into two parts and use a matching algorithm to calculate the matching pairs for verification. We don't adopt the Left-Right Consistency Error (LRCE) suggested by R7HQE, for this metric is suited in the text-to-panorama generation. Instead, we propose a Left-Right Matching Pairs (LRMP) metric to calculate the left-right consistency using matching algorithms. As suggested by R7HQE, we compare cylinder padding without data augmentation and the data augmentation method used in this paper to improve the left-right consistency. Experiments show that our simple data augmentation scheme can obtain a higher CLIP Score (CS) and Inception Score (IS), as well as lower Fréchet Inception Distance (FID). See the rebuttal to 7HQE.

We follow Text2Light to report FID and IS on panoramas to measure realism and diversity. Additionally, CLIP Score (CS) is used to evaluate the text-image consistency. While FID is widely used for image
generation, it relies on an Inception network trained on perspective images, thus less applicable for panorama images. 
We adopt a variant of FID customized for panorama, Fr´echet Auto-Encoder Distance (FAED) in PanFusion, 
which is used to better compare the realism. And the relevant results are as follows. Experiments show that our simple data augmentation scheme can obtain higher CLIP Score (CS) and Inception Score (IS), as well as lower Fréchet Inception Distance (FID) and FAED. See the rebuttal to 7HQE.

As suggested, we convert the generated panorama into a perspective view to further verify the realism. 
Experiments show that our method achieves the lowest FID, while our method is higher than MVDiffusion in IS and slightly lower than PanFusion. It should be noted that MVDiffusion directly generates perspective images and then stitches them into panoramas. It is not in the ERP format and does not have the top and bottom parts. Our panorama generation speed is faster.

We also conduct ablation experiments on one-stage and two-stage training. Experiments show that 
due to the introduction of the spherical epipolar-aware module, the multi-view consistency of the two training methods is relatively good. 
Comparing the consistency is not enough to measure the difference between the two methods. 
The IS of the two-stage training method is lower, and the diversity is reduced to a certain extent, 
which is slightly worse than the one-stage training. However, it should be noted that the images after one-stage training will have ghosting, but the two-stage will not. We show the qualitative comparison results (the FID value of the two-stage method will also be higher). See the rebuttal to Yygs.


All in all, the panoramas generated by our method have good realism and left-right consistency with faster speed. Our method can achieve scalable, diverse, and consistent panorama generation.

2) Most importantly, Reviewer K8Nd seems to have misunderstood the core content of this paper, 
the problem it solves, and the core innovation. Here, we point out the factual error. We aim to perform scalable multi-view panorama generation based on text (which can be expanded from one room to another). As far as we know, we are the first to propose text-to-multi-view panorama generation. Previous methods are limited to single-image panorama generation. The core reason is the lack of large-scale multi-view panorama datasets, and single panorama generation work cannot guarantee the multi-view consistency of the generated panorama. A new framework suitable for multi-view panorama generation is needed. To this end, we first construct a large-scale panoramic video-text dataset. Then, we design a multi-view panorama generation framework. First, we propose a diffusion model for text-to-single-view panorama generation (fine-tuned on the newly proposed panorama dataset using LoRA). To achieve multi-view consistency, we derive a spherical epipolar constraint suitable for panoramas and embed it into the network as an attention layer to realize the spherical epipolar multi-view panorama generation framework, which can achieve diverse and consistent scalable panorama generation results.

We hope that all reviewers can have in-depth discussions on K8Nd's misunderstandings to ensure that reviewer K8Nd can eliminate the factual errors and re-evaluate our work to obtain a fair review. We sincerely thank you.

3) We will polish the writing and proofread the revised version, including the typos and adding the necessary citations.

Meta-review: Reviewers are overall positive about the paper and leaning towards acceptance. The area chair also supports the decision. Nonetheless, there are many/various concerns on the submission. Please read the comments/concerns carefully and prepare the camera ready.
Decision: Accept (poster)