MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion
Authors: Shitao Tang
Fuyang Zhang
Jiacheng Chen
Peng Wang
Yasutaka Furukawa
Venue: NeurIPS 2023 spotlight (2023)
Keywords: multiview; image generation; generative model; diffusion models
TL;DR: This paper introduces MVDiffusion, a simple yet effective multi-view image generation method, tailored for scenarios where pixel-to-pixel correspondences are available, such as perspective crops from panorama or multi-view images given depth/pose.

Abstract: This paper introduces MVDiffusion, a simple yet effective method for generating consistent multi-view images from text prompts given pixel-to-pixel correspondences (e.g., perspective crops from a panorama or multi-view images given depth maps and poses). Unlike prior methods that rely on iterative image warping and inpainting, MVDiffusion simultaneously generates all images with a global awareness, effectively addressing the prevalent error accumulation issue. At its core, MVDiffusion processes perspective images in parallel with a pre-trained text-to-image diffusion model, while integrating novel correspondence-aware attention layers to facilitate cross-view interactions. For panorama generation, while only trained with 10k panoramas, MVDiffusion is able to generate high-resolution photorealistic images for arbitrary texts or extrapolate one perspective image to a 360-degree view. For multi-view depth-to-image generation, MVDiffusion demonstrates state-of-the-art performance for texturing a scene mesh. The project page is at https://mvdiffusion.github.io/.

MVDiffusion: Enabling Holistic Multi-view Image
Generation with Correspondence-Aware Diffusion
Shitao Tang!*
Fuyang Zhang Jiacheng Chen' Simon Fraser University
Peng Wang² BBytedance
Yasutaka Furukawa'
"This kitcfan is charming bland of rustic and modern, fraturing a farge reclatmen wood island with marble counterton, a sink survounded by cabtnots, stainless-stod refrigerator stands tall. the right of sink, bulle-in worden cubinets patrited muted."
Consistent Multi-view Images
Glosed-Loop Panoramic Image
"A living room with muftinfe couchiss and coffee table. A weoden broé shaff fulled with lots of books next door. A white refrigerator sitting next to weeden benchi"
Depth sequence
Prajectory
Mesh w/o texture
Consistent multi-view
Mesh w/ texture
images conditioned on depths
Figure I:
MVDiffusion synthesizes consistent multi-view images. Top: generating perspective crops which can be stitched into panorama; Bottom: generating coherent multi-view images from depths.
Abstract
This paper introduces MVDiffusion, simple yet effective method for generating consistent multi-view images from text prompts given pixel-to-pixed correspon- dences (e.g.. perspective crops from panorama multi-view images given depth maps and poses). Unlike prior methods that rely on iterative image warping and inpainting, MVDiffusion simultaneously generates all images with global aware- ness, effectively addressing the prevalent еттоr accumulation issue. At its core, MVDiffusion processes perspective images in parallel with a pre-trained text-to- image diffusion model, while integrating novel correspondence-awarc attention layers to facilitate cross-view interactions. For panorama generation, while only trained with 10k panoramas, MVDiffusion is able to generate high-resolution photorealistic images for arbitrary texts or extrapolate one perspective image a 360-degree view. For multi-vied vielw strates state-of-the-art performance for texturing a scene mesh. The project page at htst:::://mviiffusiiin .wwooon..iti github.ionn..
generation, MVDiffusion demon-
*Equal contribution. Contact the authors at shitaot @ sfu.ca
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Photorealistic image synthesis aims to generate highly realistic images, enabling broad applications in virtual reality, augmented reality, video games, and filmmaking. The field has seen significant advancements in recent years, driven by the rapid development of deep learning techniques such as diffusion- generative models
45
One particularly successful domain is text-to text-to-image generation. Effective approaches include gener- ative adversarial networks [3
12 19]. autoregressive transformers [T] [5] 49], and more recently, diffusion models [15 34 37 DALL-E 34,, Imagen [37 and others generate photorealistic images with large-scale diffusion models. Latent diffusion models [6) apply the diffusion process in the latent space, allowing more efficient computations and faster image synthesis.
Despite impressive progress, multi-view text-to-image synthesis still confronts issues of compu- tational efficiency and consistency across views. common approach involves an autoregressive generation process [S8) where the generation of the n--hh image is conditioned on the (n - 1)-th image through image warping and inpainting techniques. However, this autoregressive approach results in accumulated errors and does not handle loop closure [II]. Moreover, the reliance on the previous image may pose challenges for complex scenarios or large viewpoint variations.
Our approach, dubbed MVDiffusion, generates multi-view images simultaneously. using multiple branches of a standard text-to-image model pre-trained on perspective images. Concretely, we use a stable diffusion (SD) model [36] and add "correspondence-avee aware attention" (CAA) mechanism between the UNet blocks, which facilitates cross-view interactions and learns to enforce multi-view consistency. When training the CAA blocks, we freeze all the original SD weights to preserve the generalization capability the pre-trained model.
In summary, the paper presents MVDiffusion, a multi-view text-to-image generation architecture that requires minimal changes to a standard pretrained text-to-image diffusion model, achieving state-of-the-art performance on two multi-view image generation tasks. For generating panorama, MVDiffusion synthesizes high-resolution photorealistic panoramic images given arbitrary per-view texts, or extrapolates one perspective image to full 360-degree view. Impressively, despite being trained solely on realistic indoor panorama dataset, MVDiffusion possesses the capability to create diverse panoramas, e.g. outdoor or cartoon style. For multi-view image generation conditioned on depths/poses, MVDiffusion demonstrates state-of-the-art performance for texturing a scene mesh.
2 Related Work
Diffusion models. Diffusion models [1, 42
(DM) or score-based generative models are the essential theoretical framework of the exploding generative AI. Early works achieve su- perior sample quality and density estimation [8] 45 but require a long sampling trajectory. Advanced sampling techniques (20
45
26 40 accelerate the process while maintaining generation quality. Latent diffusion models (36) (LDMs) apply DM in a compressed latent space to reduce the computational cost and memory footprint, making the synthesis of nigh-resolution images feasible consumer devices. We enable holistic multi-view image generation by the latent diffusion model.
Image generation. Diffusion Models (DM) dominate content generation. Foundational work such as DALL-E 34, GLIDE 30, LDMs [36). and Imagen (37 have showcased significant capabilities in text-conditioned image generation. They train on extensive datasets and leverage the power of pre-trained language models. These large text-to-image Diffusion Models also establish strong foun- dations for fine-tuning towards domain-specific content generation. For instance, MultiDiffusion (I] and DiffCollage (53) failitates 360-degree image generation. However, the resulting images are not true panoramas since they do not incorporate camera projection models. Text2Light (6) synthesizes HDR panorama images from text using a multi-stage auto-regressive generative model. However, the leftmost and rightmost contents are not connected (i.e., loop closing).
3D content generation. Content generation technology has profound implications in VR/AR and entertainment industries, driving research to extend cutting-edge generation techniques from a single image to multiple images. Dreamfusion 31 and Magic3D [24 distill pre-trained Diffusion Models into a NeRF model (28) to generate 3D objects guided by text prompts. However, these works focus on objects rather than scenes. In the quest for scene generation, another approach [18] generates

---

Stable Diflusion Net
U-Net
F
F
FEN
U-Neet Block
UpwDown Sample
Correspondeace
Aware Allention
Figure 2: MVDiffusion generates multi- images in parallel through the weight-sharing multi- branch UNet. To enforce multi-view consistency, the Correspondence- Aware Attention (CAA) block is inserted after each UNet block. "FFN" is acronym for "Feed-Forward Network". The rightmost figure elaborates on the mechanisms CAA.
prompt-conditioned images indoor spaces by iteratively querying pre-trained text-to-image Diffusion Model. SceneScape (II) generates novel views zoom-out trajectories by employing image warping and inpainting techniques using diffusion models. Text2Room [18] adopts similar methods to generate complete textured 3D room geometry. However, the generation of the n-th image is solely conditioned on the local context, resulting in accumulation errors and less favorable results. Our research takes holistic approach and generates consistent multi-view images given camera poses and text prompts while fine-tuning pre-trained perspective-image Diffusion Models.
Preliminary
Latent Diffusion Models (LDM) 33 is the foundation of our method. LDM consists of three components: a variational autoencoder (VAE) 22 with encoder & and decoder D, a denoising network eg, and condition encoder Tg.
High-resolution images x E #xx33 are mapped to a low-dimensional latent space by Z Ex).. where Z IRAt xxwe, The down-sampling factor H// W/w is set in the popular Stable Diffusion (SD). The latents are converted back to the image space by x = D(Z).
The LDM training objective given
LLDM = (g(x).y. [||e cg( (,(
where l is uniformly sampled from 1 to T and Z; is the noisy latent at time step The denoising network 6e is dime-conditional UNet [8], augmented with cross-attention mechanisms to incorporate the optional condition encoding TT(y), y could be text-prompt, an image, any other user-specified condition.
At sampling time, the denoising (reverse) process generates samples in the latent space, and the
decoder produces high-resolution images with single forward pass. Advanced samplers 20 26 can further accelerate the sampling process.
MVDiffusion: Holistic Multi-view Image Generation
MVDiffusion generates multiple images simultaneously by running multiple copies/branches of stable diffusion model with a novel inter-branch "correspondence-avwee aware attention" (CAA) mechanism to facilitate multi-view consistency. Figure 2 presents an overview of multi-branch UNet and
aware
CAA designs. The system is applicable when pixel-to-pixel correspondences are available between images, specifically for cases of 1) Generating panorama or extrapolating a perspective image to panorama. The panorama consists of perspective images sharing the camera center where

---

pixel-to-pixel correspondences are obtained by planar tomography and 2) Texture mapping of a given geometry where multiple images of arbitrary camera poses establish pixel-to-pixt correspondences through depth- unprojection and projection. We first introduce panorama generation ($4.1). which employs generation modules, and then multi-view depth-to-imen to--ttgo-imeeeeeee generation ($4.2). which employs generation and interpolation modules. Since the interpolation module does not contain CAA blocks, I will also cover the design of the CAA block and explain how it is inserted into the multi-branch UNet.
4.1 Panorama generation
In MVDiffusion, a panorama is realized by generating eight perspective views, each possessing a horizontal field of view of 90" with 45 overlap. To achieve this, generate eight 512 512 images by the generation module using frozen pretrained stable diffusion model
Generation module. The proposed module generates eight 512 512 images. It accomplishes this through a process of simultaneous denoising. This process involves feeding each noisy latent into shared UNet architecture. dubbed as the multi-branch UNet, predict noises concurrently. In order to ensure multi-view consistency, a correspondence-awalt attention (CAA) block is introduced following each UNet block. The CAA block follows the final ResNet blocks and is responsible for taking in multi-view features and fusing them together.
Correspondence-wware attention (CAA). The CAA block operates N feature maps concurrently, as shown in Figure 2 For the i-th source feature map, denoted as F, it performs cross-attention with the remaining (N target feature maps, represented as F!.
For a token located at position (s) in the source feature map, we compute a message based on the corresponding pixels {t') } in the target feature maps {F') (not necessarily at integer coordinates) with local neighborhoods. Concretely, for each target pixel t', we consider a K x K neighborhood (t') by adding integer displacements (dz/d,,) the (x/y) coordinate, where |d, K/2 and (dyl < K/2. In practice, we use K 3 with neighborhood of 9 points.
M Σ Σ
SoftMax (()) (W)F((D) WVF((!)) (C.)
(2)
(EN(())
F(s) = F(s) + 7(0), F'(()==''( 7(5 s).
(3)
The message M calculation follows the standard attention mechanism that aggregates information from the target feature pixels {t.} to the source (s). Wq. WK, and Wv are the query, key and value matrices. The key difference is the added position encoding 7-)) to the target feature F' based on the 2D displacement between its corresponding location s in the source image and The displacement provides the relative location in the local neighborhood. Note that a displacement is 2D vector, and we apply a standard frequency encoding [50] to the displacement in both x and y coordinates, then concatenate. A target feature F' (t') is not at an integer location and is obtained by bilinear interpolation. To retain the inherent capabilities of the stable diffusion model [47] we initialize the final linear layer of the transformer block and the final convolutional layer of the residual block to be zero, as suggested in ControlNet (52). This initialization strategy ensures that our modifications do not disrupt the original functionality of the stable diffusion model.
Panorama extraporlation. The goal is to generate full 360-degree panoramic views (seven target images) based on a single perspective image (one condition image) and the per-view text prompts. We use SD's 's impainting model [48] as the base model as it takes one condition image. Similar to the generation model, CAA blocks with zero initializations inserted into the UNet and trained on datasets.
For the generation process, the model reinitializes the latents of both the target and condition images with noises from standard Gaussians. In the UNet branch of the condition image, we concatenate a mask of ones to the image (4 channels in total). This concatenated image then serves as the input to the inpainting model, which ensures that the content of the condition image remains the same. On the contrary, in the UNet branch for target image, we concatenate a black image (pixel values of zero) with mask of zeros to serve as the input, thus requiring the inpainting model to generate completely new image based on the text condition and the correspondences with the condition image.

---

Training, We insert CAA block into the pretrained stable diffusion Unet 47 or stable diffusion impainting Unet 48 to ensure multi-view consistency. The pretrained network frozen while we use the following loss to train the CAA block:
N Σ
T MMVTiiiussonn
E (2)=0(x))] (e0=,,))
I|e' cb([Z))
( (y)
4.2 Multiview depth-to-image generation
The multiview depth-to-image task aims to generate multi-view images given depths/poses. Such images establish pixel-to-pixel correspondences through depth-based unprojection and projection. MVDiffusion's process starts with the generation module producing key images, which are then densified by the interpolation module for more detailed representation.
Generation module. The generation module for multi-view depth-to-image generation is similar to the one for panorama generation. The module generates set of 192 256 images. We use depth-conditioned stable diffusion model [46] as the base generation module and simultaneously generate multi-view images through multi-branch UNet. The CAA blocks are adopted to ensure multi-view consistency.
Interpolation module. The interpolation module of MVDiffusion, inspired by VideoLDM (2]. creates N images between pair of key frames', which have been previously generated by the generation module. This model utilizes the same UNet structure and correspondence attention weights as the generation model, with extra convolutional layers, and it reinitializes the latent of both the in-between images and key images using Gaussian noise. A distinet feature of this module is that the UNet branch of key images is conditioned on images already generated. Specifically, this condition is incorporated into every UNet block. In the UNet branch of key images, the generated images are concatenated with a mask of ones (4 channels), and then zero convolution operation is used to downsample the image to the corresponding feature map size. These downsampled conditions are subsequently added to the input of the UNet blocks. For the branch of in-between images, take a different approach. We append a black image, with pixel values of zero, to a mask of zeros, and apply the same zero convolution operation to downsample the image match the corresponding feature map size. These downsampled conditions are also added to the input of the UNet blocks. This procedure essentially trains the module such that when the mask is one, the branch regenerates the conditioned images, and when the mask is zero, the branch generates the in-between images.
Training. we adopt a two-stage training process. In the first stage, we fine-tune the SD UNet model using all ScanNet data. This stage is single-view training (Eq. without the CAA blocks. In the second stage, we integrate the CAA blocks, and the image condition blocks into the UNet, and only these added parameters are trained. We use the same loss as panorama generation to train the model.
5 Experiments
We evaluate MVDiffusion on two tasks: panoramic image generation and multi-view depth-to-image generation. We first describe implementation details and the evaluation metrics.
Implementation details. We have implemented the system with Py while using publicly available Stable Diffusion codes from Diffusers [51]. The model consists of denoising UNet to execute the denoising process within a compressed latent space and a VAE to connect the image and latent spaces. The pre-trained VAE of the Stable Diffusion is maintained with official weights and is used to encode images during the training phase and decode the latent codes into images during the inference phase. We have used machine with 4 NVIDIA RTX A6000 GPUs for training and inference. Specifie details and results of these varying configurations are provided the corresponding sections.
Evaluation matries. The evaluation metrics cover two aspects, image quality of generated images and their consistency.
Image quality is measured by Fréchet Inception Distance (FID) 14
Inception Score (IS) 38 and
CLIP Score (CS) (32]. FID measures the distribution similarity between the features of the generated
5

---

Prompt: "a living room with large glass table, white chairs and a giant window. A TV is attached to the wall and a fancy crystal chandelier is hanging on the ceiling."
Inpainting
Text2light
SD (Pano)
Ours
Figure 3: Qualitative evaluation for panorama generation. The red box indicates the stitched
leftmost and rightmost content. More results are available in the supplementary material.
and real images. The Inception Score based on the diversity and predictabilityy of generated images. CLIP Score measures the text-image similarity using pretrained CLIP models 33
Multi-view consistency is measured by the metric based pixel-level similarity. The area of multi-view image generation is still in an early stage, and there is no common metric for multi-view consistency. We propose a new metric based on Peak Signal-to-Noiss Ratio (PSNR). Concretely, given multi-view images, we compute the PSNR between all the overlapping regions and then compare this "overlapping PSNR" for ground truth images and generated images. The final score is defined as the ratio of the "overlapping PSNR" of generated images to that of ground truth images. Higher values indicate better consistency.
The rest of the section explains the experimental settings and results more, while the full details are referred to the supplementary.
5.1 Panoramic image generation
This task generates perspective crops covering the panoramic field of view, where the challenge is to ensure consistency in the overlapping regions. Matterport3D 4 is comprehensive indoor scene dataset that consists of 90 buildings with 10.912 panoramic images. We allocate 9820 and 1092 panoramas for training and evaluation, respectively.
Baselines. We have selected three related state-of-ther oh-the-art art methods for thorough comparisons. The details of the baselines are briefly summarized as follows (full implementation details can be found in the appendix):
Tex/2Ligh/ creates HDR panoramic images from text using a multi-stage auto-regressive genera- tive model. To obtain homographic images, project the generated panoramas into perspective images using the same camera settings (FoV=902, rotation=45)).
Stable Diffusion (36] is text-to-image model capable of generating high-quality perspective images from text. For comparison, we fine-tuned Stable Diffusion using panoramic images and then extracted perspective images similar manner.
Inpainting methods [11 I8 operate through an iterative process, warping generated images to the current image and using an inpainting model fill in the unknown area. Specifically, we employed the inpainting model from Stable Diffusion v2 36) for this purpose.
Results. Table I and Figure present the quantitative and qualitative evaluations, respectively. We then discuss the comparison between MVDiffusion and each baseline:

---

Prompt: "Majestically rising towards the heavens, the snow-capped mountain stood, its jagged peaks cloaked in shroud of ethereal clouds, its rugged slopes stark contrast against the serene azure sky, and its silent gran- deur exuding air of ancient wisdom and timeless solitude."
Figure 4: Example of panorama generation of outdoor scene. More results are available in the supplementary material.
Prompt: large kitchen with center island white canbinets. dining room with tables and chairs. room with a lot of windows and a wooden floor."
Figure 5: Imaggiotext-oonditiond panorama generation results. More results available in the supplementary material.
Compared with Te Text2Light is based on auto-regressive transformers and shows low FID, primarily because diffusion models perform generally better. Another drawback is the inconsistency between the left and the right panorama borders, as illustrated in Figure 3
Compared with Stable Diffusion (panorama [36] MVDiffusion obtain higher IS, CS, and FID than SD (pano). Like Text2light, this model also en- counters an issue with inconsistency at the left and right borders. Our approach addresses this problem by enforcing explicit consistency through correspondence-awint attention, resulting in seam- less panoramas. Another shortcoming of this model is its requirement for substantial data to reach robust generalization. In contrast, our model, leveraging frozen pre-traines pre-trained stable diffusion, demonstrates robust generalization ability with small amount of training data, as shown in Figure +
Method
FID +
IS
CS 1
Impainting I
18
42.13
7.08
29.05
Text2light 6
48.71
5,41
25.98
SD (Pano) 36
23.02
6.58
28.63
SD (Perspective)
36
25.59
7.29
30.25
MVDiffusiin(OOrs))
21.44
7.32
30.04
Table 1: Quantitative evaluation with Fréchet Inception Distance (FID), Inception Score (IS),
and CLIP Score (CS).
Compared with inpainting method [1] 18 Inpainting methods also exhibit worse performance
due to the error accumulations, as evidenced by the gradual style change throughout the generated image sequence in Figure 3
Compared with Stable Diffusion (perspective) [36]: We also evaluate the original stable diffusion on perspective images of the same Matterport3D testing set. This method cannot generate multi-view images but is good reference for performance comparison. The results suggest that our method does not incur performance drops when adapting SD for multi-view image generation.
Generate images the wild. Despite our model being trained solely on indoor scene data, demonstrates remarkable capability to generalize across various domains. This broad applicability maintained due to the original capabilities of stable diffusion, which are preserved by freezing the stable diffusion weights. As exemplified in Figure stich the perspective images into

---

panorama and show that our MVDiffusion model can successfully generate outdoor panoramas. Further examples illustrating the model'ss ability to generate diverse scenes, including those it was not explicitly trained on, can be found in the supplementary materials.
we show one example of image&text- conditioned panorama generation. MVDiffsion demonstrates the extraordinary capability of extrapo- lating whole scene based on one perspective image.
Amage&text-cooniiiooe panorama generation. Figure 5
Prompt: "A kitchen with white cabinets and white appliances. A kitchen with a white stove top oven next a sink. A cutting board the counter."
Inpux Bepihs
GT
Ourx
ControlNet
Repaint
Figure 6: Qualitative evaluation for depth-to-image generation. More results are available in supplementary material.
5.2 Multi view Hepth-to-image generation
Method
FID 4
IS 1
CS
This task converts sequence of depth images into a sequence RGB images while preserving the underlying geometry and maintaining multi- view consistency. ScanNet is an indoor RGB- D video dataset comprising over 1513 training scenes and 100 testing scenes, all with known camera parameters. We train our model the training scenes and evaluate on the testing scenes. In order to construct ensuring that each consecutive pair of key frames has an overlap approximately 65%. Each training sample consists of 12 sequential keyframes. For evaluation purposes, we conduct of experiments. For our quantitative evaluation, we have carefully chosen 590 non-overlapping image sequences from the testing set, each composed of 12 individual images. In terms of qualitative assessment, we first employ the generation model to produce all the key frames within a given test sequence. Following this, the image&&text-condiioone densify these images. Notably, even though our model has been trained using a frame length of 12, has the capability to be generalized accommodate any number of frames. Ultimately, fuse the RGBD sequence into cohesive scene mesh.
RePaint (27
70.05
7.15
26.98
ControlNet 52
43.67
7.23
28.14
Ours
23.10
7.27
29.03
Table 2: Comparison in Fréchet Inception Distance (FID), Inception Score (IS), and CLIP Score (CS) for multiview depth-to-image generation.
training and testing sequences, initially select key frames,
sets
generation model is utilized to enrich or
Baselines. To our knowledge, no direct baselines exist for scene-level depth-to-image generation. Some generate 3D textures for object meshes, but often require complete object geometry optimize the generated texture from many angles 29]. This is unsuitable for our setting where geometry provided for the parts visible in the input images. Therefore, we have selected two baselines:

---

Task
Panorama
Depth-to hepth-to-image to-image
Method
PSNR 1
Ratio
PSNR t
Ratio t
G.T.
37.7
1.00
21.41
1.00
SD (Perspective)
10.6
0.28
11.20
0.44
MVDiffusion (Ours)
25.4
0.67
17.41
0.76
Table 3: Multi-view consistency for panorama generation and multi-view depth-to-image generation.
RePaint(27] This method uses image diffusion model for inpainting holes in an image, where employ the depth2image model from Stable Diffusion v2[36]. For each frame, we warp the generated images the current frame and employ the RePaint technique 2 fill in the holes. This baseline model is also fine-tuned on our training set.
Depth ceptii-condt ControlNer: We train depth-conditioned ControlNet model combined with the inpainting Stable Diffusion method with the same training set as ours. The implementation is based on public codehase 77. This model is capable of image inpainting conditioned depths. We use the same pipeline as the above method.
Results. Table 2 Figure 6 Figure 7 and Figure 8 present the quantitative and qualitative results of our generation models. Our approach achieves notably better FID, shown in Table 2 As depicted in Figure 6 the repaint method generates progressively blurry images, while ControlNet produces nonsensical content when the motion large. These issues arise since both methods depend on partial results and local context during inpainting, causing error propagation. Our method overcomes these challenges by enforcing global constraints with the correspondence-awari attention and generating images simultaneously. Figure the keyframes at the left and the right, where the intermediate frames are generated in the middle, which are consistent throughout the sequence. Figure iillustrate the textured scene meshes produced by our method.
Figure Visualization Interpolation results
Key frame
Key frame
generated frames. We interpolate frames (second image to fifth image). More visualization results presented in supplementary materials.
Figure 8: Mesh visualization. MVDiffusion first generates RGB images given depths/poses and then fuse them into mesh with TSDF fusion.
5.3 Mcasuring multi-view consistency
The multi-view consistency is evaluated with the metric as explained earlier. For panoramic image generation, we select image pairs with rotation angle of 45 degrees and resize them to 1024 x 1024. For multi-view depth-to-image generation, consecutive image pairs are used and resized to 192 256. PSNR is computed among all pixels within overlapping regions for panoramic image generation. For

---

mueti-view to-i generation, a depth check discards pixels with depth errors above 0.5m, the PSNR is then computed on the remaining overlapping pixels.
Results. In T we first use the real images to set up the upper limit, yielding a PSNR ratio of 1.0. We then evaluate our generation model without the correspondence attention (i.e., an original stable diffusion model), effectively acting as the lower limit. Our method, presented in the last row, achieves PSNR ratio of 0.67 and 0.76 for the two tasks respectively, confirming an improved multi-view consistency.
Conclusion
This paper introduces MVDiffusion, an innovative approach that simultaneously generates consistent multi-view images. Our principal novelty is the integration of a correspondence attention (CAA) mechanism, which ensures cross-view consistency by recognizing tixel-to-pxeel correspon- dences. This mechanism is incorporated into each UNet block of stable diffusion. By using a frozen pretrained stable diffusion model, extensive experiments show that MVDiffusion achieves state- of-the-art performance in panoramic image generation and multi-view depth-to-iman generation, effectively mitigating the issue of accumulation error of previous approaches. Furthermore, our high-level idea has the potential to be extended to other generative tasks like vidco prediction or 3D object generation, opening up new avenues for the content generation of more complex and large-scale scenes.
Limitations. The primary limitation of MVDiffusion lies in its computational time and resource requirements. Despite using advanced samplers, models need at least 50 steps to generate high-quality images, which is a common bottleneck of all DM-based generation approaches. Addi- tionally, the memory-intensivy nature of MVDiffusion, resulting from the parallel denoising limits its scalability. This constraint poses challenges for its application in more complex applications that require a large number of images (e.g., long virtual tour).
Broader impact. MVDiffusion enables the generation of detailed environments for video games, virtual reality experiences, and movie scenes directly from written scripts, vastly speeding up production and reducing costs. However, like all techniques for generating high-quality content, method might be used to produce disinformation.
Acknowledgements. This research is partially supported by NSERC Discovery Grants with Accel- erator Supplements and DND/NSERC Discovery Grant Supplement, NSERC Alliance Grants, and John R. Evans Leaders Fund (JELF). We thank the Digital Research Alliance of Canada and BC DRI Group for providing computational resources.
References
[1] Omer Bar-Tal Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint 2302.0000 2,2,, 2, 2023.
[2] Andreas Blatmannn Robin Rombach, Huan Ling, Tim Dockhorn, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-rcsolution video synthesis with latent diffusion models. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 11096, 2018.
[4] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: /709.66 06199 2017.
[5] Dave Zhenyu Chen, Yawar Siddiqui, Hsin- Hsin-Ying Lee, Sergey Tulyakov, and Mathias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. /3396, 2023.
[6] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2light: Zero-shot text-driven hdr panorama generation.
ACM Transactions Graphics (TOG), 41(6):1-11, 2022.
[7] Mikolaj
Czerkawski.
Controlnetinpaint
https///github..comm/mmovvereer
Contro1MetInpaind 2023.
[8] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780 87944, 2021.
10

---

[9] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition scale. arXiv preprint arXiv:2010.1
16x16 words:
2020.
[10] Patrick Esser, Rohin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[1]] Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent seene generation. arXiv preprint arXiv:2302.011 133, 2023.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1)): 139-144, 2020.
[13] Dan Hendryeks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv: 1eeo.10060..0 1606.0841. 2016.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598 2022.
[16] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[17] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-33, 2022.
[18] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. arXiv preprint arXiv:2303. 11999, 2023.
[19] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances Neural Information Processing Systems, 34: 852-863 2021.
[20] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. arXiv preprint arXiv:2206. 00364, 2022.
[21] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206. 00364, 2022.
[22] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-inen pre- traitrannng for unitied vision- languae understanding and generation. In International Conference on Machine Learning, pages 1 12900. PMLR, 2022.
[24] Chen-Hsuunn Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung-Yi Vi Lin. Magic3d: High-resolution text-to-3d content creation. arXiv preprint arXiv:2211. 1.22211..044 10440, 2022.
[25] Tlya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. arXiv preprint
1rXiv: 2017.
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.
[27] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Patiern Recognition, pages 11461-11471. 2022.
[28] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. In ECCV, 2020.
[29] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating
textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers, pages 1-8, 2022.
[30] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffuusin models. arXiv preprint arXiv:2 10741, 2021.
[31] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv: 14988, 2022.
11

---

[32] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Míshknn, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMI.R, 2021.
[34] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 2612,,,, 2022.
[35] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[36] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffussin models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[37] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[38] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[39] Jascha bick-Dickt,,in, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning. PMLR, 2015.
[40] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025000 2020.
[41] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint vrXiv:2010.02500 2020.
[42] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems (NeurIPS), 2019.
[43] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[44] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in nearal information processing systems, 33:12438-12445 2020.
[45] Yang Song, Jascha Narain Sohl-Dicksee Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR, 2021.
[46] StabilityAI.
Stable-dine diffusion-2-deethh.
http:r//aaggaagttae...o/ssssa//
stable diffusion- depth 2023.
[47]
StabilityAI. Stable-diffusionn22. https:/ http:://wgggggiie.....//aaa////s 2023.
[48]
StabilityAI. Ccable-diffusil iww.iiiiinnnnninnnn.....m 2023.
dabble-difussion-imaall
https: https:///uggghngacco..o/oux
[49] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation lcarning. Advances in neural information processing systems, 30, 2017.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[51] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan I Lambert, Kashif Rasul, Mishig Davaadorj. and Thomas Wolf. Diffusers: State-of-the-art diffusion models. htttps:/giithbb com/
huggingface/ dwwe/serfffssseee/ee//e/eeaeasss 2022.
[52] Lvmin Zhang and Maneesh Agrawala, Adding conditional control to text-to-image diffuusio models. arXiv preprint Xiv: 055444 2023.
[53] Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming Yu Liu. Diffcollage: Parallel generation of large content with diffusion models. arXiv preprint TtV:2303.110766 2023.
12

---

Appendix:
MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Awant Diffusion
The appendix provides 1) the full architecture specification of correspondence attention; 2) the implementation details of the MVDiffusion system; and 3) additional experimental results in the same format as the figures in the main paper.
A Network Architecture of correspondence-aware attention block
The architecture of correspondence-awaaae attention block similar to vision transformers 19]. with the inclusion of zero convolutions as suggested in ControlNet [2 and GELU [13] activation function. C, H, W are channel numbers, height and width respectively.
F(s)
F(())
Correspondence Attn
Zero Conv
CKIDw, 3xd
B Implementation details of MVDiffusion
Feed Forward Network
B.1 Panorama image generation
Zero Conv
C4TEWW 3xd
Training and inference details. The generation model in ap- proach built upon Stable-diffusion--2 [47]. We train the model on perspective images with resolution of 512 x 512 for 10 epochs. The training is performed using the AdamW optimizer with a batch size of 4 and learning rate of 2e-1, utilizing four A6000 GPUs. During inference, we utilize the DDIM sampler with a step size of 50 perform parallel denoising of the eight generated images. Additionally, we employ blip2 [23 to generate texts for each perspective image, and during both training and inference, we use the corresponding prompts.
Figure 9: The architecture of the correspondence at- tention block.
B.2 Implementation details of baselines
We introduce implementation details of baseline the following.
Tex12Light 6 We combine the prompts of each perspective image and use the released pretrained
model to generate the panorama.
We fine-tuned Stable Diffusion using the panorama images within our training dataset, which contains 9820 panorama images at resolusion 512 x 1024. We fine-tuned
Stable Diffusion (panorama)
the UNet layer of the Stable diffusion while keeping VAE layers frozen. We use AdamW optimizer
with a learning rate of 1e and batch size is 4, utilizing four A6000 GPUs.
employ Stable-diffusion-V [47 generate the first image in the sequence based on the corresponding text prompt. For each subsequent image in utilize image warping to align the previous image with the current image. The warped image is then passed through Stable-diffusion-nnaai generate the final image.
Inpainting methods [I] 18 In our approach,
the sequence,
48 to fill the missing regions and
Stable diffusion (perspective) In our approach, we utilize the model trained in the first stage of the generation module generate the perspective images. During testing, each perspective image is associated with its own text prompt.
B.3 Geometry conditioned image generation
Training and inference details. Our generation model derived from the stable-diffusion-2-eeae framework [46]. In the initial phase, we fine-tune the model on all the perspective images of ScanNet dataset at a resolution of 192 x 256 for 10 epochs. This training process employs the AdamW optimizer 25 with learning rate of le and a batch size of 256, utilizing four A6000 GPUs. In the second stage, we introduce the correspondence-wware attention block and extra resizing convolutional layers for image conditioned generation. The training data consists of two categories: 1) generation training data, selected from 12 consecutive frames with overlaps around 0.65, and 2) interpolation training data, derived from randomly chosen pairs of consecutive key frames and ten intermediate
13

---

frames. During each training epoch, we maintain a careful balance by randomly sampling data from both these categories in 1:1 ratio. The correspondence-awatt attention blocks and additional convolution layers are subsequently trained for 20 epochs, with batch size of 4 and learning rate of le-1, using the same four A6000 GPUs. During the inference stage, we employ the DDIM [4] sampler with step size of 50 to perform parallel denoising on eight images.
B.4 Implementation details of baselines
We introduce the implementation details of baselines the following.
[ePaint[2]]: In our method, we utilize depth-conditioned Stable-diffusion-22 47 to generate the first image in the sequence. For each subsequent image, condition it on the previous image by applying latent warping. This helps align the generated image with the previous one. To complete the remaining areas of the image, employ the Repaint technique
27 for inpainting.
Depth-conditioned ControlNet: We use the same method to generate the first image as the above method. Next, we warp the generated images to the current frame and use Stable-inpainting model 488 to fill the hole. To incorporate depth information into the inpainting model, we utilize method from public codebase 17), which adds the feature from copph-conditioned ControlNet [52] into each UNet layer of the inpainting model. For more detailed information, please refer to their code repository. In order to reduce the domain gap, the Stable-ing inbbbliiiiinniggg model has been fine-tuned on our training dataset. Similar to other fine-tuning procedures, we only fine-tuned UNet layers while keeping VAE part fixed. The fine-tuning was conducted on machine with four A6000 GPUs. The batch size is and the learning rate is le 1e-6. We used Adam W as the optimizer. During inference, we utilize the DDIM sampler with step size of 50 for generating images.
B.5
Visualization results
present supplementary results for panorama generation. In these figures, we showcase the output panorama images generated by both Stable diffusion (panorama) and Text2light methods. To compare the consistency between the left and right borders, we apply a rotation to the border regions, bringing them towards the center of the images. These additional visualizations provide further insights into the quality and alignment of the generated panorama images.
Figures 10 17
Figures 18 19 show additional results for image- & text-conditioned panorama generation. MVDiffu- sion extrapolates the whole scene based on one provided perspective image and text description.
Figure 20 provides additional results for generalization to out-of-trainngg distribution data. Our model only trained on MP3D indoor data while showing robust generalization ability (e.g. outdoor scene).
Figures 21 26 show additional results with two baseline methods (depth-conditioned ControlNet 52
and Repaint 27
Figure shows additional results of interpolated frames. The keyframes are at the left and the right, the middle frames are generated by applying our Interpolation module (see Sec. 4.2 the main paper). The consistency is maintained throughout the whole sequence.
14

---

A bedroom with a large bed and sliding glass doors. An open door to a patio with an ocean view. A room with a chair and a lamp.
Ours
Inpainting
SD
Text2light
Figure 10: Addition results for panorama generation
15

---

A room with white cabinets and a wooden floor. A walk-in closet with lots of shelves. An empty room with a closet and shelves.
Ours
Inpainting
SD
Text2light
Figure 11: Addition results for panorama generation
16

---

A living room filled with furniture, grand piano, fire place, a painting, , a large window. A living room with a couch and a ceiling fan.
Ours
Inpainting
SD(aa
Text2light
Figure 12: Addition results for panorama generation
17

---

A grand piano sitting in a living room next to window. A living room filled with furniture and a fire place. A black piano sitting in a living room next to a window.
Ours
Inpainting
SD
Text2light
Figure 13: Addition results for panorama generation
18

---

A dining room with a chandelier a table and chairs. A living room with white walls and wood floors. A room with a mirror on the wall.
Ours
Inpainting
SD
Text2light
Figure 14: Addition results for panorama generation
19

---

A bedroom with bed and mirror and window. A vase of flowers on shelf in room. A hallway with two framed pictures on the wall.
Ours
Inpainting
SD
Text2light
Figure 15: Addition results for panorama generation
20

---

A living room filled with furniture and a large mirror. A table with a lamp on it. A living room filled with furniture and a chandelier.
Ours
Inpainting
SD
Text2light
Figure 16: Addition results for panorama generation
21

---

A large kitchen with a center island and white canbinets. A dining room with table
and chairs. A view of a pool through a glass door. A room with a lot of windows.
Ours
Inpainting
SD
Text2light
Figure 17: Addition results for panorama generation
22

---

A living room with white couvh and wooden floors. A living room filled with furni- ture, ceiling fan and a flat screen TV.
A bathroom with a bathtub and window. A large bathroom with a wooden floor and two sinks and a large mirror.
A potted plant in front of window. A living room filled with furniture and a piano. A view of a patio through an open door.
A hallway heading living room. A hallway with chandelier and white walls. A large white cabinet with morrors on it.
A room with a large window and a painting on the wall. A dining room with table, chairs and a chandelier. A view of a kitchen with white canbinets and wood floors.
Figure 18: Addition results for dual-conditioned generation
23

---

A bedroom with a bed and a window. A bedroom with a couch and a bed. A vase of flowers sitting next to a window on top of table.
A bathroom with a mirror and a vase of flowers. A bathroom with toilet and a shower stall.
A room with a sink and a mirror. A couple of beds in room with clock on the wall.
A hotel room with bed and desk.
A theater room with row of recliners. A hallway leading to a movie theater room. A large empty room with projector screen in the corner.
A house with a fire pit in front of it. A patio with a fire pit in the middle it A large house with a pool in front of it. A swimming pool with waterfall and rock wall.
Figure 19: Addition results for dual-conditioned generation
24

---

Bursting with vibrant hues exaggerated proportions, the cartoon-styled room sparkled with whimsy and cheer, with floating shelves crammed with oddly shaped trinkets, a comically oversized polka-dor nolka-dot armchair perched near gravity-defying tilted lamp, and the striped wallpaper creating playful backdrop to the merry chaos, exuding of fun and boundless imagination.
Bathed in the soft, dappled light of setting sun, the silent street lay undisturbed, revealing the gran- deur of its cobblestone texture, the rusted lampposts bearing witness to forgotten stories, and the ancient, ivy-clad houses standing stoically, their shuttered windows and weather-beaten doors speaking volumes about their passage through time.
Awash with the soothing hues of an array of blossoms, the tranquil garden symphony of life and color, where the soft murmur of the babbling brook intertwined with the whispering willows, and iridescent petals danced in the gentle breeze, creating an enchanting sanctuary of beauty and serenity.
This ancient castle stands majestically on rugged hill, with towering turrets, thick stone walls, and formidable drawbridge over time-worn moat. Inside, dimly lit hallways lead to grand chambers adorned with ornate carvings, all resonating with the echoes of bygone
Figure 20: Additional results for generalization to tri---nn distribution data.
25

---

A white refrigerator freezer sitting next to window. A trash can next to refrigerator in kitchen.
Imput depth
GT
Ours
ControlNet
Repaint
A living room filled with furniture and a piano. A living room with couch, chair and pictures on the wall.
Imput depth
GT
ControlNet
Repaint
Figure 21: Addition results for generation.
26

---

A desk with a computer, keyboard, mouse and teddy bear. A green stool is next to the desk.
Imput depth
GT
Ours
ControlNet
Repaint
A desk with computer and a computer monitor and keyboard on it. A computer desk with magazine on top of
Imput depth
GT
Ours
ControlNet
Repaint
Figure 22: Addition results for depth-to-imenn to-inage generation.
27

---

A trash can sitting next to a radiator in bathroom. A bathroom with a toilet and a roll of toilet paper.
Imput depth
GT
Ours
ControlNet
Repaint
A bedroom with bunk bed and desk. A bunk bed with a desk under- neath it.
Imput depth
GT
ControlNet
Repaint
Figure 23: Addition results for depth-to-iman generation.
28

---

A flat screen TV sitting on top of tv stand. A room with standing speaker and TV.
Imput depth
GT
Ours
ControlNet
Repaint
A laptop computer sitting on top of desk next to a bed. pair of shoes sitting on the floor next to a bed.
Imput depth
GT
ControlNet
NAL
Repaint
Figure 24: Addition results for depth-to-iman generation.
29

---

A kitchen with a sink and refrigerator. A kitchen with wooden cabi- nets and stainless steel sink.
Imput depth
GT
Ours
ControlNet
Repaint
A brown couch sitting in a living room next to a window. A bookshelf
filled with lots of books next to a window.
Imput depth
GT
Ours
ControlNet
Repaint
Figure 25: Addition results for depth-to- soppt--to-image generation.
30

---

A bedroom with bed, desk and chair. A desk with computer and a chair in a room. A desk with chair and a lamp on it.
Imput depth
GT
Ours
ControlNet
Repaint
A desk with chair and a computer monitor in room. Two trash cans are on the floor.
Imput depth
GT
Ours
ControlNet
Repaint
Figure 26: Addition results for depth-to-image generation.
31

---

Key frame
Interpolation results
Key frame
AITY
Key frame
Interpolation results
Key Irame
Figure 27: Addition results for interpolated frames.
32


Reviews:
[{"review_id": "6DrcrZMNO9", "reviewer": "Reviewer_2y4s", "summary": "The paper proposes a multiview latent diffusion model that is aware of the correspondence between views.  Equipped with correspondence-aware attention blocks, the proposed generation module, interpolation module and the super-resolution module help MVDiffusion outperforms existing works.", "questions": "Please refer to weakness.", "limitations": "Please refer to weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed Correspondence-Aware Attention block connects different views. Also, keyframes are first generated and then upsampled both spatially and temporally.", "weaknesses": "My biggest concern of the paper is the evaluation of measuring multi-view consistency. On L217 the authors mentioned to use pixel-level similarity for multi-view consistency. Is the PSNR computed between generated image and ground truth image? Does that mean the generation of depth-to-image is deterministic? It might be a good idea to conduct some evaluation in 3D. For example, train a NeRF with generated images and evaluate the difference between rendered image and the MVDiffusion generated image. Another way might be calculating some reprojection error since we know the camera poses of the generated images.\nAlso, for panorama generation, the author only compares with Stable Diffusion. Would be nice if DiffCollage or MultiDiffusion is also considered for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 2y4s for the constructive suggestions and feedback. We answer the questions as follows:\n\n---\n**1. Is the PSNR computed between generated image and ground truth image?**\n\nNo. The PSNR is calculated between two consecutive generated images in their overlapping regions, L219 in the main paper, since the generated image is different from ground truth image (not deterministic).\n\n---\n**2. It might be a good idea to conduct some evaluation in 3D. For example, train a NeRF with generated images and evaluate the difference between rendered image and the MVDiffusion generated image.**\n \nWe conducted evaluations in 3D using TSDF fusion to integrate the depths and images into a mesh. For experiment details, please refer to the *GR1 of the general response* for new qualitative results.\n\n---\n**3. Would be nice if DiffCollage or MultiDiffusion is also considered for comparison?**\n\nPlease see the *GR2 of the general response* for a qualitative comparison with MultiDiffusion or DiffCollage.\n"}, {"review_id": "x1IoXX0Zqp", "reviewer": "Reviewer_x9w4", "summary": "This paper presents MVDiffusion, a new diffusion model to generate consistent multi-view images, e.g., panorama. The authors propose a novel correspondence-aware attention mechanism in order to enforce pixel-level correspondence and cross-view consistency. More specifically this mechanism is used in three modules: a generation module that generates consistent low-resolution multi-view images; an interpolation module that generates images in between; a super-resolution module. After injecting such module into Stable Diffusion Unet layers and fine-tuning the model on multi-view image dataset, the model can synthesize consistent multi-view images based on text or depth.", "questions": "It would be great if the authors could respond the points mentioned above. Thanks!", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main idea (improving multi-view consistency through a correspondence-aware attention mechanism) is novel, simple, effective and easy to understand. The generated results are very impressive, see figure 1, 4, 5, 6 and all the images in the supplementary.\n2. The paper is well-written especially the method part where the main design is demonstrated clearly, see section 4.1 and figure 3. The authors also provide the code in the supplementary.\n3. The comparison with previous methods and some straightforward baselines are comprehensive and the improvements are convincing.", "weaknesses": "1. The pipeline figure (Figure 2) could be further improved: this figure is supposed to make the audience understand the whole workflow without looking at the method text, however, for now, in this figure, it's a bit unclear to me how these modules are associated with each other at first glance.\n2. Adding a brief section on failure cases would be great: the results in the main paper and supplementary are truly impressive, however, I'm also curious what are the common failure cases of MVDiffusion. Adding a short paragraph discussion this would be beneficial to the whole community.\n3. I feel a bit confused about the panorama generation during inference time: in Line 145, it says \"The generation module generates eight 256 × 256 images, and the super-resolution module upscales to eight 1024 × 1024 images\". I am just wondering where's the interpolation module then? Or let me put this way, how many images are generated by the generation module and how many are generated by the interpolation module? I assume the key frames are generated by the generation module and more images in between are generated by the interpolation module?\n4. How to handle the conflicts in the overlap region between two generated images? If I understand correctly, although correspondence attention works very well, it still cannot guarantee the overlap region between two generated images are *exactly* matched, so I'm just curious how does this mismatch is handled? I asked this because in the stitched 8 perspective images, it looks very consistent.\n5. Three modules are proposed in this paper, it would be great if an ablation study could be conducted such that the audience could understand the influence of each individual module better, e.g., how much difference it would make if removing the interpolation module?\n6. In addition to the stitch image, it would be great if showing more *densely* (much more than 8) generated multi-view images and make it an animation, such that the multi-view consistency could be better evaluated qualitatively by the audience.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer x9w4 for the valuable questions and suggestions. We address the comments in the following:\n\n---\n\n**W1:** Thanks for your suggestion. For the final version, we plan to enhance Figure 2. We will illustrate a pipeline flow to show that images are initially created by the generation module, followed by the interpolation module (multiview depth-to-image generation), and finally by the super-resolution module. Additionally, we will include a note indicating that the interpolation module does not participate in the panorama generation.\n\n---\n**W2:** Considering our model allows users to dictate distinct prompts for each perspective image, significant content change can emerge when these prompts undergo drastic alterations. Please refer to *GR4 of the general response* for examples of failure cases. We will add more examples in the final version of the paper. \n\n---\n**W3:** In our panorama generation process, we do not employ the interpolation module. The reason for this is that the 8 perspective images we use already encompass a full 360-degree view without any occlusions. Therefore, the use of interpolation to fill in gaps or unseen areas becomes unnecessary. Conversely, the interpolation module is vital in multiview depth-to-image generation, when dealing with a large number of multi-view images. In cases where the quantity of images exceeds the memory capacity of a single GPU, the interpolation module steps in to effectively manage the images. We will clarify this in the final version.\n\n---\n**W4:** In our approach, we do not explicitly address conflicts, as our objective is for the diffusion model to learn to generate perfectly consistent images. In panorama generation, we don't see obvious inconsistencies. However, inconsistencies do occur in the multiview depth-to-image generation. We believe these inconsistencies could arise due to two key factors. Firstly, the presence of noisy depth data can introduce errors into the generation process, leading to inconsistencies in the output images. Secondly, the employed latent space might not be high-resolution enough. This lack of granularity could result in the loss of pixel-level consistency, since the attention operation is solely conducted within the latent space. \n\n---\n**W5:** If we remove the interpolation module, we cannot generate images that cover the whole scene mesh, as presented in Figure 2 of the rebuttal file. As for the ablations on the super-resolution module, we are not able to present qualitative results in the single-page rebuttal PDF due to the space constraint, but we will include it in the final version.\n\n---\n**W6:** We have made the video, and the link has been sent to AC as guided by the emailed instructions.\n"}, {"review_id": "mRgni2vGok", "reviewer": "Reviewer_xhAT", "summary": "This paper introduces MVSDiffusion, a diffusion framework that generates multi-view images with content consistency. The problem setting is interesting and essential in practice. The proposed correspondence-aware attention mechanism provides cues for multi-view consistency.", "questions": "1. The presented panorama is blurry compared with SD (pano). Why?\n2. From my point of view, MVDiffusion is able to generate multi-view images and thus can synthesize images in 3D novel views instead of pure rotations (panorama). But all the presented image sequences are almost pure rotation. What's the problem?\n", "limitations": "1. The multi-view image generation relies on given geometries.\n2. Regions that are not captured by the pre-defined camera poses cannot be recovered.\n3. As the framework needs to interpolate camera poses between keyframes. The time consumption may be quite large.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel problem setting: generating images for multiple camera poses with consistent contents.\n2. The paper proposes a multi-view diffusion technique to obtain content consistency: extract features from other views guided by depth maps.\n3. Panorama generation can also be improved via the multi-view diffusion framework.", "weaknesses": "1. The MVDiffusion relies on given high-quality multi-view depth maps. However, the depth map obtained for leaves and strings may lack details, which may impact the generated image quality.\n2. The correspondences are derived from wrapped depth so that it can only support rigid static scenes.\n3. The interpolation module is time-consuming but the time consumption of each module is not provided. Please show them.\n4. The tailored super-resolution module is not novel.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the constructive feedback and suggestions. We reply to the questions/concerns in the following:\n\n---\n**W1.** While our experiments are currently with the ScanNet dataset, we believe that the most pragmatic usage is the creation of textures for handcrafted scene meshes. These meshes typically possess flawless geometries. This holds significant potential for real-world applications, particularly in design, visual effects, and entertainment industries.\n\n---\n**W2.** Yes. Right now, our method only applies to static scenes, but non-rigid scenes should be future work since we can still obtain correspondences through non-rigid transformation.\n\n---\n**W3.** We measure the run time on a single A6000 GPU. For panorama, we measure the run time by generating 8 perspective images and we don't use the interpolation module for panorama generation. For multiview depth-to-image generation, we measure the runtime by generating 12 images for generation and super-resolution module and the runtime by generating 4 interpolated images for interpolation modules.  The time consumption is as follows, which will be added to the paper:\n| Task           | Generation model | Interpolation model | Super-resolution model |\n|------------------|:----:|:-----:|:-----:|\n| Panorama |  67s |  / | 1120s  |\n| Multiview depth-to-image generation      |  128s |  104s |  1443s\n\n---\n**Q1.** There is still inconsistency between perspective images. When they are stitched together, the panorama becomes blurry. The reason is that we use a resolution of 256 * 256 for the generation module, and thus the latent space is in low resolution, and there exists pixel inconsistency. This could be mitigated by training the generation module with a higher resolution (512 * 512). We show one example in Figure 1 of the rebuttal file, which is much clearer. Please refer to the *GR4 of the general response* for updated results of panorama generation.\n\n---\n**Q2.** As we indicated in the abstract, our current method is restricted to the generation of multi-view images where one-to-one correspondences are readily available. Unfortunately, this condition isn't fulfilled in casual multi-view image generation scenarios. Therefore, developing methods that can accommodate casual multi-view image generation without the necessity of one-to-one correspondence is a valuable direction for future work.\n"}, {"review_id": "V60ky7xYoH", "reviewer": "Reviewer_VE1c", "summary": "This paper proposes a new method to generate multi-view images, ensuring pixel-to-pixel multi-view consistency.\nThe multi-view consistency is guaranteed by the correspondence aware module, which is utilized in the latent diffusion multi-image generation process.\nThe proposed method outperforms previous works for tasks related with multi-view image generation, especially in the multi-view consistency.", "questions": "1. How does the proposed method perform for objects? Could we use the proposed method to generate textures given object meshes?\n2. How does the authors deal with the depth occlusion and the situation where the pixels in target images are not in the view of the source image?", "limitations": "The authors addressed the limitations and the work does not have negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea to guarantee pixel correspondence in the diffusion process is novel and reasonable.\n2. The supplementary material provides codes, which show some details of the implementation.\n3. The proposed method may improve the results for text to texure given meshes.\n4. For different view images, the user can input different prompt.", "weaknesses": "1. To obtain the pixel correspondence for multi-view images, the proposed method requires to input the depth maps.\n2. The authors do not explain how to deal with the depth occlusion and how to deal with the situation where the pixels in target images are not in the view of the source image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer VE1c for constructive suggestions.\n\n---\n**W1:** Our current method is restricted to the generation of multi-view images where one-to-one correspondences are readily available. Unfortunately, this condition isn't fulfilled in casual multi-view image generation scenarios, and thus our method requires depth inputs. Developing methods that can accommodate casual multi-view image generation without the necessity of one-to-one correspondence should be an important direction for future work.\n\n---\n**W2:** We have enhanced the positional encoding (Figure 3 in the main paper) used in the correspondence-aware attention by incorporating depth differences between source and target views. Specifically, we project the depths of correspondences from the target views onto the source views and subsequently calculate the depth differences. Notably, in the case of occlusions, we do not apply any specific handling. We anticipate that the standard SD modules will interpolate or inpaint in the presence of occlusions by learning from the depth differences. For areas of occlusion where two images lack overlap, there's no requirement to ensure consistent outcomes. We updated the results in the global rebuttal's PDF file. Please refer to the *GR1 of the general response* for updated results on multiview depth-to-image generation.\n\n---\n**Q1:** We believe that MVDiffusion is able to generate textures given object meshes since our method can generate textures of a scene, as shown in the PDF file of the global response. Please refer to the *GR1 of the general response*.\n\n---\n**Q2:** Please see the *GR1 of the global response* for our answers.\n"}]

Author rebuttal:
We thank all reviewers and appreciate the constructive comments and the recognition of novelty, and we are grateful for all the positive initial ratings (two accept, one weak accept, one borderline accept). 

This general response provides updated figures and accompanying discussions to answer several key comments/questions from the reviewers. We also uploaded a video showing a rotating camera view, as requested by reviewer x9w4. The link to the video has been sent to AC, following the emailed instructions. We encourage every reviewer to watch this video. 

We first summarize the key comments/questions covered in this response in the following: 

---
### Summary of key questions/comments:

**S1.** Reviewer 2y4s asks for evaluating MVDiffusion in 3D.

**S2.** Reviewer 2y4s asks for a comparison with MultiDiffusion or DiffCollage.

**S3.** Reviewer x9w4 asks for failure cases of MViffusion.

**S4.** Reviewer xhAT asks why the presented panorama is blurry compared with SD (pano).

**S5.** Reviewer VE1c asks how MVDiffusion deals with the depth occlusion.

---

We will then address the comments/questions summarized above by referring to the new experimental results attached in the single-page PDF:

**GR1. (for S1 and S5) Updated results on multiview depth-to-image generation.**

In our rebuttal, Figure 2 showcases the ability of MVDiffusion to produce high-quality textures for scene meshes. The following modifications have been implemented: 1) We improved the positional encoding used in multiview depth-to-image generation, as depicted in Figure 3 of the main paper. This was achieved by integrating depth disparities between source and target views. To be precise, we project the depth of correspondences from the target views onto the source views and then compute the resulting depth disparities (handle occlusions), 2) During the training phase, every training sample incorporates 12 consecutive keyframes, 3) During testing, we first employ the generation module to produce all the key frames within a given test sequence. Then, the interpolation module is utilized to enrich or densify these images. Notably, even though our model has been trained using a frame length of 12, it has the capability to be generalized to accommodate any number of frames, and 4) Ultimately, we fuse the RGBD sequence into a cohesive scene mesh.

---
**GR2. (for S2) Comparison with MultiDiffusion or DiffCollage.**

As pointed out in Line 64 of the paper, neither MultiDiffusion nor DiffCollage incorporates a camera model when generating 360-degree views. This leads to outputs that do not accurately represent true panoramas. In our rebuttal's Figure 5, we present the results of MultiDiffusion and MVDiffusion using the same prompts. In a true panorama, lines often appear curved due to the distinct perspective shifts inherent in panoramic photography. However, in images produced by MultiDiffusion, these lines remain linear, retaining the characteristics of a conventional perspective image

---
**GR3. (for S4) Updated results of panorama generation**
We trained the correspondence-aware attention block within our panorama generation module at a resolution of 512 x 512 while keeping the original stable diffusion UNet weights frozen using the Matterport3D indoor datasets.  As shown in Figure 1, our method is able to generate outdoor panoramas. The result indicates: 1) Even when trained on a limited dataset, MVDiffusion exhibits strong generalization capabilities due to the stable diffusion pretrained model, and 2) The clarity of the generated panorama is notably enhanced—with fewer artifacts—when trained at higher resolutions.

---
**GR4. (for S3) Failure cases of panorama generation**

In Figure 3, we highlight a failure case of our panorama generation. Considering our model allows users to dictate distinct prompts for each perspective image, significant content change can emerge when these prompts undergo drastic alterations.

---
**GR5. At last, we update a new functionality of MVDiffusion, which supports extrapolating a perspective image to a full 360 view.**

We also demonstrate our method can extrapolate a perspective image into 360 views in Figure 4.  Specifically, we use the pretrained stable diffusion impainting model as the base generation model. In the Unet branch of the conditioned image, we append a mask of ones (in 4 channels) to the image. This concatenated image then serves as the input for the inpainting model, which ensures the content remains consistent. Conversely, in the Unet branch dealing with the generated images, we concatenate a black image (pixel values of zero) with a mask of zeros. This serves as the input, enabling the inpainting model to generate a completely new image based on the text it is provided.


Meta-review: This paper appears was unanimous accept by the reviewers. They all appreciated the novelty and thorough evaluation and felt the paper is a good contribution to the field.  The author rebuttal was reviewed and commented on by all with healthy discussion between the authors and the reviewers and the AC.  Congratulations!

In revising your paper for the camera-ready, please include and comments and clarifications that were made or proposed during the rebuttal and discussion phase.
Decision: Accept (spotlight)