SlowGuess/NeurIPS_2023_OCR huggingface dataset:
leveraging a frozen pre-trained stable diffusion, demonstrates a robust generalization ability with a small amount of training data, as shown in Figure 4.
- Compared with inpainting method [11, 18]: Inpainting methods also exhibit worse performance due to the error accumulations, as evidenced by the gradual style change throughout the generated image sequence in Figure 3.
- Compared with Stable Diffusion (perspective)[36]: We also evaluate the original stable diffusion on perspective images of the same Matterport3D testing set. This method cannot generate multi-view images but is a good reference for performance comparison. The results suggest that our method does not incur performance drops when adapting SD for multi-view image generation.
Generate images in the wild. Despite our model being trained solely on indoor scene data, it demonstrates a remarkable capability to generalize across various domains. This broad applicability is maintained due to the original capabilities of stable diffusion, which are preserved by freezing the stable diffusion weights. As exemplified in Figure 4, we stick the perspective images into a
panorama and show that our MVDiffusion model can successfully generate outdoor panoramas. Further examples illustrating the model's ability to generate diverse scenes, including those it was not explicitly trained on, can be found in the supplementary materials.
Image&text-conditioned panorama generation. In Figure 5, we show one example of image&text-conditioned panorama generation. MVDiffson demonstrates the extraordinary capability of extrapolating the whole scene based on one perspective image.
Prompt: "A kitchen with white cabinets and white appliances. A kitchen with a white stove top oven next to a sink. A cutting board on the counter."
Figure 6: Qualitative evaluation for depth-to-image generation. More results are available in the supplementary material.
5.2 Multi view depth-to-image generation
This task converts a sequence of depth images into a sequence of RGB images while preserving the underlying geometry and maintaining multiview consistency. ScanNet is an indoor RGB-D video dataset comprising over 1513 training scenes and 100 testing scenes, all with known camera parameters. We train our model on the training scenes and evaluate it on the testing
scenes. In order to construct our training and testing sequences, we initially select key frames, ensuring that each consecutive pair of key frames has an overlap of approximately  $65\%$ . Each training sample consists of 12 sequential keyframes. For evaluation purposes, we conduct two sets of experiments. For our quantitative evaluation, we have carefully chosen 590 non-overlapping image sequences from the testing set, each composed of 12 individual images. In terms of qualitative assessment, we first employ the generation model to produce all the key frames within a given test sequence. Following this, the image&text-conditioned generation model is utilized to enrich or densify these images. Notably, even though our model has been trained using a frame length of 12, it has the capability to be generalized to accommodate any number of frames. Ultimately, we fuse the RGBD sequence into a cohesive scene mesh.
Baselines. To our knowledge, no direct baselines exist for scene-level depth-to-image generation. Some generate 3D textures for object meshes, but often require complete object geometry to optimize the generated texture from many angles [5, 29]. This is unsuitable for our setting where geometry is provided for the parts visible in the input images. Therefore, we have selected two baselines:
Method	FID ↓	IS ↑	CS ↑
RePaint [27]	70.05	7.15	26.98
ControlNet [52]	43.67	7.23	28.14
Ours	23.10	7.27	29.03
Table 2: Comparison in Fréchet Inception Distance (FID), Inception Score (IS), and CLIP Score (CS) for multiview depth-to-image generation.
testing sequences, we initially select key frames, times has an overlap of approximately  $65\%$ . Each times. For evaluation purposes, we conduct two sets in, we have carefully chosen 590 non-overlapping closed of 12 individual images. In terms of qualitative model to produce all the key frames within a given testditioned generation model is utilized to enrich or model has been trained using a frame length of 12, it date any number of frames. Ultimately, we fuse the
Task → Method	Panorama	Depth-to-image
PSNR ↑	Ratio ↑	PSNR ↑	Ratio ↑
G.T.	37.7	1.00	21.41	1.00
SD (Perspective)	10.6	0.28	11.20	0.44
MVDiffusion (Ours)	25.4	0.67	17.41	0.76
Table 3: Multi-view consistency for panorama generation and multi-view depth-to-image generation.
- RePaint[27]: This method uses an image diffusion model for inpainting holes in an image, where we employ the depth2image model from Stable Diffusion v2[36]. For each frame, we warp the generated images to the current frame and employ the RePaint technique [27] to fill in the holes. This baseline model is also fine-tuned on our training set.
- Depth-conditioned ControlNet: We train a depth-conditioned ControlNet model combined with the inpainting Stable Diffusion method with the same training set as ours. The implementation is based on a public codebase [7]. This model is capable of image inpainting conditioned on depths. We use the same pipeline as the above method.
Results. Table 2, Figure 6, Figure 7, and Figure 8 present the quantitative and qualitative results of our generation models. Our approach achieves a notably better FID, as shown in Table 2. As depicted in Figure 6, the repaint method generates progressively blurry images, while ControlNet produces nonsensical content when the motion is large. These issues arise since both methods depend on partial results and local context during inpainting, causing error propagation. Our method overcomes these challenges by enforcing global constraints with the correspondence-aware attention and generating images simultaneously. Figure 7 exhibits the keyframes at the left and the right, where the intermediate frames are generated in the middle, which are consistent throughout the sequence. Figure 8 illustrates the textured scene meshes produced by our method.
Key frame
Figure 7: Visualization of image&text-conditioned generated frames. We interpolate 4 frames (second image to fifth image). More visualization results are presented in supplementary materials.
on results
Key frame
Figure 8: Mesh visualization. MVDiffusion first generates RGB images given depths/poses and then fuse them into mesh with TSDF fusion.
5.3 Measuring multi-view consistency
The multi-view consistency is evaluated with the metric as explained earlier. For panoramic image generation, we select image pairs with a rotation angle of 45 degrees and resize them to  $1024 \times 1024$ . For multi-view depth-to-image generation, consecutive image pairs are used and resized to  $192 \times 256$ . PSNR is computed among all pixels within overlapping regions for panoramic image generation. For
multi-view depth-to-image generation, a depth check discards pixels with depth errors above  $0.5m$ , the PSNR is then computed on the remaining overlapping pixels.
Results. In Table 3, we first use the real images to set up the upper limit, yielding a PSNR ratio of 1.0. We then evaluate our generation model without the correspondence attention (i.e., an original stable diffusion model), effectively acting as the lower limit. Our method, presented in the last row, achieves a PSNR ratio of 0.67 and 0.76 for the two tasks respectively, confirming an improved multi-view consistency.
This paper introduces MVDiffusion, an innovative approach that simultaneously generates consistent multi-view images. Our principal novelty is the integration of a correspondence-aware attention (CAA) mechanism, which ensures cross-view consistency by recognizing pixel-to-pixel correspondences. This mechanism is incorporated into each UNet block of stable diffusion. By using a frozen pretrained stable diffusion model, extensive experiments show that MVDiffusion achieves state-of-the-art performance in panoramic image generation and multi-view depth-to-image generation, effectively mitigating the issue of accumulation error of previous approaches. Furthermore, our high-level idea has the potential to be extended to other generative tasks like video prediction or 3D object generation, opening up new avenues for the content generation of more complex and large-scale scenes.
Limitations. The primary limitation of MVDiffusion lies in its computational time and resource requirements. Despite using advanced samplers, our models need at least 50 steps to generate high-quality images, which is a common bottleneck of all DM-based generation approaches. Additionally, the memory-intensive nature of MVDiffusion, resulting from the parallel denoising limits its scalability. This constraint poses challenges for its application in more complex applications that require a large number of images (e.g., long virtual tour).
Broader impact. MVDiffusion enables the generation of detailed environments for video games, virtual reality experiences, and movie scenes directly from written scripts, vastly speeding up production and reducing costs. However, like all techniques for generating high-quality content, our method might be used to produce disinformation.
Acknowledgements. This research is partially supported by NSERC Discovery Grants with Accelerator Supplements and DND/NSERC Discovery Grant Supplement, NSERC Alliance Grants, and John R. Evans Leaders Fund (JELF). We thank the Digital Research Alliance of Canada and BC DRI Group for providing computational resources.
[1] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08113, 2, 2023.
[2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563-22575, 2023.
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
[4] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158, 2017.
[5] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11396, 2023.
[6] Zhaoxi Chen, Guangcong Wang, and Ziwei Liu. Text2light: Zero-shot text-driven hdr panorama generation. ACM Transactions on Graphics (TOG), 41(6):1-16, 2022.
[7] Mikolaj Czerkawski. Controlnetinpaint. https://github.com/mikonvergence/ControlNetInpaint, 2023.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Appendix: MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusion
The appendix provides 1) the full architecture specification of correspondence attention; 2) the implementation details of the MVDiffusion system; and 3) additional experimental results in the same format as the figures in the main paper.
A Network Architecture of correspondence-aware attention block
The architecture of correspondence-aware attention block is similar to vision transformers [9], with the inclusion of zero convolutions as suggested in ControlNet [52] and GELU [13] activation function.  $C$ ,  $H$ ,  $W$  are channel numbers, height and width respectively.
B Implementation details of MVDiffusion
B.1 Panorama image generation
Training and inference details. The generation model in our approach is built upon Stable-diffusion-v2 [47]. We train the model on perspective images with a resolution of  $512 \times 512$  for 10 epochs. The training is performed using the AdamW optimizer with a batch size of 4 and a learning rate of  $2e^{-4}$ , utilizing four A6000 GPUs. During inference, we utilize the DDIM sampler with a step size of
50 to perform parallel denoising of the eight generated images. Additionally, we employ blip2 [23] to generate texts for each perspective image, and during both training and inference, we use the corresponding prompts.
Figure 9: The architecture of the correspondence-aware attention block.
B.2 Implementation details of baselines
We introduce implementation details of baseline in the following.
- Text2Light [6] We combine the prompts of each perspective image and use the released pretrained model to generate the panorama.
- Stable Diffusion (panorama)[36] We fine-tuned Stable Diffusion using the panorama images within our training dataset, which contains 9820 panorama images at resolution  $512 \times 1024$ . We fine-tuned the UNet layer of the Stable diffusion while keeping VAE layers frozen. We use AdamW optimizer with a learning rate of  $1e^{-6}$  and batch size is 4, utilizing four A6000 GPUs.
- Inpainting methods [11, 18] In our approach, we employ Stable-diffusion-v2 [47] to generate the first image in the sequence based on the corresponding text prompt. For each subsequent image in the sequence, we utilize image warping to align the previous image with the current image. The warped image is then passed through Stable-diffusion-inpaint [48] to fill in the missing regions and generate the final image.
- Stable diffusion (perspective) In our approach, we utilize the model trained in the first stage of the generation module to generate the perspective images. During testing, each perspective image is associated with its own text prompt.
B.3 Geometry conditioned image generation
Training and inference details. Our generation model is derived from the stable-diffusion-2-depth framework [46]. In the initial phase, we fine-tune the model on all the perspective images of ScanNet dataset at a resolution of  $192 \times 256$  for 10 epochs. This training process employs the AdamW optimizer [25] with a learning rate of  $1e^{-5}$  and a batch size of 256, utilizing four A6000 GPUs. In the second stage, we introduce the correspondence-aware attention block and extra resizing convolutional layers for image conditioned generation. The training data consists of two categories: 1) generation training data, selected from 12 consecutive frames with overlaps of around 0.65, and 2) interpolation training data, derived from randomly chosen pairs of consecutive key frames and ten intermediate
frames. During each training epoch, we maintain a careful balance by randomly sampling data from both these categories in a 1:1 ratio. The correspondence-aware attention blocks and additional convolution layers are subsequently trained for 20 epochs, with a batch size of 4 and a learning rate of  $1e^{-4}$ , using the same four A6000 GPUs. During the inference stage, we employ the DDIM [41] sampler with a step size of 50 to perform parallel denoising on eight images.
B.4 Implementation details of baselines
We introduce the implementation details of baselines in the following.
- RePaint[27]: In our method, we utilize depth-conditioned Stable-diffusion-v2 [47] to generate the first image in the sequence. For each subsequent image, we condition it on the previous image by applying latent warping. This helps align the generated image with the previous one. To complete the remaining areas of the image, we employ the Repaint technique [27] for inpainting.
- Depth-conditioned ControlNet: We use the same method to generate the first image as the above method. Next, we warp the generated images to the current frame and use Stable-inpainting model [48] to fill the hole. To incorporate depth information into the inpainting model, we utilize a method from a public codebase [7], which adds the feature from depth-conditioned ControlNet [52] into each UNet layer of the inpainting model. For more detailed information, please refer to their code repository. In order to reduce the domain gap, the Stable-inpainting model has been fine-tuned on our training dataset. Similar to other fine-tuning procedures, we only fine-tuned UNet layers while keeping VAE part fixed. The fine-tuning was conducted on a machine with four A6000 GPUs. The batch size is 4 and the learning rate is  $1e^{-6}$ . We used AdamW as the optimizer. During inference, we utilize the DDIM sampler with a step size of 50 for generating images.
B.5 Visualization results
Figures 10-17 present supplementary results for panorama generation. In these figures, we showcase the output panorama images generated by both Stable diffusion (panorama) and Text2light methods. To compare the consistency between the left and right borders, we apply a rotation to the border regions, bringing them towards the center of the images. These additional visualizations provide further insights into the quality and alignment of the generated panorama images.
Figures 18-19 show additional results for image- & text-conditioned panorama generation. MVDiffusion extrapolates the whole scene based on one provided perspective image and text description.
Figure 20 provides additional results for generalization to out-of-training distribution data. Our model is only trained on MP3D indoor data while showing a robust generalization ability (e.g. outdoor scene).
Figures 21-26 show additional results with two baseline methods (depth-conditioned ControlNet [52] and Repaint [27]).
Figure 27 shows additional results of interpolated frames. The keyframes are at the left and the right, the middle frames are generated by applying our Interpolation module (see Sec. 4.2 in the main paper). The consistency is maintained throughout the whole sequence.
A bedroom with a large bed and sliding glass doors. An open door to a patio with an ocean view. A room with a chair and a lamp.
SD(Pano)
Text2light
Figure 10: Addition results for panorama generation
A room with white cabinets and a wooden floor. A walk-in closet with lots of shelves. An empty room with a closet and shelves.
Inpainting
SD(Pano)
Text2light
Figure 11: Addition results for panorama generation
A living room filled with furniture, a grand piano, a fire place, a painting, a large window. A living room with a couch and a ceiling fan.
Text2light
Figure 12: Addition results for panorama generation
A grand piano sitting in a living room next to a window. A living room filled with furniture and a fire place. A black piano sitting in a living room next to a window.
Text2light
Figure 13: Addition results for panorama generation
A dining room with a chandelier a table and chairs. A living room with white walls and wood floors. A room with a mirror on the wall.
Text2light
Figure 14: Addition results for panorama generation
A bedroom with a bed and a mirror and a window. A vase of flowers on a shelf in a room. A hallway with two framed pictures on the wall.
Text2light
Figure 15: Addition results for panorama generation
A living room filled with furniture and a large mirror. A table with a lamp on it. A living room filled with furniture and a chandelier.
Text2light
Figure 16: Addition results for panorama generation
A large kitchen with a center island and white canbinets. A dining room with a table and chairs. A view of a pool through a glass door. A room with a lot of windows.
SD(Pano)
Text2light
Figure 17: Addition results for panorama generation
A living room with a white couch and wooden floors. A living room filled with furniture, a ceiling fan and a flat screen TV.
A bathroom with a bathtub and a window. A large bathroom with a wooden floor and two sinks and a large mirror.
A potted plant in front of a window. A living room filled with furniture and a piano. A view of a patio through an open door.
A hallway heading to a living room. A hallway with a chandelier and white walls. A large white cabinet with morrors on it.
A room with a large window and a painting on the wall. A dining room with a table, chairs and a chandelier. A view of a kitchen with white canbinets and wood floors.
Figure 18: Addition results for dual-conditioned generation
A bedroom with a bed and a window. A bedroom with a couch and a bed. A vase of flowers sitting next to a window on top of a table.
A bathroom with a mirror and a vase of flowers. A bathroom with a toilet and a shower stall.
A room with a sink and a mirror. A couple of beds in a room with a clock on the wall. A hotel room with a bed and a desk.
A theater room with a row of recliners. A hallway leading to a movie theater room. A large empty room with a projector screen in the corner.
A house with a fire pit in front of it. A patio with a fire pit in the middle of it. A large house with a pool in front of it. A swimming pool with a waterfall and a rock wall.
Figure 19: Addition results for dual-conditioned generation
Bursting with vibrant hues and exaggerated proportions, the cartoon-styled room sparkled with whimsy and cheer, with floating shelves crammed with oddly shaped trinkets, a comically oversized polka-dot armchair perched near a gravity-defying, tilted lamp, and the candy-striped wallpaper creating a playful backdrop to the merry chaos, exuding a sense of fun and boundless imagination.
Bathed in the soft, dappled light of the setting sun, the silent street lay undisturbed, revealing the grandeur of its cobblestone texture, the rusted lampposts bearing witness to forgotten stories, and the ancient, ivy-clad houses standing stoically, their shuttered windows and weather-beaten doors speaking volumes about their passage through time.
Awash with the soothing hues of an array of blossoms, the tranquil garden was a symphony of life and color, where the soft murmur of the babbling brook intertwined with the whispering willows, and the iridescent petals danced in the gentle breeze, creating an enchanting sanctuary of beauty and serenity.
This ancient castle stands majestically on a rugged hill, with towering turrets, thick stone walls, and a formidable drawbridge over a time-worn moat. Inside, dimly lit hallways lead to grand chambers adorned with ornate carvings, all resonating with the echoes of a bygone era.
Figure 20: Additional results for generalization to out-of-training distribution data.
A white refrigerator freezer sitting next to a window. A trash can next to a refrigerator in a kitchen.
A living room filled with furniture and a piano. A living room with a couch, chair and pictures on the wall.
Figure 21: Addition results for depth-to-image generation.
A desk with a computer, keyboard, mouse and a teddy bear. A green stool is next to the desk.
Figure 22: Addition results for depth-to-image generation.
A trash can sitting next to a radiator in a bathroom. A bathroom with a toilet and a roll of toilet paper.
A bedroom with a bunk bed and a desk. A bunk bed with a desk underneath it.
Figure 23: Addition results for depth-to-image generation.
A flat screen TV sitting on top of a tv stand. A room with a standing speaker and a TV.
A laptop computer sitting on top of a desk next to a bed. a pair of shoes sitting on the floor next to a bed.
Figure 24: Addition results for depth-to-image generation.
A kitchen with a sink and a refrigerator. A kitchen with wooden cabinets and a stainless steel sink.
A brown couch sitting in a living room next to a window. A bookshelf filled with lots of books next to a window.
Figure 25: Addition results for depth-to-image generation.
A bedroom with a bed, desk and chair. A desk with a computer and a chair in a room. A desk with a chair and a lamp on it.
A desk with a chair and a computer monitor in a room. Two trash cans are on the floor.
Figure 26: Addition results for depth-to-image generation.
Figure 27: Addition results for interpolated frames.