SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shitao Tang $^{1*}$  Fuyang Zhang $^{1*}$  Jiacheng Chen $^{1}$  Peng Wang $^{2}$  Yasutaka Furukawa $^{1}$ $^{1}$ Simon Fraser University  $^{2}$ Bytedance
"This kitchen is a charming blend of rustic and modern, featuring a large reclaimed wood island with marble countertop, a sink surrounded by cabinets. A stainless-steel refrigerator stands tall. To the right of the sink, built-in wooden cabinets painted in a muted."
"A living room with multiple couches and a coffee table. A wooden book shelf filled with lots of books next to a door. A white refrigerator sitting next to a wooden bench."
Figure 1: MVDiffusion synthesizes consistent multi-view images. Top: generating perspective crops which can be stitched into panorama; Bottom: generating coherent multi-view images from depths.
This paper introduces MVDiffusion, a simple yet effective method for generating consistent multi-view images from text prompts given pixel-to-pixel correspondences (e.g., perspective crops from a panorama or multi-view images given depth maps and poses). Unlike prior methods that rely on iterative image warping and inpainting, MVDiffusion simultaneously generates all images with a global awareness, effectively addressing the prevalent error accumulation issue. At its core, MVDiffusion processes perspective images in parallel with a pre-trained text-to-image diffusion model, while integrating novel correspondence-aware attention layers to facilitate cross-view interactions. For panorama generation, while only trained with 10k panoramas, MVDiffusion is able to generate high-resolution photorealistic images for arbitrary texts or extrapolate one perspective image to a 360-degree view. For multi-view depth-to-image generation, MVDiffusion demonstrates state-of-the-art performance for texturing a scene mesh. The project page is at https://mvdiffusion.github.io/.
*Equal contribution. Contact the authors at shitaot@sfu.ca.
Photorealistic image synthesis aims to generate highly realistic images, enabling broad applications in virtual reality, augmented reality, video games, and filmmaking. The field has seen significant advancements in recent years, driven by the rapid development of deep learning techniques such as diffusion-based generative models [2, 16, 21, 39, 43, 44, 45].
One particularly successful domain is text-to-image generation. Effective approaches include generative adversarial networks [3, 12, 19], autoregressive transformers [10, 35, 49], and more recently, diffusion models [15, 17, 34, 37]. DALL-E 2 [34], Imagen [37] and others generate photorealistic images with large-scale diffusion models. Latent diffusion models [36] apply the diffusion process in the latent space, allowing more efficient computations and faster image synthesis.
Despite impressive progress, multi-view text-to-image synthesis still confronts issues of computational efficiency and consistency across views. A common approach involves an autoregressive generation process [5, 11, 18], where the generation of the  $n$ -th image is conditioned on the  $(n - 1)$ -th image through image warping and inpainting techniques. However, this autoregressive approach results in accumulated errors and does not handle loop closure [11]. Moreover, the reliance on the previous image may pose challenges for complex scenarios or large viewpoint variations.
Our approach, dubbed MVDiffusion, generates multi-view images simultaneously, using multiple branches of a standard text-to-image model pre-trained on perspective images. Concretely, we use a stable diffusion (SD) model [36] and add a "correspondence-aware attention" (CAA) mechanism between the UNet blocks, which facilitates cross-view interactions and learns to enforce multi-view consistency. When training the CAA blocks, we freeze all the original SD weights to preserve the generalization capability of the pre-trained model.
In summary, the paper presents MVDiffusion, a multi-view text-to-image generation architecture that requires minimal changes to a standard pretrained text-to-image diffusion model, achieving state-of-the-art performance on two multi-view image generation tasks. For generating panorama, MVDiffusion synthesizes high-resolution photorealistic panoramic images given arbitrary per-view texts, or extrapolates one perspective image to a full 360-degree view. Impressively, despite being trained solely on a realistic indoor panorama dataset, MVDiffusion possesses the capability to create diverse panoramas, e.g. outdoor or cartoon style. For multi-view image generation conditioned on depths/poses, MVDiffusion demonstrates state-of-the-art performance for texturing a scene mesh.
Diffusion models. Diffusion models [16, 21, 39, 42, 43, 44, 45] (DM) or score-based generative models are the essential theoretical framework of the exploding generative AI. Early works achieve superior sample quality and density estimation [8, 45] but require a long sampling trajectory. Advanced sampling techniques [20, 26, 40] accelerate the process while maintaining generation quality. Latent diffusion models [36] (LDMs) apply DM in a compressed latent space to reduce the computational cost and memory footprint, making the synthesis of high-resolution images feasible on consumer devices. We enable holistic multi-view image generation by the latent diffusion model.
Image generation. Diffusion Models (DM) dominate content generation. Foundational work such as DALL-E 2 [34], GLIDE [30], LDMs [36], and Imagen [37] have showcased significant capabilities in text-conditioned image generation. They train on extensive datasets and leverage the power of pre-trained language models. These large text-to-image Diffusion Models also establish strong foundations for fine-tuning towards domain-specific content generation. For instance, MultiDiffusion [1] and DiffCollage [53] failitates 360-degree image generation. However, the resulting images are not true panoramicas since they do not incorporate camera projection models. Text2Light [6] synthesizes HDR panorama images from text using a multi-stage auto-regressive generative model. However, the leftmost and rightmost contents are not connected (i.e., loop closing).
3D content generation. Content generation technology has profound implications in VR/AR and entertainment industries, driving research to extend cutting-edge generation techniques from a single image to multiple images. Dreamfusion [31] and Magic3D [24] distill pre-trained Diffusion Models into a NeRF model [28] to generate 3D objects guided by text prompts. However, these works focus on objects rather than scenes. In the quest for scene generation, another approach [18] generates
Figure 2: MVDiffusion generates multi-view images in parallel through the weight-sharing multibranch UNet. To enforce multi-view consistency, the Correspondence-Aware Attention (CAA) block is inserted after each UNet block. "FFN" is an acronym for "Feed-Forward Network". The rightmost figure elaborates on the mechanisms of CAA.
prompt-conditioned images of indoor spaces by iteratively querying a pre-trained text-to-image Diffusion Model. SceneScape [11] generates novel views on zoom-out trajectories by employing image warping and inpainting techniques using diffusion models. Text2Room [18] adopts similar methods to generate a complete textured 3D room geometry. However, the generation of the  $n$ -th image is solely conditioned on the local context, resulting in accumulation errors and less favorable results. Our research takes a holistic approach and generates consistent multi-view images given camera poses and text prompts while fine-tuning pre-trained perspective-image Diffusion Models.
Latent Diffusion Models (LDM) [36] is the foundation of our method. LDM consists of three components: a variational autoencoder (VAE) [22] with encoder  $\mathcal{E}$  and decoder  $\mathcal{D}$ , a denoising network  $\epsilon_{\theta}$ , and a condition encoder  $\tau_{\theta}$ .
High-resolution images  $\mathbf{x} \in \mathbb{R}^{H \times W \times 3}$  are mapped to a low-dimensional latent space by  $\mathbf{Z} = \mathcal{E}(\mathbf{x})$ , where  $\mathbf{Z} \in \mathbb{R}^{h \times w \times c}$ . The down-sampling factor  $f = H / h = W / w$  is set to 8 in the popular Stable Diffusion (SD). The latents are converted back to the image space by  $\tilde{\mathbf{x}} = \mathcal{D}(\mathbf{Z})$ .
The LDM training objective is given as
$$
L _ {L D M} := \mathbb {E} _ {\mathcal {E} (\mathbf {x}), \mathbf {y}, \epsilon \sim \mathcal {N} (0, 1), t} \left[ \| \epsilon - \epsilon_ {\theta} \left(\mathbf {Z} _ {t}, t, \tau_ {\theta} (\mathbf {y})\right) \| _ {2} ^ {2} \right], \tag {1}
$$
where  $t$  is uniformly sampled from 1 to  $T$  and  $\mathbf{Z}_t$  is the noisy latent at time step  $t$ . The denoising network  $\epsilon_{\theta}$  is a time-conditional UNet [8], augmented with cross-attention mechanisms to incorporate the optional condition encoding  $\tau_{\theta}(\mathbf{y})$ .  $\mathbf{y}$  could be a text-prompt, an image, or any other user-specified condition.
At sampling time, the denoising (reverse) process generates samples in the latent space, and the decoder produces high-resolution images with a single forward pass. Advanced samplers [20, 26, 40] can further accelerate the sampling process.
4 MVDiffusion: Holistic Multi-view Image Generation
MVDiffusion generates multiple images simultaneously by running multiple copies/branches of a stable diffusion model with a novel inter-branch "correspondence-aware attention" (CAA) mechanism to facilitate multi-view consistency. Figure 2 presents an overview of multi-branch UNet and the CAA designs. The system is applicable when pixel-to-pixel correspondences are available between images, specifically for cases of 1) Generating a panorama or extrapolating a perspective image to a panorama. The panorama consists of perspective images sharing the camera center where
pixel-to-pixel correspondences are obtained by planar tomography and 2) Texture mapping of a given geometry where multiple images of arbitrary camera poses establish pixel-to-pixel correspondences through depth-based unprojection and projection. We first introduce panorama generation (§4.1), which employs generation modules, and then multi-view depth-to-image generation (§4.2), which employs generation and interpolation modules. Since the interpolation module does not contain CAA blocks, §4.1 will also cover the design of the CAA block and explain how it is inserted into the multi-branch UNet.
4.1 Panorama generation
In MVDiffusion, a panorama is realized by generating eight perspective views, each possessing a horizontal field of view of  $90^{\circ}$  with a  $45^{\circ}$  overlap. To achieve this, we generate eight  $512 \times 512$  images by the generation module using a frozen pretrained stable diffusion model [47].
Generation module. The proposed module generates eight  $512 \times 512$  images. It accomplishes this through a process of simultaneous denoising. This process involves feeding each noisy latent into a shared UNet architecture, dubbed as the multi-branch UNet, to predict noises concurrently. In order to ensure multi-view consistency, a correspondence-aware attention (CAA) block is introduced following each UNet block. The CAA block follows the final ResNet blocks and is responsible for taking in multi-view features and fusing them together.
Correspondence-aware attention (CAA). The CAA block operates on  $N$  feature maps concurrently, as shown in Figure 2. For the i-th source feature map, denoted as  $\mathbf{F}$ , it performs cross-attention with the remaining  $(N - 1)$  target feature maps, represented as  $\mathbf{F}^l$ .
For a token located at position (s) in the source feature map, we compute a message based on the corresponding pixels  $\{\mathbf{t}^l\}$  in the target feature maps  $\{\mathbf{F}^l\}$  (not necessarily at integer coordinates) with local neighborhoods. Concretely, for each target pixel  $\mathbf{t}^l$ , we consider a  $K \times K$  neighborhood  $\mathcal{N}(\mathbf{t}^l)$  by adding integer displacements  $(d_x / d_y)$  to the (x/y) coordinate, where  $|d_x| < K/2$  and  $|d_y| < K/2$ . In practice, we use  $K = 3$  with a neighborhood of 9 points.
$$
\mathbf {M} = \sum_ {l} \sum_ {t _ {*} ^ {l} \in \mathcal {N} (\mathbf {t} ^ {l})} \operatorname {S o f t M a x} \left(\left[ \mathbf {W} _ {\mathbf {Q}} \bar {\mathbf {F}} (\mathbf {s}) \right] \cdot \left[ \mathbf {W} _ {\mathbf {K}} \bar {\mathbf {F}} ^ {l} \left(t _ {*} ^ {l}\right) \right]\right) \mathbf {W} _ {\mathbf {V}} \bar {\mathbf {F}} ^ {l} \left(t _ {*} ^ {l}\right), \tag {2}
$$
$$
\bar {\mathbf {F}} (\mathbf {s}) = \mathbf {F} (\mathbf {s}) + \boldsymbol {\gamma} (0), \quad \bar {\mathbf {F}} ^ {l} \left(t _ {*} ^ {l}\right) = \mathbf {F} ^ {l} \left(t _ {*} ^ {l}\right) + \boldsymbol {\gamma} \left(\mathbf {s} _ {*} ^ {l} - \mathbf {s}\right). \tag {3}
$$
The message M calculation follows the standard attention mechanism that aggregates information from the target feature pixels  $\{t_{*}^{l}\}$  to the source  $(s)$ .  $\mathbf{W}_{\mathbf{Q}}$ ,  $\mathbf{W}_{\mathbf{K}}$ , and  $\mathbf{W}_{\mathbf{V}}$  are the query, key and value matrices. The key difference is the added position encoding  $\gamma (\cdot)$  to the target feature  $\mathbf{F}^l (t_{*}^l)$  based on the 2D displacement between its corresponding location  $\mathbf{s}_*^l$  in the source image and s. The displacement provides the relative location in the local neighborhood. Note that a displacement is a 2D vector, and we apply a standard frequency encoding [50] to the displacement in both x and y coordinates, then concatenate. A target feature  $\mathbf{F}^l (t_{*}^l)$  is not at an integer location and is obtained by bilinear interpolation. To retain the inherent capabilities of the stable diffusion model [47], we initialize the final linear layer of the transformer block and the final convolutional layer of the residual block to be zero, as suggested in ControlNet [52]. This initialization strategy ensures that our modifications do not disrupt the original functionality of the stable diffusion model.
Panorama extrapolation. The goal is to generate full 360-degree panoramic views (seven target images) based on a single perspective image (one condition image) and the per-view text prompts. We use SD's inpainting model [48] as the base model as it takes one condition image. Similar to the generation model, CAA blocks with zero initializations are inserted into the UNet and trained on our datasets.
For the generation process, the model reinitializes the latents of both the target and condition images with noises from standard Gaussians. In the UNet branch of the condition image, we concatenate a mask of ones to the image (4 channels in total). This concatenated image then serves as the input to the inpainting model, which ensures that the content of the condition image remains the same. On the contrary, in the UNet branch for a target image, we concatenate a black image (pixel values of zero) with a mask of zeros to serve as the input, thus requiring the inpainting model to generate a completely new image based on the text condition and the correspondences with the condition image.
Training. We insert CAA block into the pretrained stable diffusion Unet [47] or stable diffusion inpainting Unet [48] to ensure multi-view consistency. The pretrained network is frozen while we use the following loss to train the CAA block:
$$
L _ {\mathrm {M V D i f f u s i o n}} := \mathbb {E} _ {\left\{\mathbf {Z} _ {t} ^ {i} = \mathcal {E} \left(\mathbf {x} ^ {i}\right) \right\} _ {i = 1} ^ {N}, \left\{\boldsymbol {\epsilon} ^ {i} \sim \mathcal {N} (0, I) \right\} _ {i = 1} ^ {N}, \mathbf {y}, t} \left[ \sum_ {i = 1} ^ {N} \| \boldsymbol {\epsilon} ^ {i} - \epsilon_ {\theta} ^ {i} \left(\left\{\mathbf {Z} _ {t} ^ {i} \right\}, t, \tau_ {\theta} (\mathbf {y})\right) \| _ {2} ^ {2} \right]. \tag {4}
$$
4.2 Multiview depth-to-image generation
The multiview depth-to-image task aims to generate multi-view images given depths/poses. Such images establish pixel-to-pixel correspondences through depth-based unprojection and projection. MVDiffusion's process starts with the generation module producing key images, which are then densified by the interpolation module for a more detailed representation.
Generation module. The generation module for multi-view depth-to-image generation is similar to the one for panorama generation. The module generates a set of  $192 \times 256$  images. We use depth-conditioned stable diffusion model [46] as the base generation module and simultaneously generate multi-view images through a multi-branch UNet. The CAA blocks are adopted to ensure multi-view consistency.
Interpolation module. The interpolation module of MVDiffusion, inspired by VideoLDM [2], creates  $N$  images between a pair of 'key frames', which have been previously generated by the generation module. This model utilizes the same UNet structure and correspondence attention weights as the generation model, with extra convolutional layers, and it reinitializes the latent of both the in-between images and key images using Gaussian noise. A distinct feature of this module is that the UNet branch of key images is conditioned on images already generated. Specifically, this condition is incorporated into every UNet block. In the UNet branch of key images, the generated images are concatenated with a mask of ones (4 channels), and then a zero convolution operation is used to downsample the image to the corresponding feature map size. These downsampled conditions are subsequently added to the input of the UNet blocks. For the branch of in-between images, we take a different approach. We append a black image, with pixel values of zero, to a mask of zeros, and apply the same zero convolution operation to downsample the image to match the corresponding feature map size. These downsampled conditions are also added to the input of the UNet blocks. This procedure essentially trains the module such that when the mask is one, the branch regenerates the conditioned images, and when the mask is zero, the branch generates the in-between images.
Training. we adopt a two-stage training process. In the first stage, we fine-tune the SD UNet model using all ScanNet data. This stage is single-view training (Eq. 1) without the CAA blocks. In the second stage, we integrate the CAA blocks, and the image condition blocks into the UNet, and only these added parameters are trained. We use the same loss as panorama generation to train the model.
We evaluate MVDiffusion on two tasks: panoramic image generation and multi-view depth-to-image generation. We first describe implementation details and the evaluation metrics.
Implementation details. We have implemented the system with PyTorch while using publicly available Stable Diffusion codes from Diffusers [51]. The model consists of a denoising UNet to execute the denoising process within a compressed latent space and a VAE to connect the image and latent spaces. The pre-trained VAE of the Stable Diffusion is maintained with official weights and is used to encode images during the training phase and decode the latent codes into images during the inference phase. We have used a machine with 4 NVIDIA RTX A6000 GPUs for training and inference. Specific details and results of these varying configurations are provided in the corresponding sections.
Evaluation metrics. The evaluation metrics cover two aspects, image quality of generated images and their consistency.
- Image quality is measured by Fréchet Inception Distance (FID) [14], Inception Score (IS) [38], and CLIP Score (CS) [32]. FID measures the distribution similarity between the features of the generated
Prompt: "a living room with a large glass table, white chairs and a giant window. A TV is attached to the wall and a fancy crystal chandelier is hanging on the ceiling."
Figure 3: Qualitative evaluation for panorama generation. The red box indicates the area stitched leftmost and rightmost content. More results are available in the supplementary material.
and real images. The Inception Score is based on the diversity and predictability of generated images. CLIP Score measures the text-image similarity using pretrained CLIP models [33].
- Multi-view consistency is measured by the metric based on pixel-level similarity. The area of multi-view image generation is still in an early stage, and there is no common metric for multi-view consistency. We propose a new metric based on Peak Signal-to-Noise Ratio (PSNR). Concretely, given multi-view images, we compute the PSNR between all the overlapping regions and then compare this "overlapping PSNR" for ground truth images and generated images. The final score is defined as the ratio of the "overlapping PSNR" of generated images to that of ground truth images. Higher values indicate better consistency.
The rest of the section explains the experimental settings and results more, while the full details are referred to the supplementary.
5.1 Panoramic image generation
This task generates perspective crops covering the panoramic field of view, where the challenge is to ensure consistency in the overlapping regions. Matterport3D [4] is a comprehensive indoor scene dataset that consists of 90 buildings with 10,912 panoramic images. We allocate 9820 and 1092 panoramicas for training and evaluation, respectively.
Baselines. We have selected three related state-of-the-art methods for thorough comparisons. The details of the baselines are briefly summarized as follows (full implementation details can be found in the appendix):
- Text2Light[6] creates HDR panoramic images from text using a multi-stage auto-regressive generative model. To obtain homographic images, we project the generated panoramicas into perspective images using the same camera settings (FoV=90°, rotation=45°).
- Stable Diffusion (SD)[36] is a text-to-image model capable of generating high-quality perspective images from text. For comparison, we fine-tuned Stable Diffusion using panoramic images and then extracted perspective images in a similar manner.
- Inpainting methods [11, 18] operate through an iterative process, warping generated images to the current image and using an inpainting model to fill in the unknown area. Specifically, we employed the inpainting model from Stable Diffusion v2 [36] for this purpose.
Results. Table 1 and Figure 3 present the quantitative and qualitative evaluations, respectively. We then discuss the comparison between MVDiffusion and each baseline:
Prompt: "Majestically rising towards the heavens, the snow-capped mountain stood, its jagged peaks cloaked in a shroud of ethereal clouds, its rugged slopes a stark contrast against the serene azure sky, and its silent grandeur exuding an air of ancient wisdom and timeless solitude."
Figure 4: Example of panorama generation of outdoor scene. More results are available in the supplementary material.
Prompt: "A large kitchen with a center island and white canbinets. A dining room with tables and chairs. A room with a lot of windows and a wooden floor."
Figure 5: Image&text-conditioned panorama generation results. More results are available in the supplementary material.
- Compared with Text2Light[6]: Text2Light is based on auto-regressive transformers and shows low FID, primarily because diffusion models perform generally better. Another drawback is the inconsistency between the left and the right panorama borders, as illustrated in Figure 3.
- Compared with Stable Diffusion (panorama)[36]: MVDiffusion obtain higher IS, CS, and FID than SD (pano). Like Text2light, this model also encounters an issue with inconsistency at the left and right borders. Our approach addresses this problem by enforcing explicit consistency through correspondence-aware attention, resulting in seamless panoramas. Another shortcoming of this model is its requirement for substantial data to reach robust generalization. In contrast, our model,
Method	FID ↓	IS ↑	CS ↑
Impainting [11, 18]	42.13	7.08	29.05
Text2light [6]	48.71	5.41	25.98
SD (Pano) [36]	23.02	6.58	28.63
SD (Perspective) [36]	25.59	7.29	30.25
MVDiffusion(Ours)	21.44	7.32	30.04
Table 1: Quantitative evaluation with Fréchet Inception Distance (FID), Inception Score (IS), and CLIP Score (CS).