SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Paul Yoo Jiaxian Guo* Yutaka Matsuo Shixiang Shane Gu
The University of Tokyo  
{paulyoo, jiaxian.guo} @ weblab.t.u-tokyo.ac.jp
Figure 1: Qualitative results on novel view synthesis of real-world objects from the CO3D dataset.
Synthesizing novel view images from a few views is a challenging but practical problem. Existing methods often struggle with producing high-quality results or necessitate per-object optimization in such few-view settings due to the insufficient information provided. In this work, we explore leveraging the strong 2D priors in pre-trained diffusion models for synthesizing novel view images. 2D diffusion models, nevertheless, lack 3D awareness, leading to distorted image synthesis and compromising the identity. To address these problems, we propose DreamSparse, a framework that enables the frozen pre-trained diffusion model to generate geometry and identity-consistent novel view images. Specifically, DreamSparse incorporates a geometry module designed to capture features about spatial information from sparse views as a 3D prior. Subsequently, a spatial guidance model is introduced to convert rendered feature maps as spatial information for the generative process. This information is then used to guide the pre-trained diffusion model to encourage the synthesis of geometrically consistent images without further tuning. Leveraging the strong image priors in the pre-trained diffusion models, DreamSparse is capable of synthesizing high-quality novel views for both object and object-centric scene-level images and generalising to open-set images. Experimental results demonstrate that our framework can effectively synthesize novel view images from sparse views and outperforms baselines in both trained and open-set category images. More results can be found on our project page: https://sites.google.com/view/dreamsparse-webpage.
* Correspondence to Jiaxian Guo <jiaxian.guo@weblab.t.u-tokyo.ac.jp>
"How could they see anything but the shadows if they were never allowed to move their heads?"
- Plato's Allegory of the Cave
Plato's Allegory of the Cave raises a thought-provoking question about our perception of reality. Human perception of 3D objects is often limited to the projection of the world as 2D observations. We rely on our prior experiences and imagination abilities to infer the unseen views of objects from these 2D observations. As such, perception is to some degree a creative process retrieving from imagination. Recently, Neural Radiance Fields (NeRF) [29] exhibited impressive results on novel view synthesis by utilizing implicit functions to represent volumetric density and color data. However, NeRF requires a large amount of images from different camera poses and additional optimizations to model the underlying 3D structure and synthesize an object from a novel view, limiting its use in real-world applications such as AR/VR and autonomous driving. In most practical applications, typically only a few views are available for each object, in which case leads NeRF to output degenerate solutions with distorted geometry [33, 70, 72].
Recent works [76, 33, 70, 72, 17, 7, 66, 20] started to explore sparse-view novel view synthesis, specifically focusing on generating novel views from a limited number of input images (typically 2-3) with known camera poses. Some of them [33, 70, 72, 17, 7] introduce additional priors into NeRF, e.g. depth information, to enhance the understanding of 3D structures in sparse-view scenarios. However, due to the limited information available in few-view settings, these methods struggle to generate clear novel images for unobserved regions. To address this issue, SparseFusion [76] and GenVS[4] propose learning a diffusion model as an image synthesizer for inferring high-quality novel-view images and leveraging prior information from other images within the same category. Nevertheless, since the diffusion model is only trained within a single category, it faces difficulties in generating objects in unseen categories and needs further distillation for each object, rendering it still impractical.
In this paper, we investigate the utilization of 2D image priors from pre-trained diffusion models, such as Stable Diffusion [41], for generalizable novel view synthesis without further per-object training based on sparse views. However, since pre-trained diffusion models are not designed for 3D structures, directly applying them can result in geometrically and textually inconsistent images, compromising the object's identity in Figure 6. To address this issue, we introduce DreamSparse, a framework designed to leverage the 2D image prior from pre-trained diffusion models for novel view image synthesis using a few (2) views. In order to inject 3D information into the pre-trained diffusion model and enable it to synthesize images with consistent geometry and texture, we initially employ a geometry module [57] as a 3D geometry prior inspired by previous geometry-based works [40, 39, 31, 23, 70, 57, 20], which is capable of aggregating feature maps across multi-view context images and learning to infer the spatial features, i.e. features with spatial information, for the novel view image synthesis. This 3D prior allows us to render an estimate from a previously unseen viewpoint while maintaining accurate geometry.
However, due to the modality gap, the extracted spatial features cannot be directly used as the input to the pre-trained diffusion model for synthesizing geometrically consistent novel view images. Alternatively, we propose a spatial guidance module which is able to convert rendered feature maps into meaningful guidance to change the spatial features [74, 63, 3] in the pre-trained diffusion model, thus promoting the pre-trained diffusion model to generate geometry-consistent novel view image without altering its parameters. Nevertheless, the spatial guidance from feature maps alone cannot completely overcome the hallucination problem of the pre-trained models, as the spatial information encoded in spatial features is limited. This means it cannot guarantee identity consistency in synthesized novel view images. To overcome the limitation, we further propose a noise perturbation method, where we denoise a blurry target view color estimate modulated with noise instead of a randomly initialized noise with the pre-trained diffusion model, so that we can further utilize the coarse shape and appearance information present in 3D geometry model's estimate. In this way, the frozen pre-trained diffusion model is able to effectively synthesize high-quality novel view images with improved consistency in both geometry and identity.
With the strong image synthesis capacity of the frozen pre-trained diffusion model, our approach offers several benefits: 1. The ability to infer unseen regions of objects without additional training, as pre-trained diffusion models already possess strong image priors learned from large-scale image-text
datasets. 2. A strong generalization capability, allowing the generation of images across various categories and even in-the-wild images using the strong image priors in the pre-trained diffusion models. 3. The ability to synthesize high-quality and even object-centric scene-level images without additional per-object optimization. 4. Since we do not modify the parameters or replace the textual embedding [24] of the pre-trained text-to-image diffusion model, the textual control capability of the pre-existing model is preserved. This allows us to alter the style/texture of the synthesized novel view image with textual control. The comparisons with other methods are given in Table 1.
In our experiments, we applied our framework to the real-world CO3D dataset [37]. The extensive qualitative and quantitative results demonstrated that our approach outperformed baselines in both object-level and object-centric scene-level novel view synthesis settings by a large margin (about  $50\%$  in FID and  $20\%$  in LPIPS). Specifically, the results in open-set categories of DreamSparse can even achieve competitive performance with those of the baselines in training domains, demonstrating the advantage of exploiting prior from pre-trained 2D diffusion model in open-set generalization.
Table 1: Comparisons with prior works on 1) works with sparse (2-6) input views, 2) hallucinates unseen regions, 3) generalizes to instances in unseen categories because of the pre-trained backbones, and 4) free of training during inference time for novel view synthesis. 5) The ability to edit with textual control.
RegNeRF[33]	VolSDF[73]	NeRS[72]	IBRNet[66]	NF[37]	LFN[52]	SRT[47]	PixelNerf[70]	GPNR[57]	VF[20]	3DFuse[50]	SF[76]	3DIM[67]	Zero-1-to-3[24]	GeNVS[4]	Ours
1) Sparse-Views	✓	✓	✓	✗	✓	✓	✓	✓	✗	✓	✓	✓	✓	✓	✓	✓
2) Generate Unseen	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✓	✓	✓	✓	✓	✓
3) Open-Set Generalization	✗	✗	✗	✓	✓	✗	✓	✓	✓	✓	✓	✗	✗	✓	✗	✓
4) Train-Free for NVS	✗	✗	✗	✓	✓	✗	✓	✓	✓	✓	✗	✗	✓	✓	✓	✓
5) Textual Control	✗	✗	✗	✗	✗	✗	✗	✗	✗	✗	✓	✗	✗	✗	✗	✓
Geometry-based Novel View Synthesis. Prior research on Novel View Synthesis (NVS) largely focuses on recovering the 3D structure of a scene. This is achieved by estimating the parameters of the input images' cameras and subsequently applying a multi-view stereo (MVS) technique, as indicated by several studies [55, 48, 11, 1]. These methods use explicit geometry proxies to facilitate NVS. However, it often fails to synthesize novel views that are both photo-realistic and comprehensive, particularly in the case of occluded areas. In order to address this issue, recent strategies [39, 40] have attempted to integrate the 3D geometry derived from an MVS pipeline with NVS approaches based on deep learning. Despite its progress, the overall quality may deteriorate if the MVS pipeline encounters failures. The utilization of other explicit geometric representations has also been explored by various recent NVS techniques. These include the usage of depth maps [10, 62], multi-plane images [9, 78], or voxels [53, 25]. While geometry-based methods have advanced novel view synthesis [29], their limited capacity to model uncertainties in unseen regions and the lack of a 2D prior often restrict their image synthesis quality and generalization for unseen category objects. In contrast, our approach seeks to meld the strengths of both geometry-based methods and the robust 2D pre-trained diffusion model. This fusion showcases our method's prowess in synthesizing images that are not only geometry-consistent but also of high quality, ultimately leading to enhanced generalization performance on objects in unseen categories.
Sparse-view Novel View Synthesize. Novel View Synthesis (NVS) from few views aims to generate a new image from a novel viewpoint using a limited number of 2D images [59]. Because of the limited information available in this setting, many prior works [60, 79, 54, 18, 51, 58] necessitate per-object or per-category test-time optimization, which makes them impractical. Other efforts [62, 34, 58, 70, 61, 6, 66, 20, 54, 38, 17, 57, 14, 13] learn a latent prior over shapes or encode observations into feature representations and subsequently render novel views via volume rendering. In order to hallucinate unseen regions arising from limited input view images, several recent approaches, such as 3DiM [67], SparseFusion [76], NeRDi [7], Zero-1-to-3 [24] and GeNVS
Figure 2: The illustration of the method. The first stage involves utilizing a 3D geometry module to estimate 3D structure and aggregate features from context views. In the next stage, a pre-trained 2D diffusion model conditioned on the aggregate features is leveraged to learn a spatial guidance model that guides the diffusion process for accurate synthesis of the underlying object.
[4], have utilized image diffusion models. On the other hand, we propose a framework to guide a foundational 2D diffusion model while keeping it frozen and simultaneously enforce 3D structure through a geometry-based module.
Diffusion Model for Novel View Synthesize In order to achieve high-quality novel view synthesis, recent works [16, 56, 41, 45, 26, 2, 71, 77, 7, 21, 22, 28, 68, 36, 32, 30, 8] introduce diffusion models conditioned on text or images to generate or reconstruct 3D representations. In the context of novel view synthesis, 3DiM[67] performs view synthesis only conditioned on input images and poses without 3D geometry, leaving vulnerabilities in generating 3D-consistent images. SparseFusion [76] and GeNVS [4] proposed to integrate additional geometric structure as training conditions for the diffusion model, thereby enabling the generation of 3D-consistent images. However, due to the absence of strong 2D prior in the diffusion models they employ, these approaches are challenging to generalize to objects in open-set categories. In contrast, our approach utilizes a frozen diffusion model pre-trained on a large-scale dataset [49], enhancing its generalization ability for objects in open-set categories.
Given a few context images  $\{C_i^{inputs}\}_{i=1}^N$  and their poses  $\pi_i$ , we aim to leverage the 2D image prior from pre-trained diffusion models to synthesize a novel view image at a target pose  $\pi_{target}$ . Because pre-trained diffusion models are not 3D-aware, we first employ a geometry-aware module as a 3D prior to extract features in a 3D volume encoded from given context images. In order to leverage feature map renderings for the pre-trained diffusion model, we further propose a spatial guidance model to convert geometry-grounded features into spatial features [63, 3] with consistent shape in diffusion models, guiding it to synthesize novel view images with correct geometry. However, we discover that relying solely on the diffusion model to sample an accurate reconstruction from random noise is inadequate at maintaining the object's identity due to the hallucination problem [19, 36] as shown in 6. Therefore, we propose a noise perturbation method to alleviate it, guiding the diffusion model to synthesize a novel view image with correct geometry and identity. The overall pipeline is illustrated in Fig. 2.
3.1 3D Geometry Module
In order to infuse 3D awareness into 2D diffusion models, we propose a geometry module to extract features with geometric information for 2D diffusion models. In order to obtain geometry grounding, the process begins by casting a query ray from  $\pi^{target}$  and sampling uniformly spaced points along the ray. For each 3D point, we aim to learn density weighting for computing a weighted linear combination of features along the query ray. Subsequently, this per-ray feature is aggregated across multiple context images, yielding a unified perspective on the 3D structure we aim to reconstruct. Lastly, we render a feature map at  $\pi^{target}$  by raycasting from the target view. Next, we will present the details of this model.
Point-wise density weighting for each context image. For each input context image  $C_i^{inputs}$ , our geometry module first extracts semantic features using a ResNet50 [12] backbone and then reshapes the encoded feature into a 4 dimensional volumetric representation  $V_i \in \mathbb{R}^{c \times d \times h \times w}$ , where  $h$  and  $w$  are the height and width of the feature volume, respectively,  $d$  is the depth resolution, and  $c$  is the feature dimension. We pixel-align the spatial dimensions of the volume to that of the original input image via bilinear upsampling. To derive benefit from multi-scale feature representation, we draw feature maps from the first three blocks of the backbone and reshape them into volumetric representations capturing the same underlying 3D space. Given a 3D query point  $p_j$  along a query ray  $\mathbf{r}^i$ , we sample feature vectors from all three scales of feature volumes using trilinear interpolation concatenating them together. To calculate the point-wise density weighting, we employ a transformer [64] with a linear projection layer at last followed by a softmax operation to determine a weighted linear combination of point features, resulting in a per-ray feature vector. Limited by the page limit, we leave further implementation details in the Appendix A.
Aggregate features from different context images. To understand the unified structure of the 3D object, we consolidate information from all given context images. More specifically, we employ an extra transformer, enabling us to dynamically consolidate ray features from a varying number of context images that correlate with each query ray. The final feature map rendering at a query view is constructed by raycasting from the query view and computing per-ray feature vector for each ray. We render the feature map  $\pmb{F}$  at a resolution of  $\mathbb{R}^{64\times 64}$ , compositing features sampled from a 3D volume with geometry awareness with respect to the target view. We denote  $g$  as the feature map rendering function and  $\pmb{F}$  as the resulting aggregate feature map.
$$
\boldsymbol {F} = g _ {\phi} \left(\boldsymbol {\pi} ^ {\text {i n p u t s}}, \mathbf {C} ^ {\text {i n p u t s}}, \boldsymbol {\pi} ^ {\text {t a r g e t}}\right) \tag {1}
$$
where  $\pmb{F} \in \mathbb{R}^{d \times 64 \times 64}$  with  $d = 256$ , and  $\phi$  is trainable parameters.
Color Estimation To enforce geometric consistency, we directly obtain aggregation weights from the transformer outputs and linearly combine RGB color values drawn from the context images to render a coarse color estimate  $E$  at the query view.
$$
E = g _ {\phi , c o l o r} \left(\boldsymbol {\pi} ^ {i n p u t s}, \mathbf {C} ^ {i n p u t s}, \boldsymbol {\pi} ^ {t a r g e t}\right) \tag {2}
$$
We impose a color reconstruction loss on the coarse image against the ground-truth image.
$$
\mathcal {L} _ {\text {r e c o n}} = \sum_ {\boldsymbol {\pi} ^ {\text {t a r g e t}}} \| g _ {\phi , \text {c o l o r}} \left(\boldsymbol {\pi} ^ {\text {i n p u t s}}, \mathbf {C} ^ {\text {i n p u t s}}, \boldsymbol {\pi} ^ {\text {t a r g e t}}\right) - C ^ {\text {t a r g e t}} \| ^ {2} \tag {3}
$$
3.2 Spatial Guidance Module
Because of the modality gap between the feature map rendering  $\mathbf{F}$  and the input of the pre-trained diffusion model, feature maps  $\mathbf{F}$  cannot be directly used as the input of the pre-trained diffusion model. To leverage the 3D information in the feature maps, we propose the spatial guidance module to convert the feature maps into guidance to rectify the spatial features [63, 74, 3] that have a role in forming fine-grained spatial information in the diffusion process (normally the feature maps after the 4-th layer). To derive this guidance from spatial information in the features  $\mathbf{F}$ , we construct our spatial guidance module following ControlNet [74] which trains a separate copy of all the encoder blocks as well as the middle block from Stable Diffusion's U-Net with 1x1 convolution layers initialized with zeros between each block. Let  $T_{\theta}$  be the spatial guidance module, and intermediate outputs from each block  $j$  of  $T_{\theta,j}(\mathbf{F})$  with weight  $\lambda$ . In order to change the spatial features in the pre-trained diffusion model, we directly add  $T_{\theta,j}(\mathbf{F})$  into the corresponding decoder block of the pre-trained diffusion model's U-Net. By optimizing  $T_{\theta}$  with gradients backpropagated from the pre-trained diffusion model's noise prediction objective.
$$
\mathcal {L} _ {\text {d i f f u s i o n}} = \mathbb {E} _ {x _ {0}, t, \boldsymbol {F}, \epsilon \sim \mathcal {N} (0, 1)} \left[ \| \epsilon - \epsilon_ {\psi} \left(x _ {t + 1}, t, T _ {\theta} (\boldsymbol {F})\right) \| ^ {2} \right] \tag {4}
$$
$T_{\theta}$  will be optimized to learn how to convert the feature map from the geometry module into semantically meaningful guidance to rectify spatial features in the diffusion process, enabling it to generate geometry-consistent images. In Section 4.5, we visualize the spatial features after adding the spatial guidance to show the effects of the spatial guidance model. During training, we jointly optimize  $g_{\phi}$  and  $T_{\theta}$  using the overall loss.
$$
\min  _ {\phi , \theta} \mathcal {L} _ {\text {r e c o n}} \left(g _ {\phi}\right) + \mathcal {L} _ {\text {d i f f u s i o n}} \left(T _ {\theta}\right) \tag {5}
$$
While in training time, we use a ground-truth image as  $x_0$  to optimize  $\mathcal{L}_{diffusion}$ , in inference time, we initialize  $x_0$  with an image rendered from  $g_{\phi, color}$ .
Noise Perturbation While spatial guidance module by itself is able to guide the pre-trained diffusion model to synthesize novel view images with consistent geometry. It still cannot always synthesize images with the same identity as context views because of the hallucinate problem [19, 36] in the pre-trained models. To alleviate this problem, we propose adding noise perturbation to the novel view estimate  $E$  from the geometry module and denoising the result with the pre-trained diffusion model, e.g. Stable Diffusion [42], so that it can leverage the identity information from the estimate. As shown by [27], applying the denoising procedure can project the sample to a manifold of natural images. We use the formulations from denoising diffusion models [16] to perturb an initial image  $x_0 = E$  with Gaussian noise to get a noisy image  $x_{t}$  as follows:
$$
x _ {t} = \sqrt {\bar {\alpha} _ {t}} x _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon \tag {6}
$$
where  $\bar{\alpha}_t$  depends on scheduling hyperparameters and  $\epsilon \sim \mathcal{N}(0,1)$ . During the training time, the noise is still randomly initialized, and we use the Noise Perturbation method in the inference time to improve the identity consistency. We show its ablation study in Section 4.5.
In this section, we first validate the efficacy of our DreamSparse framework on zero-shot novel view synthesis by comparing it with other baselines. Then, we perform ablation studies on important design choices, such as noise perturbation and visualization of spatial features, to understand their effects. We also present qualitative examples of our textual control ability and include a discussion on observations.
4.1 Dataset and Training Details
Following SparseFusion [76], we perform experiments on real-world scenes from the Common Objects in 3D (CO3Dv2) [37], a dataset with real-world objects annotated with camera poses. We train and evaluate our framework on the CO3Dv2 [37] dataset's fewview_train and fewview_dev sequence sets respectively. We use Stable Diffusion v1.5 [42] as the frozen pre-trained diffusion model and DDIM [56] to synthesize novel views with 20 denoising steps. The resolutions of the feature map for the spatial guidance module and latent noise are set as  $64 \times 64$  with spatial guidance weight  $\lambda = 2$ . The three transformers used in the geometry module all contain 4 layers, and the output feature map is rendered at a resolution of  $64 \times 64$  to match the latent noise dimensions. We jointly train the geometry and the spatial modules on 8 A100-40GB GPUs for 3 days with a batch size of 15. To demonstrate our framework's generalization capability at object-level novel view synthesis, we trained our framework on a subset of 10 categories as specified in [37]. During each training iteration, a query view and one to four context views of an object were randomly sampled as inputs to the pipeline. To further evaluate scene-level novel view synthesis capability, we trained our framework on the hydrant category, incorporating the full background, using the same training methodology as above.
4.2 Competing Methods