DreamSparse: Escaping from Plato’s Cave with 2D Diffusion Model Given Sparse Views
Authors: Paul Yoo
Jiaxian Guo
Yutaka Matsuo
Shixiang Shane Gu
Venue: NeurIPS 2023 poster (2023)
Keywords: Novel View Synthesis
Diffusion Model
TL;DR: Novel view synthesis from sparse context views exploiting a frozen foundational 2D diffusion model.

Abstract: Synthesizing novel view images from a few views is a challenging but practical problem. Existing methods often struggle with producing high-quality results or necessitate per-object optimization in such few-view settings due to the insufficient information provided.  In this work, we explore leveraging the strong 2D priors in pre-trained diffusion models for synthesizing novel view images. 2D diffusion models, nevertheless, lack 3D awareness, leading to distorted image synthesis and compromising the identity. To address these problems, we propose $\textit{DreamSparse}$, a framework that enables the frozen pre-trained diffusion model to generate geometry and identity-consistent novel view images. Specifically, DreamSparse incorporates a geometry module designed to capture features about spatial information from sparse views as a 3D prior. Subsequently, a spatial guidance model is introduced to convert rendered feature maps as spatial information for the generative process. This information is then used to guide the pre-trained diffusion model to
encourage the synthesis of geometrically consistent images without further tuning. Leveraging the strong image priors in the pre-trained diffusion models, DreamSparse is capable of synthesizing high-quality novel views for both object and object-centric scene-level images and generalising to open-set images.
Experimental results demonstrate that our framework can effectively synthesize novel view images from sparse views and outperforms baselines in both trained and open-set category images. More results can be found on our project page: https://sites.google.com/view/dreamsparse-webpage.

DreamSparse: Escaping from Plato's Cave with 2D Diffusion Model Given Sparse Views
Paul Yoo
Yutaka Matsuo The University of Tokyo paulyoo, jiaxian.guo @ weeblab.tuutokyo...o
Jiaxian Guo"
Shixiang Shane Gu
Synthenized
Synthesized
Figure 1: Qualitative results on novel view synthesis of real-world objects from the CO3D dataset.
Abstract
Synthesizing novel view images from few views is challenging but practical problem. Existing methods often struggle with producing high-quality results or necessitate per-object optimization in such few-view settings due to the insufficient information provided. this work, we explore leveraging the strong 2D priors in pre-trained diffusion models for synthesizing novel view images. 2D diffusion models, nevertheless, lack 3D awareness, leading to distorted image synthesis and compromising the identity. To address these problems, propose DreamSparse, framework that enables the frozen pre-trained diffusion model to generate geometry and ddentity-consisteee novel view images. Specifically, DreamSparse incorporates a geometry module designed to capture features about spatial information from sparse views 3D prior. Subsequently, spatial guidance model is introduced to convert rendered feature maps as spatial information for the generative process. This information is then used to guide the pre-trained diffusion model encourage the synthesis of geometrically consistent images without further tuning. Leveraging the strong image priors in the pre-trained diffusion models, DreamSparse capable of synthesizing high-quality novel views for both object and object-centric scene- level images and generalising to open-set set images. Experimental results demonstrate that our framework can effectively synthesize novel view images from sparse views and outperforms baselines in both trained and open-set category images. More google. com/view/
results can be found on our project page:
"Correspondence Jiaxian Guo <jiaxian guoiiiinnguo@@ioo webbbb..uoooo.l...e....p...
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
"How could they see anything but the shadows if they were never allowed to move their heads?"
Platoos 's Allegory of the Cave
Plato's Allegory of the Cave raises thought-provoking question about perception of reality. Human perception of 3D objects is often limited to the projection of the world as 2D observations. We rely on our prior experiences and imagination abilities to infer the unseen views of objects from these 2D observations. As such, perception is to some degree a creative process retrieving from imagination. Recently Neural Radiance Fields (NeRF) (29] exhibited impressive results novel view synthesis by utilizing implicit functions to represent volumetric density and color data. However, NeRF requires large amount of images from different camera poses and additional optimizations to model the underlying 3D structure and synthesize an object from novel view, limiting its use in real-world applications such as AR/VR and autonomous driving. In most practical applications, typically only a few views are available for each object, in which case leads NeRF to output degenerate solutions with distorted geometry
72.
Recent works 76
66 20 started to explore sparse-view novel view synthesis, specifically focusing on generating novel views from a limited number of input images (typically 2-3) with known camera poses. Some of them [33 70, 72
33
introduce additional priors into NeRF, e.g. depth information, to enhance the understanding of 3D structures in sparse-view scenarios. However, due to the limited information available in few-view settings, these methods struggle to generate clear novel images for unobserved regions. To address this issue, SparseFusion [76] and GeNVS propose learning diffusion model as an image synthesizer for inferring high-quality novel-view images and leveraging prior information from other images within the same category. Nevertheless, since the diffusion model is only trained within a single category, it faces difficulties in generating objects unseen categories and needs further distillation for each object, rendering it still impractical.
In this paper, we investigate the utilization of 2D image priors from pre-trained diffusion models, such as Stable Diffusion [41). for generalizable novel view synthesis without further per-object training based on sparse views. However, since pre-trained diffusion models are not designed for 3D structures, directly applying them can result in geometrically and textually inconsistent images, compromising the object's identity in Figure 6 To address this issue, we introduce DreamSparse, framework designed to leverage the 2D image prior from pre-trained diffusion models for novel view image synthesis using a few (2) views. order to inject 3D information into the pre-trained diffusion model and enable it to synthesize images with consistent geometry and texture, we initially employ geometry module (57] as 3D geometry prior inspired by previous geometry-based works 40 39
70 57 20, which is capable of aggregating feature maps across multi-view context images and learning to infer the spatial features, i.e. features with spatial information, for the novel view image synthesis. This 3D prior allows us to render estimate from previously unseen viewpoint while maintaining accurate geometry.
23
However, due to the modality gap, the extracted spatial features cannot be directly used as the input to the pre-trained diffusion model for synthesizing geometrically consistent novel view images. Alternatively, we propose a spatial guidance module which is able to convert rendered feature maps into meaningful guidance to change the spatial features 74 3 in the pre-trained diffusion model, thus promoting the pre-trained diffusion model to generate geometry-consisten novel view image without altering its parameters. Nevertheless, the spatial guidance from feature maps alone cannot completely overcome the hallucination problem of the pre-trained models, as the spatial information encoded in spatial features is limited. This means it cannot guarantee identity consistency in synthesized novel view images. To overcome the limitation, we further propose noise perturbation method, where we denoise a blurry target view color estimate modulated with noise instead of a randomly initialized noise with the pre-trained diffusion model, so that we can further utilize the coarse shape and appearance information present in 3D geometry model's estimate. In this way, the frozen pre-trained diffusion model is able to effectively synthesize high-quality novel view images with improved consistency in both geometry and identity.
With the strong image synthesis capacity of the frozen pre-trained diffusion model, our approach offers several benefits: 1. The ability to infer unseen regions of
objects without additional training, as pre- pre-trained diffusion models already possess strong image priors learned from large-scale image-text

---

datasets. 2. A strong generalization capability, allowing the generation of images across various categories and even in-the-wild images using the strong image priors in the pre-trained diffusion models. 3. The ability to synthesize high-quality and even object-centric scene-level images without additional per-object optimization. 4. Since we do not modify the parameters or replace the textual embedding [24] of the pre-trained text-to-iman image diffusion model, the textual control capability of the pre-existing model is preserved. This allows us to alter the style/texture of the synthesized novel view image with textual control. The comparisons with other methods are given in Table [
In our experiments, we applied our framework to the real-world CO3D dataset [37] The extensive qualitative and quantitative results demonstrated that our approach outperformed baselines in both object-level and object-centric scene-level novel view synthesis settings by a large margin (about 50% in FID and 20% in LPIPS). Specifically, the results in open-set categories of DreamSSarss can even achieve competitive performance with those of the baselines in training domains, demonstrating the advantage of exploiting prior from pre-trained 21 diffusion model in open-set generalization.
Table 1: Comparisons with prior works on 1) works with sparse (2-6) input views. 2) hallucinates unseen regions. 3) generalizes instances in unseen categories because of the pre-trained backbones, and 4) free of training during inference time for novel view synthesis. 5) The ability to edit with
textual control.
Zero-I 1-to-3tt33 224
PixeiNerf
RegNeRF
VoISDF
NeRS 72
IBRNel 66
32
GPNR[57
3DFuse 50
3DMM
GeNvS
NF NF13
LINNS
SRTTH 17
VF120]
SF
Ours
1) Sparse-Viert Views
J
x
xxxx
X
X
E
X
X
X
2) Generate Unseen
X
Open-Set Generalization
X
4) Train-Free for NVS
5) Textual Control
2 Related Works
Geometry-basedd Novel View Synthesis. Prior research on Novel View Synthesis (NVS) largely focuses on recovering the 3D structure of a scene. This is achieved by estimating the parameters of the input images' cameras and subsequently applying multi-view stereo (MVS) technique, as indicated by several studies [55] 48 I. These methods use explicit geometry proxies to facilitate NVS. However, often fails to synthesize novel views that are both photo-realistic and comprehensive. particularly in the case of occluded areas. In order to address this issue, recent strategies [39 40 have attempted to integrate the 3D geometry derived from an MVS pipeline with NVS approaches based on deep learning. Despite its progress, the overall quality may deteriorate if the MVS pipeline encounters failures. The utilization of other explicit geometric representations has also been explored by various recent NVS techniques. These include the usage of depth maps [00 62). multi-plane images [9 78), or voxels 53 25). While geometry- based methods have advanced novel view synthesis [29), their limited capacity model uncertainties in unseen regions and the lack of a 2D prior often restrict their image synthesis quality and generalization for unseen category objects. In contrast, our approach seeks meld the strengths of both geometry-base methods and the robust 2D pre-trained diffusion model. This fusion showcases our method's prowess in synthesizing images that not only geometry-consisten but also of high quality, ultimately leading to enhanced generalization performance on objects in unseen categories.
Sperse-view Novel View Synthesize. Novel View Synthesis (NVS) from few views aims to generate a new image from novel viewpoint using limited number of 2D images [59] Because of the limited information available this setting. many prior works [60 79
54
18
51
58)
necessitate per-object or per-category test-time optimization, which makes them impractical. Other
57 I3 learn a latent prior over shapes or encode observations into feature representations and subsequently render novel views via volume rendering. order to hallucinate unseen regions arising from limited input view images, several recent approaches, such as 3DiM [67], SparseFusion 776,, NeRDi 77) Zero- 1-to-3 24] and GeNVS
efforts 662
58
66
3

---

30 Geonetry Module
Contest View Thature Target
Centect View 2 henture
Gradient Flow
(-
Module
Ensended Feature
Wilrre
Spatial
Trandoame
Content View
Figure 2: The illustration of the method. The first stage involves utilizing a 3D geometry module to estimate 3D structure and aggregate features from context views. In the next stage, a pre-trained 2D) diffusion model conditioned on the aggregate features is leveraged to learn spatial guidance model that guides the diffusion process for accurate synthesis of the underlying object.
4] have utilized image diffusion models. On the other hand, we propose framework to guide foundational 2D diffusion model while keeping frozen and simultaneously enforce 3D structure through a geometry- baase module.
Diffusion Model for Novel View Synthesize order achieve high-qualin quality novel view synthesis, recent works [16 56 41 45
21 28 68, 36, 32 30 8 introduce diffusion models conditioned on text images to generate or reconstruct 3D representations.
the context of novel input images and poses without 3D geometry, leaving vulnerabilities generating 3D-consistent images. SparseFusion [76) and GeNVS [ proposed to integrate additional geometric structure as training conditions for the diffusion model, thereby enabling the generation of 3D-consistent images. However, due to the absence of strong 2D prior in the diffusion models they employ, these approaches are challenging to generalize to objects in open-set categories, In contrast, our approach utilizes a frozen diffusion model pre-trained large-scale dataset [49]. enhancing its generalization ability for objects in open-set categories.
view synthesis, 3DiM 67 performs view synthesis only conditioned
Method
Given few context images (Cinpuda N Ji=1 and their poses w. aim to leverage the 2D image prior from pre-trained diffusion models to synthesize a novel view image at a target pose ttarget. Because pre-trained diffusion models
first employ geometry-aware module as 3D prior to extract features in 3D volume encoded from given context images. In order to leverage feature map renderings for the pre-traine pre-trained diffusion model, we further propose spatial guidance model to convert scometry-groundee diffusion models, guiding it to synthesize novel view images with correct geometry. However, we discover that relying solely on the diffusion model to sample an accurate reconstruction from random noise is inadequate at maintaining the object's ''' identity due the hallucination problem [19 (3) as shown in 6 Therefore, we propose a noise perturbation method to alleviate it, guiding the diffusion model to synthesize a novel view image with correct geometry and identity. The overall pipeline is illustrated in Fig.
3D-aware,
features into spatial features [3] 3 with consistent shape in
3.1 3D Geometry Module
In order infuse 3D awareness into 2D diffusion models, we propose geometry module extract features with geometric information for 2D diffusion models. In order to obtain geometry grounding. the process begins by casting query ray from farget and sampling uniformly spaced points along the ray. For each 3D point, we aim to learn density weighting for computing a weighted linear combination of features along the query ray. Subsequently, this per-ray feature aggregated across multiple context images, yielding unified perspective on the 3D structure we aim to reconstruct. Lastly, we render a feature map at π farget by raycasting from the target view. Next, we will present the details of this model.

---

Point-wise density weighting for each context image. For each input context image C Ca our geometry module first extracts semantic features using a ResNet50 [[] backbone and then reshapes the encoded feature into a 4 dimensional volumetric representation V; E [Rxxdxxxx where l and w are the height and width of the feature volume, respectively, d' is the depth resolution, and c is the feature dimension. We pixel-align the spatial dimensions of the volume to that of the original input image via bilinear upsampling. To derive benefit from multi-scale feature representation, we draw feature maps from the first three blocks of the backbone and reshape them into volumetric representations capturing the same underlying 3D space. Given a 3D query point P, along a query ray r'. we sample feature vectors from all three scales of feature volumes using trilinear interpolation concatenating them together. To calculate the point-wise density weighting, we employ transformer [64] with a linear projection layer at last followed by a softmax operation to determine a weighted linear combination of point features, resulting in per-ray feature vector. Limited by the page limit, we leave further implementation details in the Appendix [A]
Aggregate features from different context images. To understand the unified structure of the 3D object, we consolidate information from all given context images. More specifically, we employ an extra transformer, enabling context images that correlate with each query ray. The final feature map rendering at a query view is constructed by raycasting from the query view and computing per-ray feature vector for each ray. We render the feature map F at a resolution of R61x61 compositing features sampled from a 3D volume with geometry awareness with respect to the target view. We denote g as the feature map rendering function and F as the resulting aggregate feature map.
dynamically consolidate ray features from a varying number of
F Io (Tr imputs
Cimpute T farget)
where E Rdx664644 with d = 256, and Φ is trainable parameters.
Color Estimation To enforce geometric consistency, we directly obtain aggregation weights from the transformer outputs and linearly combine RGB color values drawn from the context images to render a coarse color estimate E at the query view.
E colcol ( imputs
Cimputs
target)
We impose a color reconstruction loss on the coarse image against the ground- truth image.
11gg,cotor (T inputa Cémputs ,T target) - Chaaret 2
Crecon
(3)
Ty 1argge
3.2 Spatial Guidance Module
Because of the modality gap between the feature map rendering F and the input of the pre-trained diffusion model, feature maps F cannot be directly used as the input of the pre-trained diffusion
model. To leverage the 3D information in the feature maps, we propose the spatial guidance module
3 that have a role in forming fine-grained spatial information in the diffusion process (normally the feature maps after the 4-th layer). To derive this guidance from spatial information in the features F, we construct our spatial guidance module following ControlNet (74] which trains separate copy of all the encoder blocks as well as the middle block from Stable Diffusion's 's U-Net with 1x convolution layers initialized with zeros between each block. Let To be the spatial guidance module, and intermediate outputs from each block of T' as (F') with weight X. In order to change the spatial features in the pre-trained diffusion model, we directly add Te, (F)) into the corresponding decoder block of the pre-trained diffusion model's U-Net. By optimizing To with gradients backpropagated from the pre-trained diffusion model's noise prediction objective.
to convert the feature maps into guidance to rectify the spatial features 63 74
Laiffusion = E.
To will be optimized to learn how to convert the feature map from the geometry module into semantically meaningful guidance to rectify spatial features in the diffusion process, enabling it to generate geometry-consister images. In Section 4.5), we visualize the spatial features after adding the spatial guidance show the effects of the spatial guidance model. During training, we jointly optimize 96 and Ta using the overall loss.
min Crecon + Laif fusion (Te) 0.0
(5)
5

---

While in training time, we use ground-truth image as zo to optimize fusion in inference time, we iittialize zo with an image rendered from concol
Noise Perturbation While spatial guidance module by itself able to guide the pre-trained diffusion model to synthesize novel view images with consistent geometry. It still cannot always synthesize images with the same identity as context views because of the hallucinate problem [9] B6) in the pre-trained models. To alleviate this problem, we propose adding noise perturbation to the novel view estimate from the geometry module and denoising the result with the pre-trained diffusion model, e.g. Stable Diffusion [42] so that it can leverage the identity information from the estimate. As shown by [27]. applying the denoising procedure can project the sample to a manifold of natural images. We use the formulations from denoising diffusion models [16] to perturb initial image zo = E with Gaussian noise to get a noisy image Zt as follows:
Zt Vārro + vI āge
(6)
where on depends scheduling hyperparameters and e (O, 1). During the training time, the noise is still randomly initialized, and we use the Noise Perturbation method in the inference time to improve the identity consistency. We show its ablation study in Section 4.5
4 Experiments
In this section, we first validate the efficacy of our DreamSparse framework on zero-shot novel view synthesis by comparing it with other baselines. Then, we perform ablation studies on important design choices, such as noise perturbation and visualization of spatial features, to understand their effects. We also present qualitative examples of our textual control ability and include discussion on observations.
4.1 Dataset and Training Details
Following SparseFusionn 76 we perform experiments real-world scenes from the Common
Objects 3D (CO3Dv2) 37,, a dataset with real-world objects annotated with camera poses. We train and evaluate our framework on the CO3Dv2 (37] dataset's fewview_ ttwtttrrnnnn and fewview _ev sequence sets respectively. We use Stable Diffusion v1.5 [42] as the frozen pre-trained diffusion model and DDIM (56) to synthesize novel views with 20 denoising steps. The resolutions of the feature map for the spatial guidance module and latent noise are set as 64 64 with spatial guidance weight À 2. The three transformers used in the geometry module all contain 4 layers, and the output feature map is rendered at a resolution of 64 x 64 to match the latent noise dimensions. We jointly train the geometry and the spatial modules on 8 A100-40GB GPUs for 3 days with batch size of 15. To demonstrate our framework's generalization capability at object-level novel view synthesis, we trained our framework training iteration, a query view and one to four context views of an object were randomly sampled as inputs to the pipeline. To further evaluate scene-level novel view synthesis capability, we trained our framework on the hydrant category, incorporating the full background, using the same training methodology above.
subset of 10 categories as specified in (37) During each
4.2 Competing Methods
We compare against previous state-of-the-arr (SoTA) methods for which open-source code is available. We have included PixeINeRF [69]. feature re-projection method, in our comparison. Additionally, we compare our methods against SparseFusion (76). the most recently published SoTA method that utilizes a diffusion model for NVS. We train our framework and SparseFusion on 10 categories of training sets. The PiXeINeRF training was conducted per category due to its category -specfffc hyperparameters. For fair comparison, all methods perform NVS without per-object optimization during the inference time. Because we do not replace the textual embedding in the pre-trained diffusion model, we use the prompt 'a picture of <class cclass_names* as the default prompt for both training and inference.

---

SF
Qurr
Figure 3:
Novel view synthesizing results on open-set category objects with the same context image inputs, where SF denotes SparseFusion [76] and GT denotes Ground-Trie Truth image. More results are given at our project webpage and appendix
Table 2: Quantitative evaluation metrics 10 subset of training categories from CO3D, where PN denotes PixelNerf [69] and SF denotes SparseFusion [76]. Limited by the width, we only show FID
and LPIPS score here.
Apple
Teddybear
FLD LPIPS FID LPPDS
FID
LPIPS
FED
LFIPS
FID
LPIPS
RID
LPIS
FID
LPIPS
FID
LPIPS
PID
LPWSS
FID
LPIPS
FID
LPPS
PN 247.1 0,57 3192 0.56
344.0
0.33
3805
0.38
340.8
0.63
3187
0.88
335.0
0.52
3339
0.45
352.1
0.36
258.9
0.47
326.1
0.54
255.8
0.34
185.7
202.6
0.29
0.19 45.8
0.19
122.5
0.25
105.2
0.28
67.2
0.21
0.18
0.28
100.2
0.20
0.28
69.1
0.16
81.8
0.21
4.3 Main Results Analysis
Given 2 context views, we evaluate novel view synthesis quality using the following metrics; FID [5]. LPIPS 771, and PSNR We believe that the combination of FID, LPIPS, and PSNR provides comprehensive evaluation of novel view synthesis quality. FID and I.PIPS measure the perceptual quality of the images, while PSNR measures the per-pixel accuracy. We note that PSNR has some drawbacks as metric for evaluating generative models. Specifically, PSNR tends to favor blurry images that lack detail. This is because PSNR only measures the per-pixel accuracy of an image, and does not take into account the overall perceptual quality of the image. By using all three metrics, can get a more complete picture of the quality of the images generated by our model.
4.3.1 Object Level Novel View Synthesis
In-Domain Evaluation We evaluate the performance of unseen objects NVS in training 10 cat- egories. The quantitative results are presented in Table 5 which clearly demonstrates that our method surpasses the baseline methods in terms of both FID and LPIPS metries. More specifically, DreamSparse outperforms SparseFusion by a substantial margin of 53% in the FID score and 28% in LPIPS. This significant improvement can be attributed to DreamSparse's capacity to generate sharper, semantically richer images, as depicted in Figure []] This indicates the benefits of utilizing the potent image synthesis capabilities of pre-trained diffusion models.
Open-Set Evaluation We also evaluate the performance of objects NVS in open-set 10 categories, because PixelNerf is per-category trained, we do not report its open-set generalization results. According to Table it evident that our method surpasses the baseline in both evaluation metries in all categories, surpassing the second-best method by 28% in LPIPS and 43% in FID. Moreover, the results derived from our method are not just competitive, but can even be compared favourably to the training category evaluations of the baseline in Table (122.2 vs 172.2 in FID and 0.24 vs 0.29 in LPIPS). This clearly illustrates the benefits of utilizing 2D priors from large-scale, pre-trained 2D diffusion model for open-set generalization. We also show the qualitative results in Figure and it shows that the novel view image synthesised by our method can still achieve sharp and meaningful results objects in open-set categories.
7

---

Table 3: Quantitative evaluation on open-set categories outside of the training split, where SP denotes SparseFusion [76].
Bicycle
Car
Cruch
Taplop
Microwaee
Matoreycle
Rew!
Toynlanc
TV
Wineglass
LPIS
LAIS
SP 217.7 0.34
209.5
0.30
201.1
0.44
223.8
0.30
2004
0.40
205.8
0.35
198.4
0.20
202.2
0.20
2014
0.33
208.5
0.35 0.15
0.23
212.9
Ours 154.7 0.26 116.5
0.22
129.9
0.31
167.3
0.26
127
0..99
115.4
0.
61.9
0.18
98.5
0.19
167.9
0.24
82.7
122.2 0.24
Table Quantitative evaluation metrics for object-centric scene-level novel view synthesis on the hydrant category from CO3D, where SF denotes SparseFusion [76].
View
Views
5 Views
FID
LPIPS
PSNRI
FID.
LPIPS
PSNR1
FID
LPIPS,
PSNR
PiXeINeRF [69]
343.89
0.75
13.31
319.96
0.74
13.94
286.30
0.71
14.59
Sparseuusion 76
272.72
0.81
13.05
255.05
0.78
13.55
231.73
0.71
14.91
Ours
75.63
0.59
13.02
73.47
0.56
13.48
70.62
0.54
14.15
Input
Ours
SparseFusion
PixeNNERF
Input
Ours
SparseFusion
PixelNeRF
Figure 4: Qualitative results of Object-Centric Scene Level novel view synthesis outputs from all baselines.
4.3.2 Object-Centric Scene Novel View Synthesis
We report evaluation results object-centric scene-level NVS in Table 7 As shown in the table, DreamSparse significantly outperforms the baselines in terms of FID and LPIPS scores, surpassing the accond-best performance by approximately 70% in FID and 24% in LPIPS, respectively. This underscores the effectiveness of our method in the context of object-centric scene NVS tasks. Despite method showing comparable performance to the baseline in terms of Peak Signal-to-Noids Ratio (PSNR), it's worth mentioning that PSNR often favors blurry images lacking in detail [46] 44 4 This becomes evident in Figure where despite our sharp and consistent synthesis results, PSNR still leans towards the blurry image produced by PixeINeRF.
4.4 Textual Control Style Transfer
As we do not replace or remove text conditioning in the pre-trained diffusion model, our method is additionally capable of controlling the image generation with text. We demonstrate an example use case where we conduct both novel view synthesis and style transfer via text in Figure 3

---

Context
+ +Photograph
+Swow
+ +'l painting +Moeet
GT
Figure 5: Qualitative results novel view synthesis with textual control style transfer
4.5
Ablation Studies
A= ==
1=1
==2
1=5
7=
1=5
==1
A=2
GT
Figure 6: Effect of the spatial guidance weighting spatial guidance signals at the residual connec- tions in the middle and decoder blocks of the diffusion model.
Contort
2od Blook
and Bloche
4th
Cuetpact
GT
Figure 7:
The spatial feature visualization with spatial guidance model, where context denotes the input context image, 2nd block denotes the visualization of feature maps from the 2nd block of the decoder and output denotes the synthesised novel view image.
Number of Input Views We investigate the performance by varying the number of context view inputs. The results depicted in Table 6 which clearly illustrates the enhancement in three evaluation metrics as the number of input views increases. Moreover, the performance disparity between the single view and multiple input views is less pronounced our method than in other baselines- a 6% difference vs 14% difference in Sparse Fusion vs a 17% difference in PixeINerf. This observation leads two key conclusions: 1) DreamSparse exhibits greater robustness in response variations in the number of context view inputs. 2) Despite the decrease in performance, DreamSparse can efficiently synthesise novel views from single input view.
Spatial Feature Visualization To investigate the impact of the spatial guidance model, we employ Principal Component Analysis (PCA) [35] to visualize the spatial features post-integration of spatial guidance following
(63]. As shown in Figure the visualized feature maps from the 2nd, 3rd, and 4th blocks the UNet decoder indicate that despite the contextual view's geometry varying from that of the novel view, the feature maps steered by our spatial guidance model maintain alignment with the geometry of the ground truth image. This consistency enables the pre-trained diffusion model to generate images that accurately mirror the original geometry.
Spatial Guidance Weight We investigate the effects of spatial guidance weight on the quality and consistency of synthesized novel view images. Our study varies the spatial guidance weight and the results in Fig 6) showed that when = 0 (indicating no spatial guidance), the pre-trained diffusion model failed synthesize novel view image that consistent in terms of geometry and identity. However, as the weight increased, the synthesized images exhibited greater consistency with the ground truth. It important note, though, that an excessively high weight could diminish the influence of features in the pre-trained diffusion model, potentially leading to blurry output. Given the trade-off between quality and consistency, we set À as the default hyperparameter.
Effect of Noise Perturbing Color Estimation The impact of the Noise Perturbation method is showeased Figure is evident that when the diffusion process begins from random noise, the spatial guidance model can successfully guide the mre-trained diffusion model to synthesize images

---

+10
+15
+20
Randow
+100
+15
+20
Randow.
Noise
Noise
Noise
Noise
Noise
Noise
Noise
Noise
Noise
Noise
Figure 8: Effect of noise perturbation level on novel view image synthesis. The images in the top row are visualizations of noised images input to the diffusion process, and the images in the bottom row are the synthesized images from diffusion model. +5 noise denotes the addition of steps of noise the color estimate from the geometry model, and random noise denotes randomly sampled Gaussian noise. All images were generated using a DDIM [56] sampler with 20 inference steps.
with consistent geometry. However, the color illumination density information partially lost, leading to distortions in the synthesized novel view. In contrast, synthesizing the image from the noise that added to the color estimation in the geometry model yields better results. As depicted in ++20 Noise" in Figure 8 the pre-trained diffusion model can effectively utilize the color information in the estimates, resulting a more consistent image synthesis. We also experimented with varying the noise level added to the estimate. Our observations suggest that if the noise added to the blurry estimation is insufficient, the pre-trained diffusion model struggles denoise the image because of the distribution mismatch between the blurry color estimate and Gaussian distribution, thereby failing to produce a sharp and consistent output.
Conclusion
In this paper, we present the DreamSparse, framework that leverages the strong 2D priors of frozen pre-trained text-to-image diffusion model for novel view synthesis from sparse views. Our method outperforms baselines on existing benchmarks in both training and open-set object-level novel view synthesis. Further results corroborate the benefits of utilizing a pre-trained diffusion model in object-centric scene NVS as well as the generation of text-controlled scenes style transfer, clearly outperforming existing models and demonstrating the potential of leveraging the 2D pre-trained diffusion models for novel view synthesis.
Limitations and Negative Social Impact Despite its capabilities, we discovered that our 3D Geometry Module struggles with generating complex scenes, especially ones with non-standard geometry or intricate details. This is due to the limited capacity of the geometry module and limited data, and we will introduce a stronger geometry backbone and train on larger datasets. On the social impact front, our technology could potentially lead to job displacement in certain sectors. For instance, professionals fields such as graphic design or 3D modelling might find their skills becoming less in-demand as Al- blsdd techniques become more prevalent and advanced. It's important to note that these negative implications are not exclusive to this study, and should be widely considered and addressed within the realm of Al research.
6 Acknowledgement
Computational resources of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Technology (AIST) were used for the experiments.
References
[1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski. Building rome in a day. Communications of the ACM, 54(10): 12,
2011.
[2] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. Renderdifflusin: Image diffusion for 3d reconstruction, inpainting and
10

---

generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 12608-126133 2023.
[3] Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. arXiv preprint arXiv:21
2021.
[4] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. arXiv preprint (xXiv:3304022000 2023.
[5] Angel X etc. Chang. Shapenet: An information-rich 3d model repository. arXiv:1512.03012,
2015.
[6] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 14124-
14133, 2021.
[7] Congyue Deng, Chiyu Jiang, Charles R Qi, Xinchen Yan, Yin Zhou, Leonidas Guibas, Dragomir Anguelov, et al. Nerdi: Single-vier view nerf synthesis with language- guided diffusion general
image priors. arXiv preprint arXiv:2212.03222
2022.
[8] Yiqun Duan, Xianda Guo, and Zheng Zhu. Diffusiondepth: Diffusion denoising approach for monocular depth estimation. arXiv preprint arXiv:2303. 303.05021, 2023.
[9] John Flynn, Michael Broxton, Paul Debevec, Matthew Du Vall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2367 2376, 2019.
[10] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to predict new views from the world's imagery. In Proceedings the IEEE conference on computer vision and pattern recognition. pages 5515-5524, 2016.
[11] Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M Seitz. Multi- view stereo for community photo collections. In 2007 IEEE 11th International Conference on Computer Vision, pages 1-8. IEFF, 2007.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Escaping plato's cave: 3d shape from adversarial rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9984-9993, 2019.
[14] Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny. Unsupervised learning of 3d object categories from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4700-4709, 2021.
[15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time- scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. 12:0006.11333 //399, 2020.
[17] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf diet: Semantically consis- tent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5885-5894, October 2021.
[18] Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12949-12958, 2021.
[19] Ziwei Ji, Nayeon Lec, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 35(12):1-38 2023.
[20] Joná Kulhánek, Erik Demnrr, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision (ECCV), 2022.
[21] Gang Li, Heliang Zheng, Chaoyue Wang, Chang Li, Changwen Zheng, and Dacheng Tao. Towards photorealistic 3d object generation and editing with text-guided diffusion models. arXiv preprint ar 2. 1.14108, 2022.
3ddesigner:
1.
[22] Ting-Hsuan Liao, Ge Songwei, Xu Yiran, Yao-Chih Lee, AlBahar Badour, and Jia-Bin Huang. Text-driven visual synthesis with latent diffusion prior. arXiv preprint arXiv:, 2023.
11

---

[23]
David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band- lindelimited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 16252-16262, 2022.
[24] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero---to-3: Zero-shot one image to 3d object. arXiv preprint arXiv:2303. / 1328,
2023.
[25] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv: /906 07751, 2019.
[26] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR),
June 2021.
[27] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[28] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Letent-nerf for shape-guided generation of 3d shapes and textures. arXiv preprint arXiv:2211 1.07600, 2022. [29] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1))99- 2021.
[30] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guidee 3d radiance field diffusion. arXiv preprint
arXiv:2212.0 22.00206 2022.
[31] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1-
15, 2022.
[32] Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, and Paul Guerrero. 3d-ldm: Neural implicit 3d shape generation with latent diffusion models. arXiv preprint
arXiv:2212.000 00042, 2022.
[33] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5480-54404 (5000-55900400 2022.
[34] Simon Niklaus, Long Mai, Jimei Yang, and Feng Liu. 3d ken burns effect from a single image. Graphics (ToG), 38(6):1-15, 2019.
ACM Transactions
[35] Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, (11))559-572) 1901. [36] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. Xiv, 2022.
[37] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scall scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision, 2021.
[38] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of riae-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 10901-1091 1, 2021.
[39] Gernot Riegler and Vladlen Koltun. Free view synthesis. In Computer Vision-ECCV 2020:
16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pages 623-640. Springer, 2020.
[40] Gernot Riegler and Vladlen Koltun. Stable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12216-12225. 2021.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022. [43] Robin Rombach, Patrick Esser, and Bjöm Ommer. Geometry-five view synthesis: Transformers and no 3d priors, 2021.
[44] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lec, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
12

---

[45] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. I 1487, 2022.
[46] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEFE Transactions Pattern Analysis
and Machine Intelligence, 2022.
[47] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -228, 2022.
[48] Johannes L Schonberger and Jan-Michael Frahm.
revisited. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[49] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint ar 00.0402. 2022.
[50] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin- Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2d diffusion model know 3d-consistency for robust text-to-3d generation. arXiv preprint arXiv:2303.0 077377 2023.
[51] Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. Advances in Neural Information Processing Systems, 33:10136-10147, 2020.
[52] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19922 2021.
[53] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießnnr, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2437-2446, 2019. [54] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
[55] Noah Snavely, Steven M Seitz, and Richard Szeliski. Photo tourism: exploring photo collections
in 3d. In ACM siggraph 2006 papers, pages 835-846. 2006.
[56] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.
arXiv:2010.02502 October 2020.
[57] etc Suhail, Mohammed. Generalizable patch-based neural rendering. In ECCV 2022, pages
156-174. Springer. 2022.
[58] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural repre- sentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2846-2855, 2021.
[59] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Multi-view 3d models from single images with a convolutional network. In European Conference on Computer Vision (ECCV), 2016.
[60] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox Multi-view 3d models from single images with a convolutional network. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pages 322-337. Springer, 2016.
[61] Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15182-15192, 2021.
[62] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 551-560, 2020.
[63] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-plas diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.125: 2022.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
13

---

[65]
Naveen Venkat, Mayank Agarwal, Maneesh Singh, and Shubham Tulsiani. Geometry-biased transformers for novel view synthesis. arXiv preprint arXiv:2301 000.20000.4450, 2023. Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin-Bruald Noah Snavely, and Thomas Funkhouser. Ibmet: Learning
[66]
multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690 2021.
[67] Daniel Watson, William Chan, Ricardo Martin-Brualla Jonathan Ho, Andrea Tagliasacchi. and Mohammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint
ar.ivv22 10. 04628, 2022.
[68] Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models. arXiv preprint arXiv:22/2 14704, 2022.
[69] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one or few images. CVPR, 2021.
[70] Alex etco Yu. pixelnerf: Neural radiance fields from one or few images. In CVPR, pages
4578 4587, 2021.
[71] Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. In Advances in Neural Information Processing Systems (NeurIPS). 2022.
[72] Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. Ners: neural reflectance surfaces for sparse- view 3d reconstruction in the wild. Advances in Neural Information Pro-
cessing Systems, 34:29835-2999 29847, 2021.
[73] Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. Iron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition, pages 5565-5574, 2022.
[74] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.055 05500.,553, 2023.
[75] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[76] etc Zhizhuo Zhou. Sparsefusion: Distilling now-conditioned diffusion for 3d reconstruction. In CVPR, 2023.
[77] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5826-5835, October 2021.
[78] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Sterco magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint arXiv: 1805v:88050000 09817, 2018.
[79] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei Efros. View synthesis by appearance flow. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 286-301. Springer, 2016
14

---

Supplementary Material of "DreamSparse: Escaping from Plato's Cave with 2D Diffusion Model Given Sparse Views"
Paul Yoo
Jiaxian Guo
Yutaka Matsuo The University of Tokyo
Shixiang Shane Gu
(paulyoo, jiaxian. webllab..u-tooyoo.ee
To enhance visual presentation, have showcased more qualitative samples paper's website. https://sites. google. method and additional quantitative comparisons with a recent baseline GBT and ablation study on the 3D geometry prior module.
wtwp//draasppaseeeeeeeeee Next, we present the details of our
A Method Implementation Details
Feature Extraction Backbone. To encode a volumetric feature field, we input context image through a ResNet50 [12] backbone and extract multi-scale feature maps from the first three blocks having feature dimensions 256, 512, and, 1024 respectively. To accommodate an additional depth dimension, the feature dimension is divided by the depth, which we set to 64, resulting in features of size 4, 8, 16 for the three volumes respectively. The final feature vector at a query 3D point is concatenation of features from all three volume scales, making the final feature vector dimension 28. Weighting Points Along a Ray. We employ Transformer [64] learn weightings for computing a linear combination of features of points along a given query ray. To form the input sequence for the Transformer, we follow the input parameterization of (57). For each point, we concatenate the final feature vector from the backbone with an encoding of the query point and depth along the ray as a positional cue. The query point encoding is computed by first extracting relative poses for all context views with respect to the target view and representing the ray from each context camera origin to the query point Plücker coordinates. The depth of each point along the ray additionally parameterized by sinusoidal positional encoding as in [29]. The output sequence of the Transformer followed by linear projection layer and Softmax operation to yield scalar densities for computing weighted sum of output features corresponding to points on a query ray.
Multi-view Aggregation Transformer. Once per-ray feature aggregate for each context view has been computed, we similarly learn to combine across all context views via a Transformer. We again concatenate per-ray feature vector output from the previous step with the same Plücker parameterization of query point and sinusoidal positional encoding of depth. All hidden and output dimensions of the Transformers are 256.
B Table Update with PSNR and an Additional Baseline
For completeness, we make an addition to the quantitative results to reflect PSNR measurements. Furthermore, we include another baseline GBT (65) a geometry-biased Transformer-based method for novel view synthesis from sparse context views, that demonstrates robust results on 10 categories of CO3D. Although GBT acheives comparable PSNR across training and open-set categories, synthe- sized novel views tend to be blurry especially for open-set categories and thus underperforming in FID and LPIPS.
"ooreesonndence to Jiaxian Guo <jiaxian.gun www.la...u.ooooo.a..oo.o 'Corresoondence Jiaxian Guo <jiaxian suaiiiinnnuggeeoo @ www.ab...uttoooo.ae.oo.oo
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table Quantitative evaluation metrics on 10 subset training categories from CO3D, where PN denotes PixelNerf [6], SF denotes SparseFusion [76], and Geom Prior is the geometry prior module
from our method.
Apple LPTST
Ball LPINS
Bench
Cake
Donui
Hyyrrnn
TID4
PSNK
F1D
PSNK 1
[ID
LPIPS
FSNK
F1D L
LPIPS J
PSNK
RID L
LPIES 1
PSNK 1
F1D1
LPIPS
FSNK
PN
14.76
319.3
0.56
14:35
344.0
0.58
15.
318.7
0.48
15:20
GBT
22.96
175.18
0.28
19.10
264.86
0.29
237.22
SF
110.9
0.28
20.91
143.5
0.30
20.25
255.8
0.34
17.21
185.7
0.33
19.39
126.5
0.29
20.02
103.7
0.23
19.35
Ours (Glom Prior) 123.98
0.53
22.96
134.41
0.33
22.00
238.95
0.38
19.42
197.00
0.35
21.41
162.12
0.34
22.95
211.03
0.36
21,90
18.68
Plant
Suitcuse
Teddybear LPIPS
Vase
Avg.
FID.
PSNK
L PSNK 1
FID
PSNK
LPIPS
PN
335.0
0.52
18.08
333.9
0.45
20.12
352.1
0.56
14.85
288.9
0.47
15.91
326.1
0.54
15.01
GBT
254.74
0.29
21.29
283.38
0.28
23,41
296.84
0.32
19.93
255.22
0.26
22.28
246.79
0.29
21.57
SF
168.5
19.60
202.6
(Geam Prior)
21.64
210.56
0.30
0.36
20.68
204.06
22.74
191.21
0.33
Ours
86.2
0.23
20.59
100.2
0.20
23.30
91.2
0.23
20.51
89.1
0.16
23
81.8
0.21
22.03
Table 6: Quantitative evaluation on 10 open-set categorics outside of the training split, where SF denotes SparscFusion [76].
Hicycle
Car LPIPS
Couch
Laptop
Microware LPIPS
Mononeyele
PSNK
PSNK
PSNK
L.
PSNK
SF
217.7
0.34
17.37
200.3
0.30
17.19
301.1
0.44
19.25
223.8
0.36
15.73
200.1
0.40
17.55
205.8
0.35
17.52
GBT
259,77
0.33
19.10
247.6
0.33
18.18
272.82
0.43
20.10
276.15
0.36
19.75
276.44
0.41
17.40
294,84
0.35
18.50
(Goom Prioc)
235.36
21836
0.34
21.58
231.58
259.89
Ours
154.7
0.26
18.16
116.5
0.22
18.38
129.9
0.31
20.36
167
0.26
20.13
127.2
0.29
18.95
115.4
0.29
17.35
Toyplane
TV
Winegloss
Ave:
FID
LPIFS
| PSNK
FID4
LPIES
FSNK
RID
LFIPS
PSNR
F10
LPPS L PSNK 1
F10
LPIPS PSNNI
SF
198.4
0.25
19.00
202.2
0.26
18.31
261.4
11.58
208.5
0.25
19:11
2129
18.79
GBT
246.56
270.80
22.78
0.24
Ours (Geom Prior)
172.42
0.32
21.11
234.66
0.33
21.18
252.78
0.37
23.98
224.79
0.26
21:21
219.64
0.36
20.65
Ours
61.9
0.16
22.30
98.5
0.19
20.53
167.9
0.24
23.65
82.7
0.15
22.08
122.2
0.24
20.19
C Ablation Study for Geometry Model
We further evaluate novel view image estimates from the stand-alone geometry model alongside other
baselines. From Tables 5 6 we observe the geometry model outperforming the full version of our method in terms of PSNR in many cases. This is consequence of the geometry model often rendering blurry coarse images as the view deviates from the context images. As PSNR, being a measurement favoring lower mean squared eтror across pixels, encourages mean pixel color, blurry images tend to score higher. However, significant drops in FID and LPIPS indicate lower image quality and perceptual dissimilarity, highlighting the importance of the 2D prior module in imagining missing details.
Table 7: Quantitative evaluation metrics for object-centric scene novel view synthesis on the hydrant
category from CO3D.
View LPIPS
Views LPIPS
5 Views
FID.
PSNR
FID.
PSNR1
FIDJ
LPIPS.,
PSNR
PixeINeRF [69
343.89
0.75
13.31
319.96
0.74
13.94
286.30
0.71
14.59
SparseFusion [76]
272.72
0.81
13.05
255.05
0.78
13.55
231.73
0.71
14.91
Ours (Geom Prior)
322.46
0.70
14.53
279.16
0.65
15.37
216.34
0.57
17.16
Ours
75.63
0.59
13.02
73.47
0.56
13.48
70.62
0.54
14.15
D Evaluation Setup
For computing evaluation metrics, we select 10 objects per category and sample 32 uniformly spaced camera poses from the held-out test split. We then randomly select a specified number of context views from the camera poses and evaluate novel view synthesis results the rest of the poses.
16

---

E
Additional Training and Evaluation on ShapeNet
We additionally train and evaluate our method and baselines on the cars category of the ShapeNet synthetic dataset of object renderings. We train all methods on 2458 training car objects each containing 50 views. We randomly sample to 3 context views during training. For evaluation, we randomly pick objects from the test set with 251 views per object. We randomly sample one context view and evaluate on all other novel views. The final metrics averaged across all views and objects. Quantitative results Table [ show improvements in FID and LPIPS over other baselines, indicating sharper image quality and diversity in synthesized results in comparison to the image distribution ShapeNet cars and more faithful structure and texture novel view reconstructions.
Table 8: Quantitative evaluation metrics for ShapeNet [5] cars category. For all baselines, only single context view was provided input compute the metries over novel views.
FID]
LPIPS|
PSNR1
PixeINeRF 669
154.96
0.14
23.32
SparseFusion (76
127.61
0.19
19.82
Ours
101.86
0.13
20.29
F
Additional Qualitative Results
F.1
Object-Centric Scenes
Context Image
Synthesized Novel Views
17

---

F.2
Open-Set Categories
Context Views
8ynthesized
Syathesized
Byathesized
Eymthesized
18


Reviews:
[{"review_id": "fgAPUORbMu", "reviewer": "Reviewer_fx5m", "summary": "This paper aims to leverage large-scale 2D diffusion models as priors to improve the task of novel view synthesis in the setting of sparse input views. The paper proposes an approach consisting of two stages: a first 3D-aware stage in which features from context views are aggregated in an encoded feature volume (which is differentiably rendered from the target view), and a second 2D stage in which a conditional image diffusion model produces the final output image. In the first stage, the process of aggregating, processing, and rendering features is similar to PixelNeRF/NeRFormer. The output of this stage (i.e. the RGB render) is perturbed by noise before being inputted to the conditional diffusion model in the second stage. The conditional diffusion model employed by the paper is Stable Diffusion equipped with ControlNet for conditioning on the input features (S3.2).", "questions": "### Modeling the background\n* You mention in Section 4.1 that you train on the hydrant category “incorporating the full background.” How exactly do you model the background? Did you consider any alternative approaches for this aspect of the problem?", "limitations": "The limitations are mentioned clearly at the end of the paper. This section suggests that they can be solved by “stronger geometry backbone and train[ing] it on larger datasets”. The question of whether or not this is all that is needed merits a longer discussion, but of course I understand that there are space limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "### The paper is well-structured.\n* The introduction is well-written and motivates the problem well. The methods section breaks down the contribution into provides helpful preliminaries (S3.1) in a good amount of detail. It also describes the contribution (S3.2-3.4) precisely without unnecessary complications.\n\n### The paper conducts ablations on different aspects of its architecture\n* Section 4.2 is very helpful for the reader, as it contains an analysis of how different aspects of the network contribute to its performance. \n\n### The zero-shot transfer experiments are nice. \n* It is good to see that the method trained on PASCAL transfers adequately to COCO.\n\n### The supplementary information includes code.\n* I looked through the code briefly and it looks quite clear. (I did not attempt to run it.) It includes configs which are easy to refer to while reading the paper.", "weaknesses": "### Comparison with other methods\n* This field is moving very quickly, so it is obviously not reasonable to expect the paper to compare to all other recent methods. However, it would be good to discuss and compare to at least some works beyond PixelNeRF and SparseFusion. For example, some subset of: 3DiM, Triplane Diffusion, Rodin, HoloDiffusion, NerfDiff, GeNVS. There is also the line of work considering distillation of pre-trained models, which is relevant because you use Stable Diffusion, such as:  NeuralLift-360 (Xu et al. 2022), NerDI (Deng et al. 2022), RealFusion (Melas-Kyriazi et al. 2023), Dreambooth-3D (Raj et al. 2023), Make-It-3D (Tang et al. 2023), Zero-1-to-3D (Liu et al. 2023). \n* Some of these are mentioned in the introduction and include results on the same dataset as you (e.g. CO3D hydrants and ShapeNet Cars), but they are not included in the tables. \n* To be clear, I would not expect the paper to compare to a large fraction of these methods, but I would expect the paper to compare to at least a few methods apart from PixelNeRF. \n\n### Performance on ShapeNet Cars\n* Along the lines of the comment above, the ShapeNet Cars table (Table 4 in the supplementary) shows relatively weak performance, but the table does not include many methods (including very old methods).  For example, {SRN, CodeNeRF, FE-NVS, VisionNeRF, 3DiM} all achieve better PSNR/LPIPS than the proposed model.\n\n### Compute/Memory Utilization\n* Most 3D-aware generative models tend to be compute-intensive (during training). It would be interesting to know how your model compares with others on these aspects, and how varying training time/data impacts results. \n\n### [Minor] LaTeX Spacing\n* It looks like some v-space tricks were applied (L181-182, L186-187, vertical padding on Table 2 and Table 3, all of Section 4.5). It’d be preferable to remove these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort sharing critical feedback regarding our work. We have addressed your points and questions about performance comparison and training costs below.\n\n> Comparison with other methods\n\nWe agree with you and would like to open a discussion on concurrently proposed 3D novel view synthesis works, and respectively highlight that our method contributes different aspects compared to the majority of these.\n\nWe would like to first clarify that our paper primarily contributes by 1) designing a framework to enable 2D frozen pre-trained diffusion model to perform novel-view synthesis without the need for time-consuming per-object test time distillation. 2) demonstrating that the utilization of a 2D prior in a pre-trained diffusion model can enhance the generalization capabilities of novel-view synthesis for unseen and open-set category objects.\n\nIn order to substantiate our contributions, we would like to compare with concurrently proposed 3D novel view synthesis works, but considered three factors for baselines for a fair comparison: \n\n**1) the availability of open-source codebase** (but 3DiM and GeNVS do not provide, so we cannot compare with them fairly because we synthesize images with different resolutions), \n\n**2) methods that don't require laborious test-time optimization for each object** (a criterion that Nerdi, Dreambooth-3D, NeuralLift-360, RealFusion, and Make-It-3D do not fulfill), \n\n**3) image-conditional methods that can operate under sparse observations (1~3),**(Triplane Diffusion and HoloDiffusion are not designed for this). \n\n**4) Following your suggestion, we tried to compare our work with the concurrent Zero 1-to-3, however, we found that Zero 1-to-3 has strong data assumption:** it requires objects to be located at the origin and the placements of the cameras to be pointing towards the origin as a result of its synthetic 3D training data from Objaverse. CO3D, on the other hand, is a dataset of real-life video captures of objects with noisy camera trajectories where each frame does not ideally point at the center of the object. And thus, converting the camera view parameterization for compatibility with Zero 1-to-3 is not straightforward.\n\nGiven that we've benchmarked our approach against the recently published SparseFusion (CVPR 2023) and GBT (arxiv 2023), which provides open-sourced codes, we believe our contributions have been suitably assessed in our setting. In this regard, our contributions remain unaffected regardless of comparisons with these baselines for other settings, and we would like to incorporate the thorough discussions with them into our revisions.\n\n\n\n \n\n\n\n\n\n\n\n\n\n\n> Performance on ShapeNet Cars\n\nThis is a good point. This is mainly due to the following reasons. 1) PSNR results is low because of the sharpness of the results from diffusion model. In our response  PDF, we further add the results generated from our geometry model, we can see the PSNR score  improves by 10%. 2)  In our setting, we perform novel view synthesis on images with 512 resolution which is more challenging than the baselines you mentioned (they focus on  128 resolution). 3)  As described above, we focus on novel view synthesis without test time per-object optimization, a more challenging but practical setting for real-world deployment while others do not. \n\n\n> Compute/memory utilization\nMost 3D-aware generative models tend to be compute-intensive (during training). It would be interesting to know how your model compares with others on these aspects, and how varying training time/data impacts results.\n\nOur primary motivation is to enable a frozen 2D diffusion model to conduct novel view synthesis without the need for test-time optimization, so we only train the geometry module and the spatial guidance model within our framework, eliminating the need to fine-tune the entire diffusion model. This approach significantly improves our training efficiency. To provide a specific comparison, our framework requires only 3 days of training on eight A100-40GB GPUs, whereas the concurrent work, GeNVS, demands 7 days.\n\nFurther, we provide the Figure about the relationship between training time and results in the PDF of response. The Figure clearly demonstrate xxx.\n\n> Background Modeling\n\nYou mention in Section 4.1 that you train on the hydrant category “incorporating the full background.” How exactly do you model the background? Did you consider any alternative approaches for this aspect of the problem?\nSorry for the confusing description. We do not design specific algorithms to model the background, and simply discard the foreground object mask during preprocessing to preserve the entirety of the scene including the background. Afterwards, the image goes through the same pipeline as the object-only scenario. The results demonstrate that the frozen pre-trained model can synthesize better novel view quality both object and scene level novel images when compared with models trained from scratch.\nIn our paper, we focus on enabling the frozen 2D diffusion model to synthesize novel view without per-object test time optimization, and show its generalization ability on unseen category samples. In the future, we would like to add more conditions, e.g. segmentation map, for synthesizing more complex scene-level novel view synthesis.\n\n\n>  v-space tricks:\n\nThank you for mentioning. We will remove them in the final version following your suggestions.\n"}, {"review_id": "NP3OgNICaU", "reviewer": "Reviewer_MN71", "summary": "The paper presents a method for novel view synthesis given sparse image observations of a scene. A diffusion model is used to generate the novel views. A 3D structured conditioning input is first computed, and a pretrained 2D diffusion model is fine-tuned to take this input and compute the novel view renderings. Using a pretrained 2D diffusion model enables better generalization to out-of-distribution object categories and further enables text-based editing. ", "questions": "Is it possible to perform 3D fusion using score distillation, like in sparsefusion?\n\nWhat limits the method to work on more complex scenes? Does the method generalize to non-hydrant scenes?", "limitations": "Some limitations are mentioned, however, some others I mentioned in weaknesses are not present. ", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper presents a good overview of the existing literature. The idea of fine-tuning large 2D diffusion models for novel view synthesis is a good one, and has been explored concurrently as well. The quantitative results demonstrate that the approach outperforms sparsefusion. The editing results are promising. ", "weaknesses": "The main limitation is that the results are not view-consistent. The identity and appearance of the hydrant and the background changes with the viewpoint. This limitation is not shared with pixelNeRF or sparsefusion. This is a severe limitation, as the approach does not let us observe any single 3D consistent scene. \n\nThe paper lacks technical novelty. The approach combines the insights from sparsefusion for 3D reasoning, and replaces their diffusion model trained from scratch with a fine-tuning of an existing 2D diffusion model. The paper is honestly written - the introduction mentions this. However, with limited technical novelty, the expectation would be that the introduced change would lead to significant practical benefits. This does not happen; as unlike sparsefusion, the presented approach does not enable reconstructing 3D consistent scenes. The noise perturbation step further exacerbates this problem. This part of the method independently processes the different viewpoints. \n\nWhile some generalization to unseen categories is shown, one would expect much stronger generalization when fine-tuning large models. Why is it not possible to reconstruct non-hydrant scenes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitation and novelty . and thus respectfully hope you can consider the response to the final decision.\n\n> The paper lacks technical novelty. \n\nWe would like to respectfully clarify that the differences between our work and SparseFusion are not merely the replacement of the backbone. Our approach goes beyond that as follows:\n\n**1)   Different approach to enable 2D diffusion model for 3D novel view synthesis:**\tUnlike SparseFusion training the diffusion model from scratch, how to control the output of a pre-trained diffusion is very challenging because of hallucination problem. Our approach further introduce a spatial guidance model that transforms these features into spatial features for guiding frozen pre-trained diffusion models synthesizing geometry consistent novel view images. **This approach experimentally avoids the need to fine-tune the diffusion model and test-time distillation.**\n\n**2) Use of a different 3D geometry modeling method:** In modeling geometry, we further learn to encode a volumetric feature representation and then trilinear sample features as opposed to bilinearly sampling pixel features from a 2D feature map as in SparseFusion. This helps enforce stronger 3D inductive biases into the feature representation of the objects and scenes and facilitates improved reconstruction capacity. We further show the ablation study on modeling feature volumes in the response PDF, and the results clearly show that even with a basic GPNR backbone (used in SparseFusion), our method outperforms SparseFusion, but it falls short of our final approach.\n\n\n**3) Different way of inferencing:**  Unlike most baselines (SparseFusion, DreamFusion or 3DFuse) which require time-consuming optimization per object at inference time, we focus on test-time training-free novel view synthesis in our paper, making deployment more straightforward. For a more detailed evaluation, we further show novel synthesis results in our website (as referenced in the Abstract and the main paper). Specifically, without test-time optimization or distillation, our method produces noticeably superior 360-degree videos compared to both PixelNeRF and SparseFusion in the Hydrant-Scene category. These results align seamlessly with our quantitative analyses.\n\n**In summary, we believe our work can bring different contributions from SparseFusion for the community. Our work is centered on how to utilize 2D prior from a frozen image diffusion model without test-time distillation, and how it can significantly bolster the generalization and efficacy of novel view synthesis.** Our results further unequivocally showcase the advantages of utilizing the 2D prior over the baseline methods. Specifically, we achieve approximately a 50% improvement in FID and a 20% enhancement in LPIPS compared to SparseFusion.  We believe our approach to utilizing the 2D prior for novel view synthesis can inspire more subsequent works (especially for open-set novel view synthesis without test time per-object distillation).\n\n\n\n\n> The main limitation is that the results are not view-consistent. \n\nThank you for your comment.   **We'd like to respectfully clarify that the inconsistency is not due to the limited performance of our method. Instead, it arises because our approach targets a more challenging yet practical task, i.e., performing novel view synthesis on unseen objects without test-time optimization/distillation.** This would significantly promote the convenience during the deployment.\n\nFurther, novel-view synthesis on CO3D image without test-time optimization/distillation is very challenging, **we further show the novel view synthesis results (without test-time optimization/distillation) of PixNeRF and SparseFusion in our website (Please refer to the link in Abstract and the main paper). The results clearly show that our method synthesizes significantly better 360 videos than PixelNeRF and SparseFusion on Hydrant-Scene category (without test-time optimization/distillation).** We genuinely hope that this response sufficiently addresses and alleviates your performance-related concerns.\n\n> While some generalization to unseen categories is shown, one would expect much stronger generalization when fine-tuning large models. Why is it not possible to reconstruct non-hydrant scenes?\n\nThank you for the question.  **We'd like to respectfully clarify that our paper does not fine-tune existing 2D diffusion models, but instead design a framework to enable 2D frozen pre-trained diffusion models to perform novel-view synthesis without the need for time-consuming per-object test time distillation.** Without fine-tuning pre-trained diffusion model, our method can still achieve approximately a 50% improvement in FID and a 20% enhancement in LPIPS compared to SparseFusion. We believe this significant improvement is able to demonstrate the benefits of enabling pre-trained models into novel view synthesis, even without test-time distillation.\n\nOn the other hand, we would like to clarify that joint object/background scene novel view synthesis itself is very challenging, even if for many recent works(e.g. CC3D (ICCV2023) and SceneDreamer (2023)), they normally need additional conditions or per-scene optimization, so it is very challenging for our work to generalize to other scenes without specific designs. In order to demonstrate the generalization ability of our method in single category scene novel view synthesis, **we further train our framework on other object-centric scenes and show the results in our webpage. (Please refer to the link in Abstract and main paper).**\n\n> Is it possible to perform 3D fusion using score distillation?\n\nYes, following your suggestion we show our samples with distillation in the webpage. (Please refer to the link in Abstract and main paper). Our method show consistent results as expected.\n"}, {"review_id": "aG7RGy3ZFt", "reviewer": "Reviewer_ZkGz", "summary": "The paper proposes a method for novel view synthesis using diffusion models. Given a sparse set of views, the method extracts per-pixel features for each of the views and reshapes these features to have a depth dimension by splitting the feature channels. Each feature sequence of per-pixel features, along with their depth along the viewing ray direction and positional encoding , is passed to a transformer to produce a per ray feature vector. Next, all feature vectors from multiple-views are aggregated using another transformer layer. To ensure the result is 3D consistent the input rgb pixels are linearly combined based on the aggregation weights and compared with the target image using an mse loss. The resulting features are fed to a spatial guidance module before being fed to a pre-trained stable diffusion model.\nThe method is compared to state of the art methods and outperforms them on the Co3D dataset. ", "questions": "- How does the method handle background? It seems non-trivial to aggregate multiple views with backgrounds compared to masked objects. Does that network just deal with it? What does the aggregated rgb input look like in these cases?\n- The rgb input and spatial guidance module both seem to be very crucial for performance (figure 6 and 8). Both contributions provide 3D / color guidance but they seem to be quite sensitive. Can the authors provide some insights with respect to failure cases?\n", "limitations": "Both limitations and potential negative societal impact are addressed appropriately.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A solid approach addressing novel view synthesis from sparse views. Especially the way the frozen pre-trained model is used to allow backpropping the gradients through the network not losing generalisability while training the geometry module as well as the spatial guidance module.\n- The method was thoroughly evaluated and outperforms the competitor methods quantitatively. Qualitatively, the method performs very strongly despite not being perfectly 3D consistent.\n- It is evident that using a pre-trained model helps with generalization of the method. This is demonstrated by the open-set evaluation as well as the scene editing capabilities of the network.\n- The method improves with an increasing number of views even when 5 views are used despite only training with a maximum of 4 views during training. This provides evidence of the view-aggregation robustness of the method.\n", "weaknesses": "- While the method is encouraged to produce 3D consistent results due to the color estimated reconstruction loss, there is no actual 3D constraint in the architecture that enforces any 3D consistency. This will give the network an ‘easy way out’ when something ‘wrong’ is easier to explain rather than the actual correct 3D consistent solution. This could possibly encourage hallucinations as seen in the results.\n- The method divides the per-pixel features from the ResNet backbone by the number of ‘depth samples’ which results in a feature dimension of 28 per depth sample. It does not make intuitive sense to distribute spatial features along the depth dimension. Can the authors explain their thoughts behind this design choice?\n- The authors claim that pixelNeRF wins in terms of PSNR due to blurry results. While I agree that PSNR can be misleading and their results look visually a lot better I disagree that this is due to blurriness. I believe the proposed method hallucinates wrong backgrounds which can be seen on the website where the backgrounds keep changing. It should be addressed more openly in the paper that the method produces sharp results, however, it does not always produce the exact information as seen from the input. It would be interesting to show the full ground truth video spins compared with the resulting spin given N input views.\n- The ‘Open-Set Category Results’ on the website do not show input images.\n- There are some related works regarding multi-view reconstruction missing (2, 3, 4). See missing references below. Also, the paper title is very similar to an existing relevant work (1) that was not cited. The paper should include this missing reference and provide appropriate credit.\n\n\nMissing citations:\n- [1] Henzler et al., Escaping Plato's Cave: 3D Shape From Adversarial Rendering. In ICCV 2019\n- [2] Anciukevicius et al., RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation. In CVPR 2023\n- [3] Henzler et al., Unsupervised Learning of 3D Object Categories from Videos in the Wild. In CVPR 2021\n- [4] Sitzmann et al., Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations. In NeurIPS 2019\n\n\nTypos:\n- Figure 2: ‘views.In’\n- Line 273: Table 3 → Table 4\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort sharing critical feedback regarding our work. We have addressed your points and questions below.\n\n>While the method is encouraged to produce 3D consistent results due to the color estimated reconstruction loss, there is no actual 3D constraint in the architecture that enforces any 3D consistency. This will give the network an ‘easy way out’ when something ‘wrong’ is easier to explain rather than the actual correct 3D consistent solution. This could possibly encourage hallucinations as seen in the results.\n\nThis is a good point. The feature utilized for color estimation is constructed from volumetric modeling of features, making it 3D-\naware and ensuring 3D consistency. Our paper illustrates how this can enhance images synthesized by the diffusion model. Nonetheless, as you've highlighted, there remains a possibility of encountering failure cases. We believe these can be ameliorated by integrating stronger geometry cues like depth or point cloud estimation models.\n\n> The method divides the per-pixel features from the ResNet backbone by the number of ‘depth samples’ which results in a feature dimension of 28 per depth sample. It does not make intuitive sense to distribute spatial features along the depth dimension. Can the authors explain their thoughts behind this design choice?\n\nThe core reasoning behind reshaping the pixel-aligned feature map from the ResNet into a feature volume and trilinear sampling features is to infuse stronger 3D inductive bias into the feature representations of the objects/scenes. Bilinear sampling pixel features from a 2D feature map can suffer from ambiguities in 2D images such as resolving occlusions. By learning to encode volumetric features enforces the representations to be more 3D aware, leading to better reconstruction capabilities.\n\n> The authors claim that pixelNeRF wins in terms of PSNR due to blurry results. While I agree that PSNR can be misleading and their results look visually a lot better I disagree that this is due to blurriness. I believe the proposed method hallucinates wrong backgrounds which can be seen on the website where the backgrounds keep changing. It should be addressed more openly in the paper that the method produces sharp results, however, it does not always produce the exact information as seen from the input. It would be interesting to show the full ground truth video spins compared with the resulting spin given N input views.\n\nThank you for your discussion. We agree that the hallucination of background is another factor to impact the PSNR score but we want to respectfully clarify that the blurriness is the main factor. To verify this, we further report the results of color estimation from the geometry model (normally are more blurry than output from diffusion model) in Table 1-3 in our Appendix. The results clearly demonstrate that the output from our geometry model can achieve better PSNR than all baselines.\n\nFurther, we show the synthesized videos with more input views on our website following your suggestions. The results clearly show that more input views can alleviate the background hallucination problem as expected.\n\n\n\n\n>  The ‘Open-Set Category Results’ on the website do not show input images\n\nWe have updated the website to reflect the input images for better visualization of novel view synthesis capabilities from the single view.\n\n\n\n> There are some related works regarding multi-view reconstruction missing\n\nThank you for pointing out the missing citations and typos in the writing. We will make sure to revise the draft following your suggestions.\n\n> How does the method handle background? It seems non-trivial to aggregate multiple views with backgrounds compared to masked objects. Does that network just deal with it? What does the aggregated rgb input look like in these cases?\n\nSorry for the confusing description. We do not design the specific algorithm to model the background, and we simply discard the foreground object mask during preprocessing to preserve the entirety of the scene including the background. Afterwards, the image goes through the same pipeline as the object-only scenario. The results demonstrate that the frozen pre-trained model can synthesize novel views of objects jointly with the full background scene with better quality than compared with models trained from scratch.\n\n> The rgb input and spatial guidance module both seem to be very crucial for performance (figure 6 and 8). Both contributions provide 3D / color guidance but they seem to be quite sensitive. Can the authors provide some insights with respect to failure cases?\n\nBy initializing the input noise for the diffusion model with rgb color output perturbed with noise instead of random Gaussian noise helps the generated sample to preserve components of the rgb image such as the structure and color. As the rgb output is a coarse and blurry estimate from the geometry prior, perturbing it with only a few timesteps leads to the generation of images not fully refined and sharp. We find that increasing the number of perturbation timesteps virtually endows the diffusion model to make more edits during the reverse diffusion process and is crucial for high quality image synthesis. \nSpatial guidance similarly facilitates the preservation of spatial structure and semantics of the generated output. Lower the spatial guidance weight, the denoising process has to solely rely on the noise perturbed input rgb. Large spatial guidance weights, however, decrease the generated image quality by undermining the base diffusion model’s feature maps and thereby suppressing image diversity(similar to having high weights for classifier-free guidance). Through empirical trials, we found noise perturbation of 20 timesteps and a spatial guidance weighting of 2.0 to generate the best quality images with the highest fidelity.\n\n"}, {"review_id": "8B4mabSrup", "reviewer": "Reviewer_ZbmG", "summary": "This work proposes an approach for finetuning a pretrained 2D diffusion model for novel view synthesis given a few or only a single input image at test time.\nThe two key contributions are a 3D geometry module and a spatial guidance module. The geometry module fuses features from multiple input views. The spatial guidance module then integrates the features into the pretrained 2D diffusion model.", "questions": "Why is there no comparison to zero-123, NVS-Fusion (which is named GeNVS officially and should also be referred to as GeNVS to avoid confusion) and 3DIM? Is there any reason why the experimental settings of the latter two could not be reproduced s.t. the numbers from the original paper can be used for comparison?\n\n\nWhy did you not just use the architecture from PixelNeRF as geometry module? Please consider adding an ablation that demonstrates the improvements made by the proposed geometry module.\n\n\nMinor comment: \nIn paragraph 3.2, F is a feature map (dx32x32) right? In this case, I find it confusing to refer to this feature map as 3D features in paragraph 3.2. I suggest rewording this as it is misleading to the reader.", "limitations": "The discussed limitations are not thorough enough. Please discuss limitations wrt to training/inference speed when using a large diffusion model, e.g. I suspect it takes a very long time to generate a video using this model. Further, the model is not guaranteed to be 3D consistent or identity-preserving and it should be discussed to what extent this is problematic in practice.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Leveraging strong 2D priors for novel view synthesis is an interesting approach that is currently a topic of great interest in the community. By building on pretrained text-to-image 2D diffusion models the method naturally allows for textual control. \nOverall, the approach is well-explained and the quantitative evaluation indicates a substantial improvement over the considered baselines.", "weaknesses": "The comparison to existing works is not clear enough and incomplete:\n- It is not clear why there is no comparison to zero-123 (code available), NVS-Fusion (no code available but evaluated on overlapping datasets, CO3D and ShapeNet, so the experimental setting can be reproduced), and 3DIM (no code available but the experimental setting can be reproduced e.g. on SRN cars and chairs)\nThese methods are also missing from Table 1 and it is not clear why.\n- for the related work section on geometry-based nvs methods and sparse 3d reconstruction: how does the proposed approach differ from the existing approaches and where is it similar? \n\n\nThe paper leaves substantial claims unsupported:\n- L.14 “enabling it (the approach) to generate geometrically consistent images” - there are no qualitative or quantitative results reported that evaluate the geometric consistency of the results.\n- L.78 “The ability to synthesize high-quality and even scene-level images” for which results are shown on the fire hydrant category of CO3D. While these images contain background these are not scene-level images but still object centric, single category images and these results do not demonstrate the ability to generate entire scenes.\n- L.7 “2D diffusion models, nevertheless, lack 3D awareness, leading to distorted image synthesis and compromising the identity”. This is never shown in the paper but is a central aspect of the story.\n\n\nThe motivation for designing the geometry module is not entirely clear. Why not simply use PixelNeRF and its projected features? It is not clear why a new architecture is needed here and that takes away from the contribution of adding the geometry module. Given that the spatial guidance module appears to be very similar to ControlNet, the technical contribution of this work seems to be rather small. \n\n\nThe qualitative evaluation is very sparse, only a few handpicked results are shown and there are no additional qualitative results in the supplementary and also no videos that would give an indication of the consistency of the method.\n\n\nThe writing of the paper should be improved. There are multiple grammatical mistakes and spelling errors, as well as inconsistencies in using terms like “geometry module” and “geometry model” which refer to the same thing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort sharing critical feedback regarding our work. We have addressed your points and questions below.\n\n> These methods are also missing from Table 1 and it is not clear why.\n\nThanks for pointing it out. We would like to add discussions with recent existing works in Table 1 as suggested.\n\n\n>  for the related work section on geometry-based nvs methods and sparse 3d reconstruction: how does the proposed approach differ from the existing approaches and where is it similar?\n\nFollowing your suggestion, we would like to add following description in the final version:\n\n*While geometry-based methods have advanced in novel view synthesis, their limited capacity to model uncertainties in unseen regions and the lack of a 2D prior often restrict their image synthesis quality and generalization for unseen category objects. In contrast, our approach seeks to meld the strengths of both geometry-based methods and the robust 2D pre-trained diffusion model. This fusion showcases our method's prowess in synthesizing images that are not only geometry-consistent but also of high quality, ultimately leading to enhanced generalization performance on objects in unseen categories.*\n\n> The motivation for designing the geometry module is not entirely clear. Given that the spatial guidance module appears to be very similar to ControlNet, the technical contribution of this work seems to be rather small.\n\nThank you for highlighting that aspect. Our geometry module's initial design was inspired by the GPNR model [1], which exploits stacked transformers to aggregate feature over multiple context views. GPNR benefits from strong generalization to unseen scenes due to its canonicalized positional encodings of pose information. However, we observed that due to the lack of an explicit 3D constraint, GPNR occasionally exhibited suboptimal performance with certain intricate objects, such as plants. Consequently, we integrated feature volumes into GPNR to bolster its 3D modeling capabilities. Our supplementary response PDF (Table 1 and 2) presents an ablation study of our geometry model. The results clearly show that even with a basic GPNR backbone, our method outperforms SparseFusion, though it falls short of our approach. From this, we deduce two critical insights:\n\n1) **Even when leveraging previous geometry-based methods, our strategy delivers superior outcomes compared to SparseFusion. This underscores the pivotal role the 2D prior plays in novel view synthesis and the importance of our work.**\n\n2) **Our geometry model design can indeed improve the performance of our framework.**\n\n**Regarding the similarity to ControlNet, we would like to clarify that we mainly inspire the spatial feature concept from ControlNet and Plug and Play Diffusion model.** By converting the features aggregated from sparse-view images into spatial feature, our framework successfully enable frozen pre-trained diffusion model to perform novel view synthesis. **Specifically, as far as we know, we are the first one to use spatial feature for novel view synthesis, and achieve the significant better results (50% in FID and 20% in LPIPS better than baselines). **We believe this will inspire subsequent works that apply pre-trained diffusion models into 3D area without fine-tuning pre-trained models or test-time distillation.**\n\n>  Why is there no comparison to zero-123, NVS-Fusion and 3DIM? Is there any reason why the experimental settings of the latter two could not be reproduced \n\n**In our setting, we perform novel view synthesis on image with 512 resolution which is more challenging setting than GeNVS and 3DiM (they synthesize images with 128 resolution), so it is unfair to compare with the results reported in their papers,  but we would like to compare with them in a fair setting once we have their codes.**  Following your suggestion, we tried to compare our work with the concurrent Zero 1-to-3, however, we found that Zero 1-to-3 has strong data assumption: it requires objects to be located at the origin and the placements of the cameras to be pointing towards the origin as a result of its synthetic 3D training data from Objaverse. CO3D, on the other hand, is a dataset of real-life video captures of objects with noisy camera trajectories where each frame does not point at the center of the object. And thus, converting the camera view parameterization for compatibility with Zero 1-to-3 is not straightforward.\n\nGiven that we've benchmarked our approach against the recently published SparseFusion (CVPR 2023) and GBT (arxiv 2023) with open-sourced codes, we believe our contributions have been suitably assessed. We also would like to discuss with GeNVS, 3DiM and zero-1-2-3 in our revisions.\n\n> The qualitative evaluation is very sparse, \n\n**Thank you. We indeed show more novel view synthesis samples in our website (Please refer to the link in Abstract and the main paper). The results clearly show that our method synthesise significant better 360 videos than PixelNeRF and SparseFusion on CO3D (without test-time optimization/distillation).** These results also are consistent with our quantitative comparisons (about 50% in FID and 20% in LPIPS better than SparseFusion). We will add these samples into Appendix as suggested and genuinely hope that this response sufficiently addresses and alleviates your performance-related concerns.\n\n>  I suspect it takes a very long time to generate a video using this model.\n\nGood point. We focus on novel-view synthesis without the test-time optimization. With our approach, we can take just roughly 2~3 seconds to generate per frame and generate a 360-degree video in just 3-5 minutes on a single A100-40GB GPU. In contrast, methods based on the diffusion model, such as SparseFusion, typically take much longer due to their reliance on test-time optimization. For instance, SparseFusion requires over an hour to produce a similar video.\n\n> writing\n\n We will revise them following your suggestions.\n\n"}, {"review_id": "4culpF0xyM", "reviewer": "Reviewer_rgvp", "summary": "This paper introduces a framework of 3D reconstruction from sparse using 2D image priors from pretrained diffusion models. It proposes a 3D geometry module which extracts 3D features from 2D images, and then incorporates these features into the diffusion process at novel views, enabling 3D awareness and view consistency. Experimentally, it shows good reconstruction results both quantitatively and qualitatively, with generalization to unseen object categories and (object-centric) scenes with complex backgrounds.", "questions": "- I'm very interested in the style transfer application. However, I've looked at the website and only found static results of individual views. Is this style transfer multiview consistent (or in other words, 3D aware)? I would like to see videos of consistent novel-view renderings, similar to the \"Single Image Scene-level Novel View Synthesis Results\". If there's no multiview consistency then it would seem to be a straightforward combination of novel-view synthesis and single-image style transfer.", "limitations": "Limitations and potential social impacts are well-discussed in this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes a 3D geometry module which incorporates 3D awareness into the 2D diffusion process. Instead of using NeRF as the 3D representation as in most 2D-to-3D generation works, this paper adopts the intuitions and techniques from image-based rendering or lightfield rendering methods, which I believe, is a very interesting and novel point.\n- It shows good experimental results both quantitatively and qualitatively compared to existing SoTA sparse-view methods. Adequate ablation studies and visualizations are provided to show the effectiveness of each module in the framework.\n", "weaknesses": "- Section 4.3.2 naming the task \"Scene Level Novel View Synthesis\" sounds a bit overclaimed to me. Though I agree that compared to pure object settings with foreground masks this setting with detailed backgrounds is more difficult, many scene-level works deal with rather complex scenes with multiple objects and complex configurations (like indoor environments, DTU dataset, etc.). So I would suggest a change of naming here.\n- It seems that input views are not shown except for Figure 4 -- it would be better to also show the input views side-by-side to the novel-view synthesis results, which can give a clearer sense of how much hallucination the model is performing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort sharing critical feedback regarding our work. We have addressed your points and questions below.\n\n>Section 4.3.2 naming the task \"Scene Level Novel View Synthesis\" sounds a bit overclaimed to me. Though I agree that compared to pure object settings with foreground masks this setting with detailed backgrounds is more difficult, many scene-level works deal with rather complex scenes with multiple objects and complex configurations (like indoor environments, DTU dataset, etc.). So I would suggest a change of naming here.\n\nThanks for your feedback. We would like to rephrase the task as “Object Centric Scene Level Novel View Synthesis” in the final version following your suggestions.\n\n\n\n\n\n\n\n\n>It seems that input views are not shown except for Figure 4 -- it would be better to also show the input views side-by-side to the novel-view synthesis results, which can give a clearer sense of how much hallucination the model is performing.\n\nWe have updated the website to reflect the input images for better visualization of novel view synthesis capabilities from the single view. (Please refer to the link in Abstract and the main paper).\n\n\n> I'm very interested in the style transfer application. However, I've looked at the website and only found static results of individual views. Is this style transfer multiview consistent (or in other words, 3D aware)? I would like to see videos of consistent novel-view renderings, similar to the \"Single Image Scene-level Novel View Synthesis Results\". If there's no multiview consistency then it would seem to be a straightforward combination of novel-view synthesis and single-image style transfer.\n\nUnlike prior work which often requires either the replacement of textual control capabilities or the fine-tuning of a pre-trained diffusion model, our paper is the first to demonstrate that it's feasible to utilize a frozen pre-trained diffusion model for novel view synthesis while simultaneously retaining the textual control ability. Moreover, while it's feasible to guarantee editing consistency via test-time per-object distillation like InstructNeRF2NeRF, our study intentionally centers on novel view synthesis devoid of this distillation. Given this constraint, ensuring editing consistency poses significant challenges and falls outside the purview of our current paper. We believe this is an interesting and practical direction, and we would like to introduce more conditions to enhance consistency throughout the editing process without test-time per-object distillation.\n"}]

Author rebuttal:
Thanks for the time and effort sharing critical feedback of every reviewer regarding our work. o address the questions and points raised by the reviewers, we have provided additional experimental results in our response PDF.

1) In the hydrant-scene dataset, we've plotted a figure (Figure 1 in the response PDF) that demonstrates the consistent improvement in our method's performance over training time.

2) Our ablation study on the geometry module shows how our design enhances the overall performance. Notably, even in the absence of the new geometry model design, our framework outperforms the baselines.

3) We also provde addition qualitative results in our response PDF, which include both input images and images synthesized by our method.

4) **For more samples, please refer to the website linked in the abstract and the main paper.**

    a) Comparisons of synthesized 360-degree videos. **The results clearly show that our method can synthesise more consistent samples than baselines without test-time distillation.**
 
    b) Additional object-centric scene level 360-degree videos.

    c) Results with test-time distillation, highlighting improved consistency.

    d) Hydrant Scene Novel View Synthesis results with 5 context views.

In summary, we believe our work can bring different contributions from previous works for the community. Our work is centered on 

1) how to utilize 2D prior from a frozen image diffusion model without test-time distillation.
2) how 2D prior from a **frozen** image diffusion can significantly bolster the generalization and efficacy of novel view synthesis. 

Our results further unequivocally showcase the advantages of utilizing the 2D prior over the baseline methods. **Specifically, we achieve approximately a 50% improvement in FID and a 20% enhancement in LPIPS compared to open-sourced SparseFusion (CVPR 2023).** We believe our approach to utilizing the 2D prior for novel view synthesis can inspire more subsequent works (especially for open-set novel view synthesis without test time per-object distillation).

**We also provide specific responses to address concerns from each reviewer. We believe our detailed response addresses concerns about our method's performance and novelty. We kindly hope that you can consider our response in your final decision. If there are additional concerns or questions, we are happy to further discussion and would appreciate your feedback.**

Warm regards,

The Authors of Submission 12217


Meta-review: The paper proposed a method for novel view synthesis from sparse views with the key idea being introducing a spatial guidance model (which is 3D consistent) for a frozen 2D diffusion model, such that the 2D diffusion model can generate novel view images (with a certain amount of 3D consistency) The results demonstrated a clear improvement over prior work (sparseFusion) both quantitatively and qualitatively.  The rebuttal successfully addressed the concerns of all the reviewers, and the key idea is a great contribution to the 3D community. Thus, I follow the consensus and recommend accepting this paper.

As mentioned by the reviewers and agreed by the authors, please tune down the claim on 3D consistency (in the introduction, Table 1 and corresponding experimental results), as the method does have the limitation of not guaranteeing 3D consistency.  Please also add the promised discussions and comparisons with other related works, as well as the clarification on the background modelling.
Decision: Accept (poster)