Novel View Synthesis with Diffusion Models
Authors: Daniel Watson
William Chan
Ricardo Martin Brualla
Jonathan Ho
Andrea Tagliasacchi
Mohammad Norouzi
Venue: ICLR 2023 poster (2023)
Keywords: 3D
diffusion
ddpm
novel
view
synthesis
generative
models

Abstract: We present 3DiM (pronounced "three-dim"), a diffusion model for 3D novel view synthesis from as few as a single image. The core of 3DiM is an image-to-image diffusion model -- 3DiM takes a single reference view and their poses as inputs, and generates a novel view via diffusion. 3DiM can then generate a full 3D consistent scene following our novel stochastic conditioning sampler: the output frames of the scene are generated autoregressively, and during the reverse diffusion process of each individual frame, we select a random conditioning frame from the set of previous frames at each denoising step. We demonstrate that stochastic conditioning yields much more 3D consistent results compared to the naive sampling process which only conditions on a single previous frame. We compare 3DiMs to prior work on the SRN ShapeNet dataset, demonstrating that 3DiM's generated videos from a single view achieve much higher fidelity while being approximately 3D consistent. We also introduce a new evaluation methodology, 3D consistency scoring, to measure the 3D consistency of a generated object by training a neural field on the model's output views. 3DiMs are geometry free, do not rely on hyper-networks or test-time optimization for novel view synthesis, and allow a single model to easily scale to a large number of scenes.

Published as a conference paper at ICLR 2023
NOVEL VIEW SYNTHESIS WITH DIFFUSION MODELS
Daniel Watson
William Chan
Ricardo Martin-Brualla Google Research
Google Research, Brain
Google Research, Brain
Jonathan Ho
Andrea Tagliasacchi Google Research, Brain
Mohammad Norouzi Google Research, Brain
Google Research, Brain
ABSTRACT
We present 3DiM, diffusion model for 3D novel view synthesis, which is able translate single input view into consistent and sharp completions across many views. The core component of 3DiM is pose-conditional image-to-image dif- fusion model, which is trained take source view and its pose as inputs, and generates novel view for a target pose as output. 3DiM can then generate mul- tiple views that are approximately 3D consistent using a novel technique called stochastic conditioning. At inference time, the output views generated autore- gressively. When generating each novel view, one selects a random condition- ing view from the set of previously generated views each denoising step. We demonstrate that stochastic conditioning significantly improves 3D consistency compared to naïve sampler for image-to-iman diffusion model, which in- volves conditioning on single fixed view. We compare 3DiM to prior work on the SRN ShapeNet dataset, demonstrating that 3DiM's generated completions from a single view achieve much higher fidelity, while being approximately 3D consis- tent. We also introduce a new evaluation methodology. 3D consistency scoring, quantify the 3D consistency of a generated object by training a neural field the model's output views. 3DIM is geometry free, does not rely on hyper-networks or test-time optimization for novel view synthesis, and allows a single model to easily scale to large number of scenes.
Input view
3DIM ourputs conditioned different poses
Figure 1: Given a single input image on the left, 3DiM performs novel view synthesis and generates the four views on the right. We trained a single 441M parameter 3DiM on all of ShapeNet (without class- conditioning) and sample frames with 256 steps (512 score function evaluations with classifier free guidance). See the Supplementary Website (htips//3dddiiffusion. githuu..ooo for video outputs.
INTRODUCTION
Diffusion Probabilistic Models (DPMs) (Sohl-Dickstein et al.) 2015 Song & Ermon 2019 Ho
al. 2020), also known as simply diffusion models, have recently emerged as a powerful family of generative models, achieving state-of-the-art performance on audio and image synthesis (Chen ctt al., 2020 Dhariwal & Nichol 202I). while admitting better training stability over adversarial approaches (Goodfellow et al. 2014), well likelihood computation, which enables further ap- plications such as compression and density estimation (Song et al.) 2021 Kingma et al., 2021 Diffusion models have achieved impressive empirical results in variety of image-to-image trans- lation tasks not limited to exxt-to-imaee super- super-resolution, inpainting, colorization, uncropping, and
artifact removal (Song al. 2020 Saharia et al. 2021al Ramesh al. 2022 Saharia et al. 2022)

---

Published as a conference paper at ICLR 2023
One particular image-to-image translation problem where diffusion models have not been investi- gated is novel view synthesis, where, given a set of images of a given 3D scene, the task is to infer how the scene looks from novel viewpoints. Before the recent emergence of Scene Representation Networks (SRN) (Sitzmann et al., 2019) and Neural Radiance Fields (NeRF) (Mildenhall et al.) 2020), state-of-the-art approaches novel view synthesis were typically built on generative mod- cls (Sun et al. 2018 8) or more classical techniques on interpolation or disparity estimation (Park ct al.) 2017 Zhou al. 20188). Today, these models have been outperformed by NeRF-class models (Yu et al. Niemeyer et al., 2021 Jang & Agapito 202T), where 3D consistency is guaranteed by construction, as images are generated by volume rendering of a single underlying 3D representation (a.k.a. "geooettyetttaaaaa
models).
Still, these approaches feature different limitations. Heavily regularized NeRFs for novel view syn- thesis with few images such as RegNeRF (Niemeyer et al., (02T) produce undesired artifacts when given very few images, and fail to leverage knowledge from multiple scenes (recall NeRFs are trained on single scene, i.., one model per scene), and given one or very few views of novel scene, a reasonable model must extrapolate to complete the occluded parts of the scene. Pixel- NeRF (Yu et al., 2021) and VisionNeRF (Lin et al. 2022) address this by training NeRF-like models conditioned on feature maps that encode the novel input view(s). However, these approaches are regressive rather than generative, and as a result, they cannot yield different plausible modes and are prone to blurriness. This type of failure has also been previously observed in regression-based models (Saharia al. (0021b) Other works such as CodeNeRF (Jang & Agapito 2021) and LoL- NeRF (Rebain et al. (021) instead employ test-time optimization to handle novel scenes, but still have issues with sample quality.
In recent literature, geometry-free approaches (i.e., methods without explicit geometric inductive biases like those introduced by volume rendering) such as Light Field Networks (LFN) (Sitzmann et al., (021) and Scene Representation Transformers (SRT) (Sajjadi et al., 202T) have achieved re- sults competitive with 3D-aware methods in the "few-shot" setting, where the number of condition- ing views is limited (i.e., 1-10 images vs. dozens of images as in the usual NeRF setting). Similarly to our approach, EG3D (Chan et al.; 2022) provides approximate 3D consistency by leveraging generative models. EG3D employs a StyleGAN (Karras et al. 2019) with volumetric rendering, followed by generative super-resolution (the latter being responsible for the approximation). In comparison to this complex setup, we do not only provide significantly simpler architecture, but tuning experience compared GANS, which are well-known to be
also a simpler notoriously difficult to tune (Mescheder al. 2018).
Motivated by these observations and the success of diffusion models in image-to-image tasks, we introduce 3D Diffusion Models (3DiMs). 3DiMs are image-to-image diffusion models trained on pairs of images of the same scene, where we assume the poses of the two images are known. Draw- ing inspiration from Scene Representation Transformers (Sajjadi et al., 202T 3DiMs are trained to build conditional generative model of one view given another view and their poses. Our key dis- covery is that we can turn this image-to- to-igage model into a model that can produce an entire set of 3D-consistent frames through autoregressive generation, which we enable with our novel stochastic conditioning sampling algorithm. We cover stochastic conditioning more detail in Section 2.2 and provide an illustration in Figure 3 Compared to prior work, 3DiMs are generative (vs. regressive) geometry free models, they allow training to scale to a large number of scenes, and offer a simple end-to-end approach.
We now summarize our core contributions:
1. We introduce 3DiM, geometry-free image-to-image diffusion model for novel view synthesis. We introduce the stochastic conditioning sampling algorithm, which encourages 3DiM to gener- ate 3D-consistent outputs.
3. We introduce X-UNet, a new UNet architecture (Ronneberger et al. 2015) variant for 3D novel view synthesis, demonstrating that changes in architecture are critical for high fidelity results.
4. We introduce an evaluation scheme for geometry-free view synthesis models, 3D consistency scoring, that can numerically capture 3D consistency by training neural fields on model outputs.
2

---

Published as a conference paper at ICLR 2023
Inputs
Target*
noise
(R, 1)
Figure 2: Pose-conditional image-to-image training Example training inputs and outputs for pose-conditional image-to-image diffusion models, presented in Section 2.1 Given two frames from common scene and their poses (R. t). the training task is to undo the noise added to one of the two frames. (*) In practice, our neural network is trained to predict the Gaussian noise used to corrupt the original view the predicted view still just a linear combination of the noisy input and the predicted e.
2 POSE-CONDITIONAL DIFFUSION MODELS
To motivate 3DiMs, let consider the problem of novel view synthesis given few images from probabilistic perspective. Given a complete description of a 3D scene S, for any pose p. the view ((p) at pose p is fully determined from S, i... views are conditionally independent given How- ever, interested in modeling distributions of the form q g(.,..... F'm (1m++1 Fx,) without S, where views are no longer conditionally independent. A concrete example is the following: given the back of person's 's head, there multiple plausible views for the front. An image-to-image model sampling front views given only the back should indeed yield different outputs for each front view with guarantees that they will be consistent with each other especially if it learns the data distribution perfectly. Similarly, given a single view of an object that appears small, there is ambiguity on the pose itself: is it small and close, or simply far away? Thus, given the inherent ambiguity in the few-shot setting, we need a sampling scheme where generated views can depend on each other in order to achieve 3D consistency. This contrasts NeRF approaches, where query rays are conditionally independent given 3D representation S even stronger condition than imposing conditional independence among frames. Such approaches try to learn the richest possible representation for single scene S, while 3DiM avoids the difficulty of learning generative model for S altogether.
2. IMAGE-TO-IMAGE DIFFUSION MODELS WITH POSE CONDITIONING
Given data distribution q 222) of pairs of views from common we define an isotropic Gaussian process that adds increasing amounts of noise to data samples as the signal-to-noise-ratio À decreases, following Salimans & Ho (002))
poses P1 P2 SE(3),
q(=) (A) (xx) N(==! (A)+ o
where (-)) the sigmoid function. We can apply the reparamctrization trick (Kingma & Welling 2013) and sample from these marginal distributions via
(2) ((A) 2 a (- E e~ 1
Then, given a pair of views, we learn to reverse this process in one of the two frames by minimizing the objective proposed by Ho et al. (020)), which has been shown to yield much better sample quality than maximizing the true evidence lower bound (ELBO):
L(O)= = E.(ar. Ex,e [|E0(2) (A) 21, X, X.P1, - =
where Eg is neural network whose task is denoise the frame z (A) 21, and À is the log signal-to-noise-ratii To make our notation more legible, we slightly abuse
given a different (clean) frame
notation and from now will simply write =0(2)2) ..)). We illustrate training in Figure 2
3

---

Published as a conference paper at ICLR 2023
conditioning set
diffusion + stochastic conditioning
sung step
Figure 3: Stochastic conditioning sampler- We illustrate proposed inference procedure for 3DiM, outlined Section 2.2 There are two main components to our sampling procedure: (1) the autoregressive generation of multiple frames (illustrated vertically "step 1", "step 2", etc.), and (2) the denoising process to generate each individual frame (illustrated horizontally). When generating frame, we select previous frame the conditioning frame randomly at each denoising step (illustrated with the dice). Note that this is not part of 3DiM training; also, we omit the pose inputs the diagram to avoid overloading the figure.
2.2 3D CONSISTENCY VIA STOCHASTIC CONDITIONING
Motivation. We begin this section by motivating the need of our stochastic conditioning sampler. In the ideal situation, would model our 3D scene frames using the chain rule decomposition:
p(x) II p(2;|=<<)
(4)
This factorization is ideal, it models the distribution exactly without making any conditional in- dependence assumptions. Each frame is generated autoregressively. conditioned on all the previous frames. However, we found this solution to perform poorly. Due to memory limitations, we can only condition limited number of frames in practice, (i.e., k-Markovian model). We also find that, as we increase the maximum number input frames k, the worse the sample quality becomes. In order to achieve the best possible sample quality, an image-tu toagee-t-iin peee-tto-mmmgeg image model). Our key discovery is that, with A = 2, we can still achieve approximate 3D consistency. Instead of using sampler that is Markovian over frames, we leverage the iterative nature of diffusion sampling by varying the conditioning frame at each denoising step.
thus opt for the bare minimum of k = 2 (i.e.,
Stochastic Conditioning. We detail our novel stochastic conditioning sampling procedure that allows us to generate 3D-consistent samples from 3DiM. We start with a set of conditioning views X' {I1, ..., } of static scene, where typically k frame by running a modified version of the standard denoising diffusion reverse process for steps Amin \T AT-1 < ... < do Amaxi
very small. We then generate new
Ek+1 = a(2,) k+l
Dv) - -o(x,)) (2) (2(22 xx:)
(2e)
(5)
(Ae-1) *k+1
(de-1) (2)) #2+1)
9 (2x+1
(6)
where, crucially, i Uniform([1,. k)) ())n is re-sampled each denoising step. In other words, each individual denoising step is conditioned on a different random view from (the set that contains the input view(s) and the previously generated samples). Once we finish running this sampling chain and produce a final Øx+11, we simply add it to X' and repeat this procedure if we want to sample more frames. Given sufficient denoising steps, stochastic conditioning allows each generated frame to be guided by all previous frames. See Figure 3 for an illustration. In practice, we use 256 denoising steps, which find to be sufficient achieve both high sample quality and approximate 3D consistency. As usual in the literature, the first (noisiest sample) is just a Gaussian, i.e., 2, (Ar) (0, I), and at the last step Ag. we sample noiselessly.
We can interpret stochastic conditioning as a naïve approximation to true autoregressive sampling that works well in practice. True autoregressive sampling would require model of the form
4

---

Published as a conference paper at ICLR 2023
Inpux
Outpet
40
BigGAN residual black
self-attention block
3x
frame cross-attention block
weight sharing over frames
Figure 4: X-UNet Architecture We modify the typical UNet architecture used by recent work on diffusion models to accomodate 3D novel view synthesis. We share the same UNet weights among the two input frames, the clean conditioning view and the denoising target view. We add cross attention layers mix information between the input and output view, illustrated in yellow.
V log q [+++1 2i, æx)) but this would strictly require multi-view training data, while we are *k+1 ultimately interested in enabling novel view synthesis with as few as two training views per scene.
2.3 X-UNET
The 3DiM model needs neural network architecture that takes both the conditioning frame and the noisy frame as inputs. One natural way do this simply to concatenate the two images along the channel dimensions, and use the standard UNet architecture (Ronneberger et al. [0155] 5 Ho al. 2020). This "Concat- UNet" has found significant success in prior work of image-to- ponee-to-mmnn to-image diffusion models (Saharia et al. 2021b However, in our early experiments, we found that the Coneat-UNet yields very poor results there were severe 3D inconsistencies and lack alignment to the conditioning image. We hypothesize that, given limited model capacity and training data, is difficult to learn complex, nonlinear image transformations that only rely on sclf-attention. We thus introduce our X- K-UNet, whose
changes are (1) sharing parameters to process each of the two views, and (2) using cross attention between the two views. We find our X-UNet architecture to be very effective for 3D novel view synthesis.
We now describe X-UNet in detail. We follow Ho et al. 2020); Song et al. 2020)) and use the UNet (Ronneberger al., 2015) with residual blocks and self-attention. We also take inspiration from Video Diffusion Models (Ho et al., 2022) by sharing weights over the two input frames for all the convolutional and self-attention layers, but with several key differences:
1. We let each frame have its own noise level (recall that the inputs to DDPM residual block are feature maps as well positional encoding for the noise level). We positional encoding of >max for the clean frame. Ho et al. (2022) conversely denoise multiple frames simultaneously, each at the same noise level.
2. Alike Ho et al. (2020),
modulate each UNet block via FiLM (Dumoulin et al. 2018 but we use the sum of pose and noise-level positional encodings, as opposed to the noise-level embed- ding alone. Our pose encoding additionally differs in that they are of the same dimensionality as frames- they are camera rays, identical to those used by Sajjadi et al. (2021).
3. Instead of attending over "time" after each self-attention layer like Ho et al. (2022). which in our case would entail only two attention weights, we define a cross-attention layer and let each frame's 's feature maps call this layer to query the other frame's feature maps.
For more details on our proposed architecture, we refer the reader to the Supplementary Mate- rial (Sec We also provide comparison the "Concat-UNet" architecture in Section 3.2
5

---

Published as a conference paper at ICLR 2023
Input View
SRN
PixeINeRF
VisionNeRF
3DiM (ours)
Ground Truth
Figure 5: State-of-the-art comparisons Example input and output views of a 3DiM trained on the SRN cars dataset at the 128x 128 resolution, compared to existing geometry- methods. Results are best appreciated in the Supplementary Website (https://ddddiffusion githuu.in.ttubbo......
SRN cars
SRN chairs
PSNR (t)
SSIM (1)
FID (4)
PSNR (1)
SSIM (t)
FID (4)
Geometry-aware
SRN
22.25
0.88
41.21
22.89
0.89
26.51
PixcINeRF
23.17
0.89
59.24
23.72
0.90
38.49
VisionNeRF
22.88
0.90
21.31
24.48
0.92
10.05
CodeNeRF
23.80
#0.91
23.66
*0.90
Geometry-free
LFN
22.42
*0.9
22.26
*0.90
ENR
22.26
22.83
3DiM (ours)
21.01
0.57
8.99
17.05
0.53
6.57
Table 2: State-of-the-art comparisons Results the SRN ShapeNet benchmark comparing
3DiMs to prior work on novel view synthesis from a single image. (*) SSIM scores may not nec- essarily be computed with Gaussian kernel following Wang al. (2004) in practice we observe this can lead to differences of up to 0.02. We report these marked numbers directly from prior work.
EXPERIMENTS
We benchmark 3DiMs on the SRN ShapeNet dataset (Sitzmann et al.) 2019) to allow comparisons with prior work novel view synthesis from single image. This dataset consists of views and poses of car and chair ShapeNet (Chang et al.) 2015) assets, rendered at the 128x128 resolution.
We compare 3DiMs with Light Field Networks (LFN) (Sizzmann et al. 2021 and Equivariant Neural Rendering (ENR) (Dupont et al. 2020). two competitive geometry-Iree approaches, as well seometry-awart approaches including Scene Representation Networks (SRN) (Sitzmann et al. 2019). PiXeINeRF (Yu et al.] (2021). CodeNeRF (Jang & Agapito (022]) and the recent Vi- Sio (Lin et al. 2022). We include standard metrics the literature: peak signal-to-noise- ratio (PSNR), structural similarity (SSIM) (Wang et al. 20044), and the Fréchet Inception Distance (FID) (Heusel et al, 2017). To remain consistent with prior work, we evaluate over all scenes in the held-out dataset (each has 251 views), conditioning our models
single view (view #64) to pro- duce all other 250 views via a single call to our sampler outlined in Section 2.2 and then report the average PSNR and SSIM. Importantly, FID scores comparing the set all generated views against the set of all the ground truth views. Because prior work did not include FID scores, we acquire the evaluation image outputs of SRN, PixeINeRI and VisionNeRF and compute the scores ourselves. We reproduce PSNR scores, and carefully note that some of the models do not follow Wang et al. (2004) on their SSIM computation (they use uniform rather than Gaussian kernel), so we also recompute SSIM scores following (Wang et al. 2004). For more details, including hyperparameter choices, see Supplementary Material (Sec
computed with this same number of views,
Out-of-distribution poses on SRN chairs. We note that the test split of the SRN ShapeNet chairs was unintentionally released with diitrii-iitiiiiio poses compared to the train split: most test

---

Published as a conference paper at ICLR 2023
OOD input
3DIM outputs conditioned on different relative poses
Figure 6: 3DIM samples on OOD synthetic images We qualitatively demonstrate the text-to-3ID capabilities of 3DiM by synthethizing an input view with Imagen, a text-to-image model, and then create more views with 3DiM.
views are at a much larger distance to the object than those in the training dataset; we confirmed this via correspondence with the authors of Sitzmann et al. (2019). Because 3DiMs geometry-free, we (unsurprisingly) observe that they do not perform well on this out-of-distrbbttion evaluation task, as all the poses used at test-time are of scale never seen during training. However, simply merging, shuffling, and re-splitting the dataset completely fixes the issue. To maintain comparability with prior work, results on SRN chairs in Table 2 are those on the original dataset, and we denote the re-split dataset as "SRN chairs*" (note the *) all subsequent tables to avoid confusion.
State-of-the-art comparisons. While 3DIM does not necessarily achieve superior reconstruction errors (PSNR and SSIM, see Table qualitatively, we find that the fidelity of our generated videos can be strikingly better. We include an example in Figure [5] The use of diffusion models allows us to produce sharp samples, as opposed to regression models that are well-known to be prone to blurriness (Saharia al. 2021b) in spite of high PSNR and SSIM scores. This why we introduce and evaluate FID scores. In fact, we do not expect to achieve very low reconstruction errors due to the inherent ambiguity of novel view synthesis with a single image: constructing consistent object with the given frame(s) is acceptable, but will still be punished by said reconstruction metrics if it differs from the ground truth views (even if achieving consistency). We additionally refer the reader to Scction[3.]] where we include simple ablations demonstrating how worse models can improve the different standardized metrics in the literature. E.g., we show that with regression models, similarly to the baseline samples, PSNR and SSIM do not capture sharp modes well they scores these models as much better despite the samples looking significantly more blurry (with PixeINeRF, which qualitatively seems the blurriest, achieving the best scores).
Results on -istr--distriiution images.. We additionally show qualitative results attempting to lift single images in the wild (without known poses) into 3D. To this end, we train a 471M parameter 3DiM on all of ShapeNet (except for 10 objects per class which we leave out for testing). This model has identical parameters to the 3DiM used for SRN cars. However, it differs in that poses with relative camera extrinsics (i.e., the camera position and rotation corresponding to the noisy input frame is always the same) we do not know the poses of images in the wild. Our qualitative results on
generation include:
1. ShapeNet objects held out from the training data
2. Images in the wild we directly took from the internet (with a white background, minimal shadow, and corresponding any ShapeNet class)
3. Images synthesized by Imagen (Saharia et al. 2022) a text-to-image diffusion model
In order to make Imagen synthesize objects with white backgrounds and minimal shadows at the 128x128 resolution, in addition to natural language prompting, we inpaint a 5px white border at each denoising step when generating 64x64 image, and then upsample it using the second model on the Imagen cascade, i.e., contiionniiional super-resolution model. We find that this strategy is much more reliable than only relying on prompt engineering to create images with white backgrounds: this leads the text-to-image model to make the rest of background consistent with the fully white border, so the generated image results in an distr---iitiitttinn object while maintaining an in di-istributiin background. We present example novel views given ShapeNet test object in Figure [] and given a synthetic image from Imagen in Figure 6 We include several video samples for all the cases described above in the Supplementary Website (http://3d-difftsiion .ith..b.oo...t.ooo.........
7

---

Published as a conference paper at ICLR 2023
SRN cars
SRN chairs*
PSNR (1)
SSIM (1)
FID (L)
PSNR (t)
SSIM (t)
FID (U)
3DiM
21.01
0.57
8.99
14.29
0.48
3.30
+ no stochastic conditioning regression
23.82
0.59
1.87
15.73
0.51
1.05
22.55
0.85
17.45
16.85
0.76
20.70
Table 3: Ablation study removing stochastic conditioning and diffusion altogether from 3DiM.
Results included for novel view synthesis from single image on the SRN ShapeNet benchmark. (+) Re-split chairs dataset as detailed in Section not comparable to numbers in Table 2
Input view
3DiM
Image-to-Image
Concat-UNet
Regression
Figure 7: Ablations Example input and output views of 3DiM ablations on the SRN chairs dataset the 128x122 resolution. The second "Image-to-Image" column corresponds to removing stochastic conditioning sampler from Section 2.2. The third "Concat-UNet" column corresponds to removing our proposed architecture in Section 2.3 resorting to UNet that concatenates both images along the channel axis instead of weight sharing over frames. The fourth "Regression" column is one-step diffusion model trained from scratch.
3.1 ABLATION STUDIES
We present some ablation studies 3DiM. First, remove proposed sampler and use a naïve amage-to-image model as discussed at the start of Section 2.2 We additionally remove the of diffusion process altogether, i.e., regression model that generates samples via single denoising step. Naturally, the regression models are trained separately, but we can still use an identical architecture, always feeding white noise for one frame, along with its corresponding noise level Amin- Results are included in Table 3 and samples are included in Figure 7
Unsurprisingly, we find that both of these components are crucial achieve good results. The of many diffusion steps allows sampling sharp images that achieve much better (lower) FID scores than both prior work and the regression models, where in both cases, reconstructions appear blurry. Notably, the regression models achieve better PSNR and SSIM scores than 3DiM despite the severe blurriness, suggesting that these standardized metrics fail to meaningfully capture sample quality for geometry-free models, at least when comparing them to geometry-awan or non-stochastic re- construction approaches. Similarly, FID also has failure mode: naive image-to-image sampling improves (decreases) FID scores significantly, but severely worsens shape and texture inconsisten- cies between sampled frames. These findings suggest that none of these standardized metrics are sufficient effectively evaluate geometry-free models for view synthesis; nevertheless, observe that said metrics do correlate well with sample quality across 3DiMs and can still be useful indicators for hyperparameter tuning despite their individual failures.
8

---

Published as a conference paper at ICLR 2023
cars
chairs*
PSNR (1)
SSIM (1)
FID (1)
PSNR (f)
SSIM (t)
FID (4)
Concat-UNet X-UNet
17.21
0.52
21.54
12.36
0.44
5,15
21.01
0.57
8.99
14.29
0.48
3.30
Table 5: Architecture comparison study comparing the X-UNet and Concat- neural archi- tectures. Results are included for novel view synthesis from single image on the SRN ShapeNet benchmark, with the re-split chairs as detailed in Section 3] (*) Re-split chairs dataset as detailed in Section these numbers are not comparable to those Table
3.2 UNET ARCHITECTURE COMPARISONS
In order demonstrate the benefit of our proposed modifications, we additionally compare our proposed X-UNet architecture from Section 2.3 to the simpler UNet architecture following Saharia al. (2021b a (which we simply call "Concat-UNet") The Concat-UNee architecture, unlike ours, does not share weights across frames; instead, it simply concatenates the conditioning image to the noisy input image along the channel axis. To this comparison, we train 3DiMs on the Concat- UNet architecture with the same number of hidden channels, and keep all other hyperparameters identical. Because our architecture has the additional cross-attention layer at the coarse-resolution blocks, our architecture has slightly larger number of parameters than the Concat- Concat-UNet (~477 IM v.s. 42) Results are included in Table 5 Figure
While the Concat- architecture is able to sample frames that resemble the data distribution, we find that 3DiMs trained with our proposed X-UNet architecture suffer much less from 3D in- consistency and alignment to the conditioning frame. Moreover, while the metrics should be taken with a grain of salt as previously discussed, we find that all the metrics significantly worsen with the Coneat-UNet. We hypothesize that our X-UNet architecture better exploits symmetries between frames and poses due to our proposed weight-sharing mechanism, and that the cross-attention helps significantly align with the content of the conditioning view.
4 EVALUATING 3D CONSISTENCY IN GEOMETRY-FRES VIEW SYNTHESIS
As we demonstrate in Sections 3.1 and 3.2 the standardized metrics in the literature have failure modes when specifically applied to geometry-free novel view synthesis models, e.g., their inability to successfully measure 3D consistency, and the possibility of improving them with worse models. Leveraging the fact that volumetric rendering of colored density fields are 3D-consistent by design, thus propose additional evaluation scheme called "3D consistency scoring". Our metrics should satisfy the following desiderata:
1. The metric must penalize outputs that not 3D consistent.
2. The metric must not penalize outputs that are 3D consistent but deviate from the ground truth.
3. The metric must penalize outputs that do not align with the conditioning view(s).
In order to satisfy the second requirement, we cannol compare output renders to ground-truth views. Thus, one straightforward way to satisfy all desiderata is sample many views from the geometry- free model given single view, train a NeRF-like neural field (Mildenhall et al. (0200) on a fraction of these views, and compute set of metrics that compare neural field renders the remaining views. This way, if the geometry-free model outputs inconsistent views, the training of neural field will be hindered and classical image evaluation metrics should clearly reflect this. Additionally, to enforce the third requirement, we simply include the conditioning view(s) that were used to generate the rest as part of the training data. We report PSNR, SSIM and FID on the held-out views, although one could use other metrics under our proposed evaluation scheme.
We evaluate 3DiMs on the SRN benchmark, and for comparison, we additionally include metrics for models trained on (1) the real test views and (2) on image-to-image samples from the 3DiMs, i.e.. without our proposed sampler like we reported in Section 3 To maintain comparability, we sample the same number of views from the different 3DiMs evaluate in this section, all at the same poses and conditioned on the same single views. We leave out 10% the test views (25 out of 251) from neural field training, picking 25 random indices once and maintaining this choice of indices for all

---

Published as a conference paper at ICLR 2023
SRN
SRN chairs*
Training view source
PSNR (1)
SSIM (t)
FID (4)
PSNR (1)
SSIM (t)
FID (L)
Original data (3D consistent)
28.21
0.96
10.57
24.87
0.93
17.05
3DiM [<3. params)
28.48
0.96
29.55
22.90
0,86
58.61
3DiM (r>471M params)
28.53
0.96
22.09
18.84
0.79
98.78
no stochastic conditioning 25.78
0.94
30.51
17.61
0.75
116.16
Table 6: 3D consistency scores - For neural fields trained with different view sources, we compare renders to held-out views from the same sources.
subsequent evaluations. We also train models with more parameters (~1.3BB in order to investi- gate whether increasing model capacity can further improve 3D consistency. See Supplementary Material ( for more details on the neural fields we chose and their hyperparameters.
We find that our proposed evaluation scheme clearly punishes 3D inconsistency as desired the neural fields trained on image-to-image 3DiM samples have worse scores across all metrics com- pared to neural fields trained on 3DiM outputs sampled via our stochastic conditioning. This helps quantify the value of our proposed sampler, and also prevents the metries from punishing reasonable outputs that are coherent with the input view(s) but do not match the target views a desirable prop- erty due to the stochasticity of generative models. Moreover, we qualitatively find that the smaller model reported in the rest of the paper is comparable in quality to the ~ 133B parameter model on cars, though chairs, do observe significant improvement 3D consistency in our samples. Importantly, 3D consistency scoring agrees with our qualitative observations.
5 CONCLUSION AND FUTURE WORK
We propose 3DiM, a diffusion model for 3D novel view synthesis. Combining improvements in our X-UNet neural architecture (Section 2.3), with our novel stochastic conditioning sampling strategy that enables autoregressive generation over frames (Section [.2]. we show that from as few as single image we can generate approximately 3D consistent views with very sharp sample quality. We additionally introduce 3D consistency scoring to evaluate the 3D consistency of geometry-fiee -reee generative models by training neural fields on model output views, as their performance will be hindered increasingly with inconsistent training data (Section 4. We thus show, both quantitatively and visually, that 3DiMs with stochastic conditioning can achieve approximate 3D consistency and high sample quality simultaneously. and how classical metrics fail to capture both sharp modes and 3D inconsistency.
A noteworthy limitation of 3DiM is that, due to the geometry-free setup, it can only handle distri- butions of poses that exposed to durining training. In particular, further study is required where more variations to the poses are introduced. For example, varying focal lengths, sensor widths, making the camera not look exactly at the objects, and varying the distances from the camera to the objects more aggressively. Our use of stochastic conditioning might also exacerbate the need for many denoising steps. While we believe that finding effective architectures that allow conditioning on sets of images remains an important open problem, stochastic conditioning unlocks the ability to train on examples that only have two views and helps overcome memory-intensive neural archi- tectures. Similar approaches using randomized, sparse conditioning are already being explored in follow-up work on video generation (Davtyan et al., 2022) in order to enable efficient training when modeling the joint distribution computationally prohibitive.
We are most excited about the possibility of applying 3DiM, which can model entire datasets with a single model, to the largest 3D datasets from the real world though more research is required to handle noisy poses (due to the need for pose estimation) and other challenges, present in this context. Developing end-to-end approaches for high-quality generation that are 3D consistent by design like more recent work (Müller et al. 2022) and that also yield high-quality 3D meshes remain important research problems. Another potentially significant application of 3DiM is its use as a prior order to achieve 3D consistent generation via approaches that operate exclusively at sampling time (Poole) (et al.] 2022, Zhou & Tulsiani| 2022 Xu et al. (202)) in order to enable wide adoption of similar, disruptive technologies the 3D design industry.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal
Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao
Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com/
Eric R Chan, Connor Z. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry- -wware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16123-16133, 2022.
Angel Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 251..03022222 2015.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave-
grad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009. 00713, 2020.
Aram Davtyan, Sepehr Sameni, and Paolo Favaro. Randomized conditiinal flow matching for video prediction, arXiv preprint arXiv:221 I. 14575,, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 3(7), 2018
Emilien Dupont, Miguel Bautista Martin, Alex Colburn, Aditya Sankar, Josh Susskind, and Qi Shan. Equivariant neural rendering. In International Conference on Machine Learning, pp. 2761-2770. PMLR, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 2022.
Wonbong Jang and Lourdes Agapito. Codenerf: Disentangled neural radiance fields for object categories. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 12949-12988 12958, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.00366 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances neural information processing systems, 34:21696-21707 2021.

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
Diederik Kingma and Max Welling. arXiv: 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoor- thi. Vision transformer for nerf-based view synthesis from single input image. arXiv preprint arXiv:2207.05736. 2022.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference machine learning, pp. 3481-3490 PMLR,
2018.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421- Springer, 2020.
Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, and Matthias Nießner. Diffrf: Kendering-guidee 3d radiance field diffusion. arXiv preprint arXiv:2212.012066 2022.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Trans. Graph,, 41(4):102:11102:::: July 2022. doi: 10.1445/528223355012, URL https: /dtss:///oii.e. wttp./..gg/...11335555555.1 8530127
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjad,, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. arXiv preprint arXiv:21 2021.
Eunbyung Park, Jimei Yang. Ersin Yumer, Duygu Ceylan, and Alexander C Berg. Transformation- grounded image generation network for novel 3d view synthesis. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 3500-3509, 2017.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
Daniel Rehain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi. Lolnerf:
Learn from one look, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint arXiv:2111.05826, 2021a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07633 2021b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11 11487, 2022.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent scene representations. arXiv preprint arXiv:21/1.13152. 2021.
12

---

Published as a conference paper at ICLR 2023
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous ad-strecture-aware neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp. 2256-2266222 2265. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021.
Shao-Hua Sun, Minyoung Huh, Yuan-Hong Liao, Ning Zhang, and Joseph Lim. Multi-view to novel view: Synthesizing novel views with self-learned confidence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 155-171, 2018.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srini- vasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. CVPR, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Dejia Xu, Yifan Jiang, Peihao Wang. Zhiwen Fan, Wang. and Zhangyang Wang. Neurallift-360: Lifting an in-the-wild 2d photo to 3d object with 360 views. arXiv e-prints, pp. arXiv-2211, 2022.
Alex Yu, Vickie Yc, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4578 4587, 2021.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805. 09817, 2018.
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. arXiv preprint 1rXiv:221..00792, 2022.
13

---

Published as a conference paper at ICLR 2023
Novel View Synthesis with Diffusion Models Supplementary Material
ARCHITECTURE DETAILS
In order to maximize the reproducibility of our results, provide code in JAX (Bradbury al., 2018) for our proposed X-UNet neural architecture from Section 2.3 Assuming we have an input batch with elements each containing
"z", noisy input image (HxWx3 tensor)
"x", # conditioning view (HxWx3 tensor)
"logsnr", log sigmal-to-noise ratio noisy image (scalar)
"t", # camera positions (two 3d vectors)
"R", # camera rotations (two 3x3 matrices)
"K", # camera intrinsics (a.k.a, calibration matrix) (3x3 matrix)
neural network module is follows:
from typing import Optional
import flax. linen as nn import numpy as jnp import numpy as onp import visu3d as v3d
nonlinearity nn. swish
def initializer. .**
"""zeros
return nn in.intiialszers www.::/ww.a.raanncccaaiinnn..... 'fan_in', "truncated_nozmal))
def nearest_ pewprst__eighbboruuppappes (h: jnp. ndarray):
"""Nerest neighbor upsampling for multiple frames. "**
B, F, H, W, s.shape
h = h. reshape (B, F, H, 1, W, 1, C)
h jap. brp.broadcast_ff to (h, (B, F, H, 2, W, 2, C))
return rerehh (B, F, H 2, W 2, C)
def
(h: inp.ndarray. k: int 2)
"" """Average pooling downsampling for multiple frames. .**
return nn an.o (h, (1, k, k), (1, k, k))
def posenc_deee (timesteps: hmp.rry emb_ch: int, max_time: int 1000.0)
"""positional encodings for noise levels, following DDPM, 1000 is the magic number from DDPM. With different timesteps, we normalize by the number of steps but still multiply by 1000. timesteps == 1000.0 / max half dim emb ch // 2 10000 is the magic number from transformers.
emb onp. (10000) / (half dim
14

---

Published as a conference paper at ICLR 2023
emb jnp.ext (inp. arange (half_ dim, dtype=timesteps dt -emb)
emb = emb reshape (* ([1] * (timesteps, ndim 1)), emb. shape [-1])
emb timesteps [..., None] emb
emb = concatenate ( [[jpp.ssii sin (emb), jnp. cos (emb) l, axis=-1)
return emb
def posenc_nerffee (x: jnp. adarray,,,, min_deg: int o, max dex_ int = 15)
""*oonatteatee and its positional encodings, following NeRF. "** if min_deg == max_deg:
return x
scales = jnp.array ([2*x) for i in range (mi (min_deg, max_deg)
xb jnp. (eshape
(x...., None, :] scales Nonel), 1ist (x. + [-1]
emb jnp. sin (jnp. concatenate [xb, xb onp.pi / 2.01, axis=-1)) return jnp. concatenate ( [x, emb], axis=-1)
class GroupNorm (nn.Muulle): """Group normalization applied over frames.
@nn. compact
def _call (self, h: jnp.ndarray)
B, - Н, W, = h. shape
h = nn Groupoor (num 320ups=3)))) (h. reshape (B 2, 2, H, W, C)
return h. reshape (B, 2, H, W, C)
class FnIM
"""Feature linear modulation, "*"
features: int
@nn.
def __call 1__caall_ssee_______________ h: inprrddrray, emb: inp. ndarray)
emb nn nn.Dense (2 * self. features) monnineariin (emb) scale, shift jnp.split (emb, 2, axis=-1) return h (1.0 + scale) shift
class ResnetBlock (nn. Monoule)
SasigGAN-styll residual block, applied over frames.
features: Optional [int] None
dropout: float 0.0
resample: Optional [str] None
enn.compact def _aal__ll_ (self, h_in: jnp.ndarray, emb: ndarray, *, train: bool)
b h_in.
features if self. features is None else self. features
h = nonl innaritiy (GroupNorm() (h_in))
if self. resample is not None:
updown
'up': wearest_neigbbor_upsmmy
'down': ovgpool_downsampll
} [self. resamlee
15

---

Published as a conference paper at ICLR 2023
h updown (h)
h in = updown (h_in)
h = mn conv (featurre, kernel_size= (1, 3, 3), strides= (1, 1, 1)) (h)
h Fí (fauee) (GroupNorm() (h), emb)
h nonlinearity (h)
h nn nn.Dropout (rate- sell.seuff dropout) (h, deterministic-not train)
h = nn.Conv(
features,
kernel_size= (1, 3, 3),
strides= 1, 1),
ker cetptutiiintttaaeliie
) (h)
if C != features:
in nn. Dense (features) (h_in)
return (h h_in) / onp.sqrt (2)
class AttnLayer (nn :
"""Attention layer usable for self and cross attention.
attn_heads: int = 4
cnnpcc
def callallllaa
_____aall___________________ (self, *, q: jnp.ndarray, kv: ndarray)
C = q. [-ape head _im
// attr head
q
Dww.eeeeenerrllll (sself.tttn_heads head_dim) (q)
k = mn nn.
head_dim) (kv)
v Dm.DeeeeGeneal (( (tte. head_dim) (kv)
return nn. pee (g, k, v)
class AttnBlock Con.Module)
""*Attention block with skip connection.""
attn attn_type: str attn_heads. int 4
enn.compace
def _call (self, h_in: imp.ndarray)
B, _ H, W, C h_in.shape shape
h = GroupNorm () (h_in)
h0 h[:, re.reshape (B, W, C)
h1 = h[;, 1] reshape (B, W, C)
attn_ layer
if self.
'self';
h0 attn_layer (g=h0, kv=h0)
hl = atta_layer kv=h1)
elif self.attn_type 'cross':
h0 = (q=h00, kv=h1)
h1 (q=h_, kv=h0)
else:
raise Not.mpleeentedError (sel.
h jnp jpp. stack [hhO h1], axis=1)
16

---

Published as a conference paper at ICLR 2023
h h. reshape (B, H, w, -1)
h = mm..enseGeneral (c, axis= -1), kernel init=out, _ scale () (h)
return (h h in) / onp. (2)
class XUNetBlock (nn. Monule)
"" """X """X-UNet block. "**
features: int
use_attn: bool = False attn_heads: int 4
dropout: float = 0.0
@nn. compact
def __al__aaa_______________________ (self, x: jnp.ndarray, emb: jnp. adarray, *, train: bool):
h ResnetBlock (features=self. features dropout=self. dropoot)) emb, train=train
if self self.use.re
use attn: use_attn:
h = AttnBlock
hetd_ stt heads) (h)
h AttnBlock (attn_type= (arts
cross', attn_heads=sell h.ttn_headds) (h)
return
class ConditioniigProcessor (nn Module)
"""Process conditioning inputs into embeddings. ""*
emb_ch: int
nun_resolutions: int use_pos_emb: bool = True use_ ase_ref_pose__eeeeeeeeeeeees____ bool True
@nn.compact
def _call _ batch: dict[str, Idarray) cond_mask: jnp ndarray)
B, H, W, batch ['x')
Log togaa-to-nooiee-zat.o
embedding.
loganr jnp. clip
-20.0, 20.0)
logsnr = 2.0
(-1ogsnr / 2.0)) / onp.pi
logsnr_emb posenc_ddpm (logsnr, smb_ch=self.emb_cc max_time=1.0
logsnr_emb nn. Dense (self.emb_ch) emb_ch) (logsnr_emb)
logsnr_emb nnnsense (self. emb_ch) (nonlinearity (logsnz_emb))
Pose embeddings.
world_from_cal = v3d. (aarssforr((-bbcchh]hh11 cam_spec v3d. PinholeCamera (resolution= (H, W), K=batch['K']
(-batch['t")
rays v3d. Camera (spec- cam_spec,
forp:ooaaaaoolllorrrroooooooo. rays
pose_emb pospppos__ posenc ceric_nerf(ic (rays pos, min_deg=t max_deg=15) pose_emb_dee dir posenc_ner's (rays. dir, min_deg=0, max_ max_deg=8)
pose_emb inp inp.com concatenate (pose_emb_peg pose_emb_dir] axis=-1)
Enable classifier-fre guidance over poses.
D = pose_emb shape[-1]
assert cand_mask.shage None, None, None, None] cond_mast pose_emb jnp. where mask, pose_emb, jnp. zeros like (pose_emb)
(3,)
17

---

Published as a conference paper at ICLR 2023
Learnable position embeddings over (H, W) of frames (optional).
if self uue__uueppp___ emb:
pos_emb self. param
'pos_emb', nn nn. im.iiitializers.oo normall (stddev=1.0 / onp.sqrt (D)),
(H, W, D), cose_emb.dtype
pose_emb += pos_emb [None, None]
Binary embedding to let the model distinguish frames (optional)
if celt.uus_rre_pose___eee
first_emb self
"ref_pose_emb fim__firs'', nn in.inittializes.noo (stddev=1.0 / onp.sqrt (D) (D,), pose_emb, dtype, [None, None, None, None]
other_end self.param,
'ref Lmb__m____________ pooss_embbother"
nn. am.initializers normal (D,), pose_emb pose_emb.dtype,
normal (stddev=1.0 / onp. (D)
[None, None, None, None]
pose_emb += h www.atenateeeeeeeccee [first_emb, other_embl, axis=1)
Downsample ray embeddings for each UNet resolution.
pose_ []
for i_level in range (self wtt..mmmmrsooution........
pose_embe cembs.append.
nn. Convo
features=self. htt.:/eeellleeeeehheeeeeeeee
kernel_size= (1, 3, 3),
strides= (1, 2*xi_ 2*xi_ (pose_emb)
return logsnr_emb, pose_embs
class XUNet (nn. Module)
"*rour proposed XUNet architecture.
ch: int = 256
ch_mult: [uple[int] (1, 2, 2, 4)
emb_ch: int 1024
nun_res_blocks: int = 3
attn_resolutions: tuple [int] (8, 16, 32)
attn_heades int = 4
dropout: float 0.1
use_pos_emb: bool = True use_ref_pose_ee bool True
@nn.compact
def __aaall_ self,
18

---

Published as a conference paper at ICLR 2023
batch: dict Jap.ndarray]
cond mask: jnp ndarray, train: bool,
_ - batch [ x'x'] "'x''.saap shape
nun_resolutions len (self. (a..il.hh_mltt
logsnr_emb, pose_embs ConditioningProcessoor
emb_ch=self.embl emb_ch,
num_reesolatiiossssunnrrreed www.eeooolloosooooo
use_pos_emb=self.uu
use_ref_pose__nte
) (batch, cond_mask)
del cond_mask
h = inp.stack ( (baach['x')) batch['z']], }], axis=1)
h nn.Conv (self.ch, kerrel_size= (1, 3, 3), strides= (1, 1, 1)) (h)
Downsampling.
hs = [h]
for i_level self, ch_mult:
emb = logsnr_emb[..
None, None, :] + pose_embs [i [_veel]]
for in range (self. wtw.fffuuuussseeeeee num_res_blocks)
ase_attia s.shape in self.attn. ata.faatttrreslutiiito h XONetBlock
features=self.ch self. [aa__mull[[____ [ilvei__eeel,
tropout=seef..dropou
atth_heads=self..atthheee
use_attn-use_attn,
(h, emb, train-train) hs append
if i_level != num_resolutions
emb = logsnr_emb ...., None, None, :] + [es__embs[i_lever + 11
ResnetBlock (dropout=sell .wl.foddoou,, resample= 'down'' ')
h, emb, train-train
hs append (h)
Mi Middle. [..., [.....
emb = logsnr_emb
None, None, :] + pose_embs [ [-1]
use_attn h. shape [2] aelf..att_resolutt.oo
h = XUNetBlock(
features=self.ch self. ch
mul.chhmmliiii mult ievel
dropout=seef.droppaue
atth_heeads-sel..attn....
uss_attn-use_ath,
) (h, emb, train-train)
Upsampling.
for i_level in reversed (range (num_resolutiions)]
emb = logsnr_emb(...
None, None, :] pose_embs [i level]
for in range num wwwelffnuuure__lllleeeooo blocks 1): use ast_ hs[-1] shape [2] in self.attn_resolution
h = jnp. concatenate ( [h, hs hs.pop ()), axis=-1)
XUNetBlock
19

---

Published as a conference paper at ICLR 2023
features=self.ch self.ch, mwll[ii_leeelee dropoat-
attn http:/sellaattthhheeeeeeeee use_attn=use_atn,,
(h, emb, train=train)
if i_level 0:
emb logsnr_emb ...., None, None, :] + pose_embs| [i_level]
ResnetBlock (dropout=self. dropout,, resample='un 'pp') h, emb, train=train
End. assert not hs h = monlinearity (GroupNorm() (h))
return Conv
C, kernel_size= (1, 3, 3),
strides= 1, 1),
pernel_init-on init scale ())
) (h) [i, 1]
20

---

Published as a conference paper at ICLR 2023
7
HYPERPARAMETERS
We now detail hyperparameter choices across our experiments. These include choices specific to the neural architecture, the training procedure, and also choices only relevant during inference.
7.1 NEURAL ARCHITECTURE
For our neural architecture, our main experiments use ch=256 (~471M params), and we also experiment with ch=448 (~1.3B params) in Section One of early findings that we kept throughout all experiments in the paper is that ch_multt (1, 2, 2, 4) (i.e., setting the lowest UNet resolution to 8x8) was sufficient to achieve good sample quality, wheras most prior work includes UNet resolutions up to 4x4. We sweeped over the rest of the hyperparameters with the input and target views downsampled from their original 128x 128 resolution the 32x32, selecting values that led to the best qualitative improvements. These values are the default values present in the code we provide for the XUNet module in Section We generally find that the best hyperparameter choices at low-resolution experiments transfer well when applied to the higher resolutions, and thus recommend this strategy for cheaper and more practical hyperparameter tuning.
For the 11.3 parameter models, we tried increasing the number of parameters of our proposed UNet architecture through several different ways: increasing the number of blocks per resolution, the number of attention heads, the number of cross-attention layers per block, and the base number of hidden channels. Among all of these, we only found the last to provide noticeably better sample quality. We thus run 3D consistency scoring for models scaled this way, with channel sizes per UNet resolution of 448 x [1, 2, 2, instead of 256 x [1, 2, 2, 4] (we could not ch=512 in TPUv4 memory without model parallelism). On cars, find that the smaller model reported in the of the paper is comparable in quality to the ~ 1.3B parameter model, though on chairs, we do observe significant improvement on 3D consistency, both visually and quantitatively (see T)
7.2 TRAINING
Following Equation 3 the paper, our neural network attempts to model the noise added to a real image in order to undo it given the noisy image. Other parameterizations predicting æ directly rather than predicting e, though we did not sweep over these choices. For our noise schedule, we cosine-shaped log signal to noise ratio that monotonically decreases from 20 to -20. It can be implemented in JAX as follows:
possible, c.g.,
def seaanr_schedule_com
(t, *,
togsnr_max=20
b onp. aretan (onp. exp (-.5 loogsrr_maal a = oww.oraat....................... arctan (onp. exp (-.5 logsnr logsnx_min)) min) b
return -2. inp. log (jnp.
+ b))
Additionally:
We use a learning rate with peak value 0.0001, using linear warmup for the first 10 million exam- ples (where one batch has bat ch__ch______ size examples), following Karras et al.) (2022).
We use a global batch size of 128.
We train each batch element an unconditional example 10% of the time enable classifier-fres
guidance. This is done by overriding the conditioning frame to be at the maximum noise level. We note other options are possible (c.g., zeroing-out the conditioning frame). but we chose the option which is most compatible with our neural architecture.
We use the Adam optimizer (Kingma Bal 20114) with 31 0.9 and 8g 0.99.
We use EMA decay for the model parameters, with a half life of 500K examples (where one batch has batch_ss size examples) following Karras et al. (2022).
7.3 SAMPLING
While Ho et al. (000)) note that their ancestral sampler can be used with any variances between 3, (variances of 21, 2, æ)) and 3, (variances of the underlying SDE), we simply use the former those correspond their ancestral sampler. We do not observe this choice have significant
21

---

Published as a conference paper at ICLR 2023
effect sample quality. All our samples are generated with 256 denoising steps. As is standard in the literature, we also clip each predicted æ at each denoising step to the normalized range of the images [-1.1].
Classifier-free guidance. Our models use classifier-free guidance (Ho Salimans 2021), as we find that small guidance weights help encourage 3D consistency further. All our models were trained unconditionally with probability of 10% for each minibatch element. For unconditional examples, zero out the (positionally encoded) pose and replace the clean frame with standard Gaussian noise (leveraging our weight-sharing architecture, see Section 2.3). We swept over various guidance weights, and simply picked those where 3D inconsistency was qualitatively least apparent pled videos. For SRN cars, we use a guidance weight of 3.0, while for SRN chairs we use a weight of 2.0.
7.4 3D CONSISTENCY SCORING
Note that traditional NeRFs (Mildenhall et al. 2020) can be 3D inconsistent the model allows for view-dependent radiance. We thus employ simpler and faster to train version based on instant- NGP (Müller et al. 2022) without view dependent components, and with additional distortion and orientation loss terms for improved convergence (Barron et al. 2022, Verbin et al.] 2022). Note that the specific implementation details of the neural field method chosen will affect the metrics considerably, so it is utmost importance to apply the same method and hyperparameters for the neural fields make 3D consistency scores comparable across models.
We design the neural fields as simple Multi-Layer Perceptrons (MLPs) of hidden size 64 and no skip connections. The density MLP has one hidden layer, while the MLP that predicts the color components uses 2 hidden layers. We use learning rate of 0.01 linearly decayed to 0.001 for the first 100 steps. We the Adam optimizer with weight decay exceeding 1.0. We only apply 1000 training steps for each scene and do not optimize camera poses. On each training step, we simply optimize over all available pixels, rather than sampling a random subset of pixels from all the training views.
0.1 and clip gradients of norm
To render the neural fields after training, we set near and far bounds t, where 7 min: max are the minimum and maximum distances from the camera positions to the origin (center of each object) in the corresponding dataset. All renders post-training are also performed with differentiable volume rendering.
3r .,
22


Reviews:
[{"review_id": "hvv-h5TYBw", "reviewer": "Reviewer_Zv2q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an approach for rendering novel views from a single image. The approach is based on a pose-conditional image-to-image diffusion model. The paper shows the proposed approach can be generalized to the shapes that are not seen during the test time. The results of the proposed approach are superior then the results optained by prior art.", "summary_of_the_review": "The paper introduced an interesting approach for the view synthesis of an object using the diffusion model. It is also interesting to see that the additional random view images would help to generate view-consistent images. However, the paper requires significant revision in the exposition of the proposed approach to be more clearly to be understood. ", "strength_and_weaknesses": "**Strengths**\n1. The paper introduces a new approach for the novel view synthesis using the diffusion model. The generated images are plausible and consistent over the various viewpoints. The approach shows compelling results on the ShapeNet dataset.\n2. The approach introduces an interesting approach for rendering novel views from a single image. The paper explains the proposed idea in detail.\n3. The authors proposed a new approach to measure view consistency. By applying NeRF with the viewing direction, the better reconstruction of the radiance field indicates that the input images are more consistent across viewing directions, which is a reasonable and good attempt.\n\n**Weakness**\n1. The paper exposition requires significant improvement. The paper does not read well. For instance, the caption Figure 3 states, \"There are two main components to our sampling procedure 1) the autoregressive generation of multiple frames, and 2) the denoising process to generate each frame\". However, it is unclear which step indicates which section in the main paper, and there is no description of how the two steps can be utilized for the training. It is also confusing what the subfigures for Step 1 and Step 2 indicate. I think step 2 explains Eq 5 and Eq 6 since the input of the denoising model is the random viewpoint of the clean image, and I think Sec 2.1 corresponds to Figure 2. Then, what is Step 1 indicate? What are dice in the figure mean? Why does the pose of the dices varies while the poses of the images next to the dices are not changing?\n2. In addition, the paper requires an overview figure that can clearly show the flow of the training and testing phases. The figure should be consistent with the sentence in the abstract, \"a pose conditional image-to-image diffusion model, which takes a source view and its pose as inputs, and generates a novel view for a target pose as output\". The reader would expect the pipeline that takes the input image with its pose and target pose as the condition and the network to produce the novel view. The current figures need to explain how the trained network can be used to generate target view images. Instead, Figures 2 and 3 show additional clean images required for the training, and in Figure 4, the network does not require camera poses, but the network only requires noise and clean image pairs.\n3. Please build a connection between sections and figures. For instance, I think \\epsilon_\\theta in Sec 2.1 is described to be used in Eq. 6 in Sec 2.2, right? It would be good to mention how the notations and equations can be connected, resulting in the final equation.\n4. The limited dataset demonstrates the proposed approach, where the scenes are synthetic. Can the proposed method be applied to real images? For instance, I wonder if a bus on the road identified by a semantic segmentation mask can be view-interpolated. This would be tricky for the proposed approach since the synthetic images used for the training is biased toward specific camera intrinsic parameters. Given the image of the different camera intrinsic, not observed in the training scene, the view-interpolation would be failed. Another dataset that can be applied to the proposed approach is the DTU MVS dataset [Jensen et al. 2014]. The such dataset would be helpful to show the benefit of the proposed approach.\n5. Another limitation of the proposed approach is that approach is trained with the view-aligned 3D shapes. The input image and its viewing direction need to be an input of the proposed framework, limiting the applicability of the proposed approach. How can the approach be applied when the initial pose of the object is not provided? \n6. The technical contribution is not significant. It is an adaptation of the well-developed diffusion model [Salimans & Ho 2022, Ho et al. 2020, Kingma & Welling 2013] for the view of consistent image synthesis. The key modification is to provide random viewpoint images for denoising.\n7. At the time of ICLR submission, the project page of this paper was unveiled in the public domain for a few days. @AC: Would this violate the ICLR submission policy?\n8. I think the term \"3D consistent\" is misleading since the proposed approach is not utilizing 3D shapes. Instead, view consistency would be better terminology.\n9. The approach requires 256 denoising steps to generate a single image. It would take significant inference time. The runtime of the proposed approach should be compared with the baseline approaches. The current version of the paper lacks computational comparison.\n10. The proposed approach is actually performing worse than other baseline approaches, such as PixelNeRF and CodeNeRF, in terms of PSNR and SSIM. Although PixelNeRF performs better in terms of PSNR, the result in Figure 5 is much worse than the images shown in the PixelNeRF. It is suspicious the results are cherry-picked images. Authors claim that it is not necessary to achieve the top PSNR, but this is not very convincing since the view interpolation has the ground truth solution. The comparison with the GT solution is essential. \n11. Minor comment: CodeNeRF is not mentioned on page 6.\n12. In Figure 6, it is questionable that the black-silhouette images can be interpolated as the orange-colored chairs. Is this the ablation study tested with unseen images? I suspect the approach is overfitted to the training dataset.\n\n**Reference**\n\n[Jensen et al. 2014] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In CVPR, pages 406–413, 2014.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_yYBJ-Y4L5M", "reviewer": "Reviewer_o2Dj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a geometry-free method for novel view synthesis of objects utilizing probabilistic diffusion. Main contributions are a novel scheme for stochastic sampling of conditioning examples, a refined U-Net architecture for denoising, and a new metric for evaluating 3d consistency of novel view synthesis approaches. ", "summary_of_the_review": "In summary, I tend to vote for accepting the paper as it provides novel insight into geometry-free novel view synthesis (e.g. diffusion is able to capture 3D information based on 2D conditioning) and a list of smaller useful contributions, such as a novel metric for evaluating 3D consistency. It would be great if the authors could address the weaknesses, especially in experimental evaluations, as well as answer the following questions.\n\nQuestions and requests:\n- I hope that the method actually starts from pure noise when generating novel views with a trained model (inference). The paper does not make that clear and Figure 2 and Figure 4 suggest that it receives a noisy version of the target. If that is the case it would heavily dampen significance.\n- I would be interested in an analysis of cross attention pairs in the U-Net. The authors have the suspicion that it helps exploiting symmetries. A qualitative analysis of what-attends-to-what would increase insight.\n", "strength_and_weaknesses": "Strengths:\n- A metric for qualitatively evaluating 3D consistency of novel view synthesis is a great contribution to the field.\n- It is exciting to see that the proposed diffusion method is able to infer very consistent 3D structure only from view pairs.\n- The cross attention scheme in the presented X-UNet makes sense in the given setting.\n- The paper is very well written, well motivated and gives a useful overview about the area, structuring some related methods\n- The generated novel views have impressive visual quality\n- The individual contributions (stochastic sampling, diffusion and architecture changes) are evaluated in ablation studies.\n\nWeaknesses:\n- The authors present a novel metric for evaluating 3D consistency but put not enough effort on evaluating that metric itself. At least I would expect a comparison like in Table 2 which compares related methods under the new metric. Of course, it is expected that geometry-based methods outperform the presented method. It would be interesting to see how close 3DiM gets though. Also might be a good sanity check for the metric.\n- The authors provide some details about the metric in the supplemental materials, in which they mention that they use Instant-NGP instead of NeRF but provide no further information about for example efficiency. I wonder how different architectures compare in a speed/metric quality tradeoff.\n- Some details of the experimental setup are not clear to me (see below)\n- There are very few qualitative results in the paper and supplemental. Only two object categories of one dataset are shown and compared.\n- There is no dedicated related work section. Most important related work is cited and sufficiently introduced in the introduction but many works are missing.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RJAIP0Am3vv", "reviewer": "Reviewer_Am8T", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the use of diffusion models for novel view synthesis. The authors introduce a geometry-free image-to-image model, dubbed X-UNet, based on a new stochastic conditioning sampling algorithm. They compare their work with state-of-the-art baselines on SRN ShapeNet dataset, apparently, achieving better qualitative results. Regarding quantitative scores, the authors point out some limitations of standardised metrics and, consequently, propose a new evaluation protocol.", "summary_of_the_review": "The paper introduces a novel and well-principled method for novel view synthesis. The diagrams and overall architecture pipeline could be better presented. Qualitative results are better than state-of-the-art. However, quantitative scores fall short. Regarding this issue, the authors point out the reasons for this problem and propose a novel evaluation protocol. In summary, I believe the paper is worthy of publication conditioned to the improvement of presentation.", "strength_and_weaknesses": "Main Weaknesses:\n- Figs. 1, 2, 4 and 5 are not referred to in the text. Partially due to this fact, it is a little hard to follow the overall functioning of the methodology. Initially, Fig. 2 seems to show that, yet the \"green\" block appears \"inside\" Fig. 3 diagrams. In the latter, the little \"dice\" is not mentioned in the caption and its meaning is not completely clear. It is also not clear how the X-UNet architecture (Fig. 4), relates to the other diagrams in Fig. 2 and 3. It could be a good idea to who the overall architecture initially and clearly relate its block with the following diagrams in the subsequent figures.\n- The provided \"webpage\" in the supplementary material is not working properly. I also believe the supplementary should be in a separate pdf other than along with the main paper.\n\n\nMain Strengths:\n- The paper is well-written and based on very recent and updated literature.\n- The proposed method is sound and tackles a relevant problem with an original method.\n- The authors overcome the limitations of the standard evaluation metrics with the proposal o a new evaluation protocol.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IIy2Au_kWmR", "reviewer": "Reviewer_wGc3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose 3DiM, a diffusion model for novel view synthesis. The input to the model is a set of images with poses representing different views of the same object plus a target pose; the output of the model is a new image of the same object from the target pose. By applying the model repeatedly, the authors can generate many novel views.\n\nThe model consists of a U-Net-style diffusion image-to-image model. Notably, it only takes a single conditioning image as input, and in order to condition on multiple frames, the conditioning is varied through the diffusion process. The authors additionally modify the traditional U-Net used in diffusion modeling with a few architecture tweaks and show via an ablation that their modifications are an improvement over the default network structure.\n\nFinally, the authors propose a new metric for measuring the 3D consistency of their model. This metric is measured by training a NeRF on a subset of the model outputs and evaluating by comparing the NeRF outputs to the rest of the model outputs.", "summary_of_the_review": "Overall, the paper is above the acceptance threshold. It is well-written and reasoned, with a novel approach and interesting results which suggest a number of further research directions. However, as mentioned in the weaknesses section, one of the fundamental contributions of this work (stochastic conditioning) is underexplored; the need for it could be better justified and its behavior could be better evaluated and quantified. Given that this is fundamental to the approach and may significantly limit the significance of the work, I believe the paper is not *strongly* above the acceptance threshold.", "strength_and_weaknesses": "Strengths:\n- The model results are very promising. The rendered views are detailed and geometrically consistent with the input views. \n- The approach, details, and results are described clearly, including author reasoning for the choices they have made.\n\nWeaknesses:\n- The authors note that although one might expect to condition jointly on all available views, \"We found this solution to perform poorly ... We also find that, as we increase the maximum number of input frames k, the worse the sample quality becomes.\" This is counterintuitive. It seems likely that rather than being a fundamental aspect of the problem, this is a weakness of the model architecture or something else. The value of one of the core contributions – stochastic conditioning – relies on this observation, so it would be better if the authors spend more time to experiment with and explain why this is the case, rather than taking it for granted.\n- It seems like stochastic conditioning relies on having the number of diffusion steps be significantly greater than the number of input views. Additionally, it seems like the order in which the input views are used may have a significant impact on the resulting image and perhaps the resulting image quality or consistency. It would help if the authors investigate the impact of number of diffusion steps as well as impact of view ordering for stochastic conditioning.\n- Between the two weaknesses listed previously, it seems like stochastic conditioning could prove to be a poor choice and have limited significance. If you reduce the number of diffusion steps (as many contemporary works aim to do), stochastic conditioning could prove to perform poorly, or be incredibly sensitive to the order of views. Similarly, if stochastic conditioning is simply a workaround for a network architecture that cannot scale to multiple input views, stochastic conditioning will quickly become irrelevant once the network architecture is improved.\n- Relatively minor – The modified U-Net has several modifications; however, the ablations only compare *all* modifications against *no* modifications, without individually showing that the modifications are useful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster