NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer
Authors: Meng YOU
Zhiyu Zhu
Hui LIU
Junhui Hou
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion sampling
Novel view synthesis
Primary area: generative models
TL;DR: Our method adaptively modulates score functions of a pre-trained video diffusion to synthesize impressive novel views.

Abstract: By harnessing the potent generative capabilities of pre-trained large video diffusion models, we propose a new novel view synthesis paradigm that operates without the need for training. The proposed method adaptively modulates the diffusion sampling process with the given views to enable the creation of visually pleasing results from single or multiple views of static scenes or monocular videos of dynamic scenes. Specifically, built upon our theoretical modeling, we iteratively modulate the score function with the given scene priors represented with warped input views to control the video diffusion process. Moreover, by theoretically exploring the boundary of the estimation error, we achieve the modulation in an adaptive fashion according to the view pose and the number of diffusion steps. Extensive evaluations on both static and dynamic scenes substantiate the significant superiority of our method over state-of-the-art methods both quantitatively and qualitatively. The source code can be found on https://github.com/ZHU-Zhiyu/NVS_Solver.

Published as a conference paper at ICLR 2025
NVS-SOLVER: VIDEO DIFFUSION MODEL AS ZERO- SHOT NOVEL VIEW SYNTHESIZER
Meng You',, Zhiyu Zhu" Hui Liu & Junhui Hou' 'City University of Hong Kong, Saint Francis University (mengyou2, zhiyuzhin zhiyuzhu2-cJemy emc.myyy.ycc....mmmmmmmmmmmmmm edu. h2liuesfu. hk jh. gh.hou@cityu.en. hk
edu.. hk
(c)
Figure 1: Visual demonstrations of the proposed algorithm with input of (a) single-view, (b) monoc- ular video, and (c) multi-view (2 views). The middle row refers to the algorithm's 's inputs for each scenario. Please use Adobe Acrobat display these videos.
ABSTRACT
By harnessing the potent generative capabilities of pre-trained large video dif- fusion models, propose new novel view synthesis paradigm that operates without the need for training. The proposed method adaptively modulates the diffusion sampling process with the given views enable the creation of visu- ally pleasing results from single or multiple views of static scenes or monocular videos of dynamic scenes. Specifically, built upon theoretical modeling, we iteratively modulate the score function with the given scene priors represented with warped input views control the video diffusion process. Moreover, by theoretically exploring the boundary of the estimation error, we achieve the mod- ulation in an adaptive fashion according to the view pose and the number of diffusion steps. Extensive evaluations on both static and dynamic scenes sub- stantiate the significant superiority of our method over state-of-the-ar meth- ods both quantitatively and qualitatively. The source code can be found on https: / / thtthu..hh.... com/ ZHU 2hiyu/NVS_Solver.
INTRODUCTION
In the realm of computer vision and graphics, novel view synthesis (NVS) from limited visual data remains formidable challenge with profound implications across various domains, from
Equual Contribution.
"Corresponding Author: This project was supported in part by the NSFC Excellent Young Scientists Fund 624221 18, in part by the Hong Kong RGC under Grants 11219422 and 11219324, and in part by Hong Kong UGC under grants UGC/FDST 1/F02/22 and UGC/FDS1 1/R03/24.

---

Published as a conference paper at ICLR 2025
entertainment (Tewari et al., 2020; Jiang et al., 2024) to autonomous navigation (Adamkiewicz et al., 2022; Kwon et al., 2023) and beyond (Avidan & Shashua, 1997; Zhou et al., 2016; Riegler & Koltun, 2020; Mildenhall et al., 2021: Kerbl et al., 2023). However, addressing this challenge demands sophisticated method capable of extracting meaningful information from sparse visual inputs and synthesizing coherent representations of unseen viewpoints (Zou et al., 2024; Zhang et al., 2021). In this context, the emerging field of deep learning has witnessed remarkable strides, particularly with the advent of advanced generative models (Voleti et al., 2024; Chen al., 2023; Gu et al., 2023).
Recently, diffusion models (Ho et al., 2020; Song et al., 2020; 202 have garnered significant attention due to their exceptional ability to synthesize visual data. A prominent area of focus within this domain is video diffusion models (Blattmann et al., 2023; Khachatryan et al., 2023; Ho et al., 2022; Ni et al., 2023; Karras et al., 2023; Khachatryan et al., 2023; Wu et al., 2023), which have gained popularity for their remarkable video generation capabilities. In this paper, we explore the problem of NVS from single or multiple views of static scenes or monocular videos of dynamic scenes, leveraging the pre-trained large video diffusion model without additional training. Specifically, we theoretically formulate the -orieneet diffusion process as guided sampling, in which the intermediate diffusion results are modulated with the scene information from the given views. Moreover, we empirically and theoretically investigate the potential distribution of the error map to achieve adaptive modulation in the reverse diffusion process, with a reduced estimation error boundary.
In summary, the main contributions of this paper lie in:
we propose new training-free novel view synthesis paradigm by leveraging pre- pre-trained video diffusion models;
we theoretically formulate the process of adaptively utilizing the given scene information to control the video diffusion process; and
we demonstrate the remarkable performance of our paradigm under various scenarios.
RELATED WORK
Diffusion Model indicates kind of deep generative model (Sohl-Dickstein et al., 2015), which is inspired by non-equilibrium statistical physics by iteratively appending noise into image data and then reversely removing noise and transferring noise-free data distribution. (Ho al., 2020) proposed variance preserving (VP) diffusion denoising probabilistic model via progressively removing noise. (Song et al., 2021b; Song & Ermon, 2019; 2020: Song et al., 2021a) proposed a score-based image generation model by iteratively calculating the derivative of data distribution and utilizing the stochastic differential equation (SDE) (Anderson, 1982) or ordinary differential equation (ODE)-based solvers (Maoutsa et al., 2020: Song et al., 2021b) b) to reverse the noise-adding process and derive the clean image distribution. Inspired by the success of image diffusion models, many works attempted to build video diffusion model to directly achieve video generation prompted by text (Khachatryan et al., 2023; Wu et al., 2023) or a single image (Blattmann et al., 2023; Karras et al., 2023).
Diffusion sampling algorithm aims to speed up or control the diffusion process via regularizing or re-directing the reverse trajectory of diffusion models (Lu al., 2022a;b; Zheng et al., 2024; Zhang& & Chen, 2022; Wang et al., 2023b; Cao et al., 2024; Chung et al., 2023; 2022a;b). (Song et al., 2020) proposed denoising diffusion implicit models (DDIM) to accelerate the diffusion sampling by jumping to the clean image-space at each step. (Dhariwal & Nichol, 2021) utilized the classifier to guide the sampling process of diffusion model for controlling the results' categories. (Lu et al., 2022a;b; Zheng et al., 2024) proposed a series of fast ODE diffusion solvers given an analytic solution of ODE by its semi-linear nature. Moreover, the integration of non-lineat network-relatee parts was further approximated by its Taylor series. (Zhang & Chen, 2022) explored the huge variance of distribution shift and then decoupled an exponential variance component from the score estimation model, thus reducing the discretization error. (Chung et al., 2023) proposed regularize the intermediate derivative from the reconstruction process to achieve image restoration. (Wang et al., 2023b) decoupled the image restoration into range-null spaces and focused on the reconstruction of null space, which contains the degraded information.
Novel view synthesis (NVS) targets at generating images of scene from viewpoints not presented in the original data and has been a subject of extensive research in computer vision and graphics (Park et al., 2017; Choi et al., 2019; Tretschk al., 2021; You al., 2023; Avidan & Shashua, 1997;
2

---

Published as a conference paper at ICLR 2025
Riegler & Koltun, 2021; You & Hou, 2024). Early methods in this domain often relied geometric approaches, such as multi-view stereo reconstruction (Seitz et al., 2006; Jin et al., 2005) and structure- (Schonberger & Frahm,, 2016; Ōzyeşil et al., 2017) techniques, to synthesize novel viewpoints from multiple images captured from different angles. The advent of deep learning has revolutionized the field of NVS, enabling the synthesis of realistic images from pre-trained feature volumes. (Rombach et al., 2021; Ren & Wang, 2022) employed autoregressive Transformer to synthesize 3D scene from single image. Neural Radiance Fields (NeRF) (Mildenhall al., 2021; Pumarola et al., 2021; Barron et al., 2021; Martin-Brualla et al., 2021; Kosiorek et al., 2021) enabled stunningly detailed renders from 2D images through volumetric rendering. Additionally, differentiable rendering has allowed gradients of rendering outputs with respect to scene parameters, facilitating direct optimization of scene geometries, lighting, and materials. Recently, 3D Gaussian Splatting (Kerbl et al., 2023; Liu et al., 2024) presented an explicit representation of the scene.
While these methods have shown promising results, they often suffer from limitations such as dependence dense scene geometry, challenges in handling complex scene dynamics, and being hard to generalize (Kerbl et al., 2023). Moreover, they require significant computational resources and suffer from artifacts such disocclusions and view-dependent effects. More recently, (Charatan et al., 2024; Chen et al., 2025) introduced the use of feed-forward 3D Gaussians for novel view interpolation. (Wu et al., 2024b; Watson et al., 2023; Yu et al., 2023; Cai et al., 2023; Tseng et al., 2023; Chan et al., 2023; Sargent al., 2024) explored the integration of 3D reconstruction techniques with diffusion models.
In light of these challenges, our work builds upon the strengths of recent advancements in deep learning-basse NVS, with a focus on leveraging the robust zero-shot view synthesis capabilities of the video diffusion model. By harnessing latent representations derived from sparse or single-view inputs, our approach aims overcome the limitations of existing methods and produce high-quality novel views with improved realism.
3 PRELIMINARY OF DIFFUSION MODELS
We briefly introduce some preliminary knowledge of diffusion models (Sohl-Dicke Dickstinn et al., 2015),
which facilitates our subsequent analyses. We also refer readers to (Song & Ermon, 2019; Song et al., 2021b) for more details. Generally, the forward SDE process of the latent diffusion model can be formulated
dæ = f()ædd + g())dw,
(1)
where x is the noised latent state, I for the timestamp of diffusion, and the two sealar functions f((I) and (t) output drift and diffusion coefficients, indicating the variation of data and noise components during the diffusion process, respectively. Naturally, we have the following ODE solution:
1 dx = f()x 2
(x) Vo lz)( dt.
(2)
Through training a score model Sa 1) parameterized with 0 to approximate the V ((=)( as
C - So (xx t) Va (og])( with y(t) > O, we can calculate the clean image wo via utilizing the score function So(x, t) to calculate the data distribution gradient
(3)
1 da [(1)a -0 (a) dl.
Moreover, since the noise of the diffusion process is usually parameterized by i..dd Gaussian noises N(e,; (fII) with variance of a (t), the above diffusion process be calculated
1 dx f(t)æ= - 7 (x)
14
dt,
(5)
02(()
where b4 Xo,,,, t) is the estimated clean image from the noised image I, at step & by the denosing U-Net (c(;,)) .T Finally, considering the special case of variance exploding (VE) diffusion process (Song et al., 2021b) of the stable video diffusion (SVD) (Blattmann et al., 2023), Eq. (5) can be further simplified as
2 - Mt (t). a(t
da
(6)
For simplicity, here combine some parameterizations stemmed from EDM (Karras al., 2022) into (-).
3

---

Published as a conference paper at ICLR 2025
4 PROPOSED METHOD
Motivated by the powerful generative capability with realistic and consistent frames by large video diffusion models, aim to adapt the pre-trained video diffusion model to the task of NVS without any additional training, leading to a score modulation-basl based approach. Generally, our method warps the input image set to the target view and leverages these warped images prior information to guide the reverse sampling of a video diffusion process. This not only ensures the accurate generation of the warped regions but also facilitates the meaningful synthesis of unknown occluded areas, resulting in high-quality and visually consistent novel views with enhanced fidelity and coherence. Specifically, we first theoretically reformulate NVS-oriented reverse diffusion sampling by modulating the score function with the given views (Sec. 4.1). Based on theoretically exploring the boundary of the diffusion estimation and depth- based warping errors, we propose to adaptively modulate the prior of the given view into the diffusion process to reduce the potential boundary of the estimation error (Sec. 4.2).
In the following, we take the single view-based NVS to illustrate our method, as summarized in Algorithm 1, which involves from a given view Xo.ps with pose Po. reconstructing N novel views at target poses {PI...., .P. for scenarios involving NVS from multiple views, as well as from monocular videos, as explained in Sec. 4.3. Also, we consider the pre-trained image-to-video diffusion model SVD (Blattmann et al., 2023).
PN-1). denoted as Xo.p.. Note that our method is suitable
Notations. Let X, RBxW be the spatial-temporal latent of a set of N images/views at the t-th diffusion step, Pa E E the pose of the i-th image (0 S i < N - 1), Xe (X,, t) a U-Net denoising X, estimate the clean image set μl, E IR Hxx Xp. E R typical latent step 4, time i, and spatial pose pi, and pt.p.
20 (X,, I), indicating a intermediate estimation of clean image Xo.p.-
4.1 SCENE PRIOR MODULATED REVERSE DIFFUSION SAMPLING
Based on the formulation of the image reverse diffusion sampling process in Eq. (6), we have Xt.p. Xg (X,,t) t)
dX
da
(1)
According to the intensity function (McMillan & Bishop, 1995), we can formulate the relationship between Xo.po and Xo.p; through Taylor expansion:
Xo.p. = I(po)= + dI(p) Ap+
(8)
dp
where I(.) the intensity function, Ap = P: Pa is the pose variation, and O²(Ap) is the high order Taylor expansions. Based the depth-driven image-warping operation, we further have
I(po) W (Xo.po u:)
KDApK Uno
-
where W(, .) denotes the image warping function; U. and u, are the pixel locations of the views poses Po and p:. respectively; D is the depth map; K is the camera intrinsic matrix. Moreover, since the ground-truth depth D is usually not available in practice, we can also use the estimated depth map D by an off-the-shelf depth estimation method to calculate the estimated pixel location, i.e., u = KDApK- -u. By substituting Eq. (9) with an estimated depth map to Eq. (8), we then have aW (Xu.pn> (z). Iu Au dI(() Ap + (3(p)), Au KADApK uns
Xo.p, W (Xo,po> ū
dp
&a
(10)
where AD := D D. Xo.p. refers to the warped view from Po to pose P, through D, and Er is a residual term that contains high-order Taylor expansion series and warping error.
Based on the above analysis that the rendered novel view at pose pa is highly correlated with Xa,. and pt.p, an effective score function for NVS should take advantage of them. Then, we formulate the score function of NVS-oriented reverse diffusion sampling, which is modulated with the given scene information as
Pt.p. = arg min || pt 112 + X(1, p:)ll Ne
(11)
/
4

---

Published as a conference paper at ICLR 2025
Algorithm Zero-shot NVS from Single Images
1: Input: given view (Xpo): ). estimated depth map Dp targel view poses {Po. ... PN- J. and diffusion U-Net Xe (-)..
2: Derive the warping component po Ko.. corresponding poses and depth maps by Eq. (10).
Xo.px- J. using the given views,
a Preparing diffusion guidance images.
3: Initialize XT ~ (0, III)
4: For T..,, do
Video diffusion reverse sampling steps.
5: Diffusion network forward μ4: - (X,,,,)).
For = 0, 1 do
Calculate weights X(t. p) via Eq. (17) and Pt.p via Eq. (12).
8:
If Directly guided sampling then Reverse Xt,p to -I via Eq. (13).
10:
End If
11:
End for
12: If Posterior sampling then
13:
Derive the optimized X via Eq. (14) and apply a standard reverse step as Eq. (7) to get the X,-1-
14:
End If
15:
End for
16:
Return Reconstructed sequence Xo-
where A(t, p:) > ( balances the two terms. It is obvious the closed-fome form solution of Eq. (11) is
1
A(t,Pi)
Ptp. 1 1 p.). P.p.. + 1 A(t,tp Xo.p.
(12)
With the optimized clean image expectation term jt..p. we further utilize two ways to guide the reverse sampling of the pre-trained SVD: (1) Directly Guided Sampling, which is fast with limited quality; and (2) Posterior Sampling, which is slow but more effective. Specifically, the former directly replaces the μt.p in with pt..p. as
Xt.p Pr... a (1)
dX Pp
do(().
(13)
While the latter embeds the knowledge of μt into X, via the backward gradient as
X = X, <Vx, I|+ 7.el2.
(14)
where K > 0 controls the updating rate, which is empirically set to 2e Vo(()( Vx, - Rella computes the back- pack-propagated gradient on X,; X stands the optimized noised latent X,. Here, we also normalize the gradient Vx, I| f to stabilize the updating process.
4.2 ADAPTIVE DETERMINATION OF A(t, p.)
Although we have reformulated the NVS-oriented diffusion process in the preceding section, the value of X(t. P.) Eq. (11) and Eq. (12), which is crucial to the quality of syn- thesized views, has to appropriately determined. Here, we theoretically ex- plore the Formulation of A(t, p.) via analyzing the upper boundary of the estimation error of ju.pp.
08
0.40
0.00
027 0.3 EST 055 0.3
Figure 2: Experimental observations of the relationship (a) between the diffusion estimation error &n and the noise level a, and (b) between the error of warped image E and the changed amount of view pose A
Let pt be the ideal value for the score function estimation in the NVS-
oriented diffusion process, i.e. the ground-truth view at P:- Based the formulation of Pt.p. in
5

---

Published as a conference paper at ICLR 2025
Eq. (12) and the triangle inequality, we have
Pt.p. p., 1 1 1+ A(t.pp..) Hcpp. pi + X(t, p4)
Xuu.. - -i.p.
(15)
1 X(t, p.)
Thus, we propose to minimize the estimation emror upper bound to obtain an appropriate and adaptive A(t, i..,
A(t, arg min 1 ((-P..) 1+ A(t, p.) Ex (x,- 1+ X(t,
X(t, (x-P(x)(Ee) Po))
(16)
where 81 0 and the last regularization term prevents the weights from being overfitting the empirically estimated errors. In the following, we will provide the explicit formulations of the two error terms ED |pp... - -i. ll2 and & = [o.... - Pi.p. lla 2 based on theoretical analyses and experimental observations.
Diffusion Estimation Error Ep. This enror is caused due to the fact that the diffusion model cannot perfectly estimate P..p, from the given noised latent X,, Recent works (Zhang & Chen, 2022; Zheng al., 2024) indicate that the derivative (X,,,) of SDE-based diffusion models varies intensely when (t) is huge. Accordingly, large values of would potentially lead to large errors. Moreover, we experimentally investigated the correlation of ||Epll ll with (t). As demonstrated in in Fig. 2 (a). ||E l2 gradually decreases with the diffusion reverse process (decreasing of (t)). Thus, we empirically formulate [[ED]]2 = U2 where < U/2
Intensity Truncation Error Ep. This error is mainly induced by the truncation of the Taylor series (here Pi.p, is the same with X), as shown in Eq. (10). If omitting the high-order terms, we have
Ep @W(Xo. ū.) Au dTíp) Ap. According the definition of Au in Eq. (10), we also have dp
on
|| ll2 S 1 Aplp 2- Together with the experimental observation of the relationship between ||Epll2 and Apll2 Fig. 2 (b), we empirically formulate ||Ep||2 v3l Apl12-
Finally, with the explicit formulations of the two error terms, we can rewrite Eq. (16) as
U2a(() p()glllall en log (A(,, ((),,
A(t, pa)
arg min X(Ip.)) 1+ X(t.pa)
(17)
1 1+ X(t,tpp) p:)
whose closed-form solution is (we have visualized Ā in Appendix D) Ā(t, p.) = (2 + Q) + /Q2 + 401
Q = 23||Ap112 - U20
(18)
201
Claim of Novelty. Our method presents a theoretical analysis that links NVS with diffusion processes. Examining the relationships between different views and assessing their potential error distributions enable adaptive modulation of the diffusion score function, effectively minimizing eтrors from both the warping operator and the diffusion process. This makes our method uniquely suited to addressing NVS challenges. In contrast, traditional guided sampling algorithms typically rely on image degradation models, such as blurring and noise, to guide the diffusion process, setting our approach apart. Moreover, we have theoretically discussed that the proposed method regularizes the diffusion trajectories towards a more accurate direction in Appendix A.I. We have also proved the proposed method (DGS) can safely regularize the samples on the data manifolds in Appendix A.2. We also experimentally compare with inpainting- based methods in Appendix G.
4.3
NVS FROM MULTIVIEWS AND MONOCULAR VIDEOS
We have illustrated the single viewd NVS via the proposed method in Algorithm 1, which can be further modified for NVS from multiple views or monocular videos. Specifically, given multiple views of a static scene and target poses, we first estimate the depth map of each of the given views and derive the warped view at target poses by warping the nearest given views to each target pose. For monocular video, we warp each frame of the video sequence the corresponding target pose with the same timestamp (See the Appendix E for the detailed warp strategy pipeline). Then, we sample the novel view via the proposed method as Lines 3-16 of Algorithm
"There is also inherent discretization error as investigated by (Lu et al., 2022a;b; Zhang & Chen, 2022). However, we could reduce such error by enlarging the number of reversing steps.

---

Published as a conference paper at ICLR 2025
Table Quantitative comparison of different methods on view synthesis, where we measure the FID and the егrоr of view pose. We name our method with Ours (DGS) or (Post) for utilizing directly guided sampling Eq. (13) or posterior sampling in Eq. (14), respectively. "** indicates incomplete evaluation, where the corresponding metric cannot output meaningful value on some sequences by the method due to the failure of pose estimation. "--" denotes that the method cannot work on the condition or the metrics cannot be calculated. For all metrics, the lower, the betler.
Methods
Overlitiigg
Single view
Multi Multi-view
FID
ATE
RPET
RPE.R
FII
ATE
RPET
RPE.R
Sparse Caussian (Xiong et al., 2023)
369.19
324.60
Sparse Nerf (Wang et 2023a)
180.26
6.129
1.711
1.804
Text2Nerf (Zhang al., 2024)
187.05
2.223*
0.718
0.107
Photocomsistent-NVS (Yu ct al., 2023)
x
193.87
7.64
1.19
1.45
3D-aware (Kiang
2023)
X
217.19
2.836
1.258
1.662
211.25
2.159*
5.679*
2.119
MotionCtrl (Wang et al., 2024)
X
179.24
3,851
0.705
0.835
154.27
37.68
19.61
1.646
Ours (DGS)
X
166.50
4.533
0.810
0.742
124.31
22.00
17.34
1.338
Ours (Post)
165.12
0.767
0.156
0.170
126.44
4.052
2.030
EXPERIMENT
Datasets. For single view-based NVS, we employed a total of nine scenes, with six scenes from the Tanks and Temples dataset (Knapitsch et al., 2017), containing both outdoor and indoor environments. The other three additional scenes are randomly chosen from the Internet. For multiview-bbssed NVS, we used three scenes from the Tanks and Temples dataset (Knapitsch et al., 2017), including both outdoor and indoor settings, well six scenes from the DTU dataset (Jensen et al., 2014), which feature indoor objects. For each scene, we selected two images as input to perform view interpolation. For monocular video-based NVS, we downloaded nine videos from YouTube, each comprising 25 frames and capturing complex scenes in both urban and natural settings.
Implementation Details. We conducted all the experiments with PyTorch using a single NVIDIA GeForce RTX A6000 GPU-48G. We adopted the point-based warping (Somraj, 2020) to achieve (?) to estimate the maps of the input views (See the Appendix for more results with different depth estimation methods). We simultaneously rendered 24 novel views and set the reverse steps as 100 for high-quality sample generation. For the implementation of Eq. (11), since applying directly weighted sum usually results in blurry, we ordered the feature pixels by the Mr.p, -o. ll2 and take the ratio of A(t.p.)
W(-) and employed Depth Anything
I+M(1.p) smaller pixels from Xo.po. and the others from pt,, (le- 7,,7,,
to modulate pt. We choose the values (v1, U2 03) as (le-5, 9,-1,,5 1,5=",) 2),
ne-1,,0 1e 2), and
(le -6 1.75, 3e- for single, sparse, dynamic scene view synthesis.
Metries. We utilized four metrics measure the reconstruction performance, i.e., Fréchet Inception Distance (FID) (Heusel et al., 2017) evaluating the quality and diversity of synthesized views; Absolute trajectory error (ATE) (Goel al., 1999) measuring the difference between the estimated trajectory of camera or robot and the ground truth trajectory; Relative pose error (RPE) (Goel et al., 1999) measuring the drift, where we separately calculated the transition and rotation as RPE-T and RPE-R, respectively. We utilize Particle-SFM (Zhao et al., 2022) to estimate the camera trajectory and assess the pose metrics. Since current depth estimation algorithms struggle derive absolute depth from single view or monocular video, resulting in a scale gap between the synthesized and ground truth images, we only compare the paired metrics in Appendix such as LPIPS.
5.1 RESULTS OF NVS FROM SINGLE OR MULTIPLE VIEWS OF STATIC SCENES
Fig. visualizes synthesized novel views of two scenes by different methods from single views, where we can see that our method can consistently generate high-qualing novel views with visually pleasing geometry and textures. The quantitative results listed Table also validate the significant superiority of our method over -fite--ff-hh-art methods. Although MontionCtrl (Wang et al.. 2024) can generate high-quality images reflected by the FID value, its results have significantly large view pose errors reflected by the higher ATE and RPE values. For Text2Nerf (Zhang al., 2024), only 7 out of the total 9 sequences can be calculated metrics, which may be induced by the inherent errors of the learned geometric structure.
Fig. 4 shows the synthesized views by different methods from two given views, where it can be seen that our method outperforms state-of-the-ant methods by clearer views, especially for the 2nd and 3rd
7

---

Published as a conference paper at ICLR 2025
Figure 3: Visual comparison of single view-based NVS results by (a) Tex12Nerf (Zhang et al., 2024), (b) 3D-aware (Xiang et al., 2023), (c) MotionCtrl (Wang et al., 2024), (d) Ours (Post). The middle view of each scene highlighted with the red rectangle refers to the input view. Here, we only show the results of the best two of all compared methods. We also refer reviewers to the Appendix B and video demo contained in the supplementary file for more impressive results and comparisons.
60
Figure 4: (a) The two input views of each scene highlighted with the red rectangle. Visual results of seltiiiiew-bbsed NVS by (b) 3D-aware (Xiang et al., 2023), (c) MotionCtrl (Wang et al., 2024), (d) Ours (Post).
scenes. In addition, it is worth noting that the lower ATE of 3D-aware (Xiang al., 2023) due to the incomplete evaluation (See the Appendix C for more results).
Note that our method can also achieve 360 NVS through iterative execution of the proposed algorithm. Fig. 5 shows the synthesized 360 NVS from both single-view and multi-view inputs, demonstrating the capability of the proposed method effectively handle the NVS task (See the Appendix H for the detailed strategy).
5.2 RESULTS OF NVS FROM MONOCULAR VIDEOS OF DYNAMIC SCENES
The non-generative Gaussian-based methods Deformable-Gaussian (Yang et al., 2024c) and 4D-Gaussian (Wu et al., 2024a) usually can- not handle the marginal area, as illustrated in 2nd 4th and 6th columns of Fig. 6 (b) and (c). Although the SVD-based method MotionCtrl can generate the boundary region as shown in Fig. 6 (e), the synthesized views are blurry and the poses of generated samples cannot follow the prompt, especially on the 3rd sample. However, our method consistently generates high-quality novel views with lower pose errors, indicating its strong potential.
Table 2: Quantitative comparison of different meth- ods NVS from monocular videos of dynamic scenes. For all metrics, the lower, the better.
Muhods
Train
FID
ATE
RPE-T
RPC-R
Defor Belle-bbbh--innnee (Yang
0615"
4D-Gaussian (Wu all. 2024s)
74.34
2.087
0.625
0.625
3D-oware (Xiang et al., 2023)
159.03
3,100
1.343
1,368
(Nlng 212,
Ours (DGS)
x
37.973
2,256
0.601
0.446
Ours (Post)
39.86
2.308
0.725
0.500
8

---

Published as a conference paper at ICLR 2025
Figure 5: The input views of each scene highlighted with the red rectangle. Visual results of synthesized 360 NVS from (a) single view and (b) multi-view input.
Figure 6: Visual comparison on dynamic scene view synthesis of (a) input frames in the corresponding time of generated images, (b) Deformable- Gaalssann (Yang et al., 2024c), (c) 4D-Gaussian (Wu et al., 2024a), (d) 3D-aware (Xiang et al., 2023), (e) MotionCtrl (Wang et al., 2024), (f) Ours (Post).
5.3 ABLATION STUDY
Reverse Inference Steps. The quantitative results in Table 3 show that the synthesized image quality of our method does not improve intensely with the number of inference steps increasing. However, the pose error decreases significantly, indicating the necessity of a sufficient number of inference steps for accurately rendering novel views.
Posterior Sampling vs. Directly Guided Sampling. The quantitative results listed in Tables 1 and 2 show that both the Ours (DGS) and Ours (Post) can generate high-quality images with comparable FID. However, the view pose of Ours (Post) much more accurate than Ours (DGS), which is also visually verified by the results in Figs. 7 (c) and (d). Besides, Ours (DGS) takes minutes render 25 views, while Ours (Post) uses hour.
Table 3: Quantitative comparison of different numbers of inference steps. For all metrics, the lower, the better.
Inference step
FID
ATE
RPE-T
RPE-R
175.432
5,317
0.691
0.847
50
168.938
2.275
0.428
0.402
100
165.12
0.767
0.156
0.170
Table 4: Quantitative comparison of ablating updating rate k and the normalization schemes on Ours (Post). For all metrics, the lower,
Effectiveness of Back- propppaopanntion in Posterior Sampling. We performed ablation studies on the updating rate к and the normalization schemes in Eq. (14). The experimental results shown in Tab. 4 empirically guarantee that the back pack-propagation brings the latent space closer to the inherent oowdiimensional manifold structure during each update step, further validating the performance of the proposed acore-modulation schemes.
the better.
Normaliation
RPS-T
Y
166.12
0.277
0.156
0.170
0,5
Y
268.17
1.92
0.315
0,399

---

Published as a conference paper at ICLR 2025
(0)
[b)
tc
(d)
Figure 7: Ablation studies of the proposed weight strategy of X(t. p() and embedding strategy. Visual comparison of (a) warped input image Xo.p. (b) results without our proposed weight strategy, (c) results of Ours (DGS), and (d) results of Ours (Post).
Effectiveness of Adaptive A(t, p.). Here, we illustrate the effectiveness of adaptively adjusting A(t, p.) via setting it to +00, i..,, Ht.p. Xu.p.- Visual comparison re- sults in Fig. 7 indicates that the proposed method significantly correct the warping errors (1st, 2nd and 4th samples in Fig. 7) and non-Lambert reflection (3rd sample in Fig. 7) as indicated in E9 of Eq. (10).
Different Trajectories. We detailedly visu- alize the result the proposed method on different trajectories. We apply transitions in eight different directions. The results shown in Fig. 8. The proposed method can consistently render high-quality novel views.
Figure 8: NVS results of Ours (Post) with different trajectories, where the given view is bounded by blue circle. We draw the pose at the right bottom corner of each sub-figure.
CONCLUSION & DISCUSSION
We have presented training-free novel view synthesis paradigm. The proposed method achieves remarkable performance compared with state-of-the-ar methods. The advantages are credited to the powerful generative capacity of the pre-trained large stable video diffusion model and our elegant designs of the adaptive modulation of the diffusion score function through comprehensive theoretical and empirical analyses.
Although our method takes longer than existing methods, the promising generative capacity inherent in the large pre-trained diffusion model may attract improvements the future. Since the proposed method can synthesize views more accurate poses, we believe it may be a potential solution for the distillation of pose controllable video diffusion model. In the age of generative intelligence, the authors expect the proposed algorithm would inspire future work to unify computer graphics pipelines with generative models. Finally, the authors sincerely appreciate Stability AI for the open-sourced SVD (Blattmann et al., 2023).
10

---

Published as a conference paper at ICLR 2025
ETHICS & REPRODUCIBILITY STATEMENTS
The proposed method is specifically designed for novel view synthesis through video diffusion. As such, no additional information regarding human subjects potentially harmful insights is introduced during the process. This approach prioritizes privacy and ethical considerations by relying solely on the data available from the input images videos, without requiring any sensitive external information. Furthermore, all of our experiments are conducted a training- frrenn manner, which not only simplifies implementation but also ensures reproducibility. More importanty, the source code Solver.
can be found on the webpage: https: / //t///gtttt tuu........ com/ ZHU
REFERENCES
Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jean- nette Bohg, and Mac Schwager. Vision-only robot navigation in a neural radiance world. IEFE Robotics and Automation Letters, 7())4606-4611, 2022.
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):31-326, 1982.
Shai Avidan and Amnon Shashua. Novel view synthesis in tensor space. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1034-1040. IEEE, 1997.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Brriin-,Brall,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855-5864, 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Sealing latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Shengqu Cai, Eric Ryan Chan, Songyou Peng, Mohamad Shahbazi, Anton Obukhov, Luc Van Gool, and Gordon Wetzstein. Diffdreamer: Towards consistent unsupervised single-view scene extrapola- tion with conditional diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2139-21500 2023.
Jiezhang Cao, Yue Shi, Kai Zhang, Yulun Zhang, Radu Timofte, and Lue Van Gool. Deep equilibrium diffusion restoration with parallel sampling. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2824-2834, 2024.
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view synthesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4217-4229, 2023.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19457- 19467, 2024.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single- stage diffusion nerf: A unified approach to 3d generation and reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2416-2425, 2023.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision, pp. 370-386. Springer, 2025.
Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz. Extreme view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7781-7790,
2019.
11

---

Published as a conference paper at ICLR 2025
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffee Liffuse-fasterr Accelerating conditional diffusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12413-12422, 2022a.
Hyungjin Chung. Byeongsu Sim, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. 2022b.
Hyungjin Chung, Jcongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Conference Learning Representations, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Puneet Goel, Stergios Roumeliotis, and Gaurav S Sukhatme. Robust localization using relative and absolute position estimates. In Proceedings 1999 IEEE/RSJ International Conference Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No. 99CH36289), volume pp. 1134-1140. IEEE, 1999.
Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In International Conference on Machine Learning, pp. 11808-1188 11826. PMLR, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646 2022.
Chin Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron C Courville. Riemannian diffusion models. Advances Neural Information Processing Systems, 35:2750-2761, 2022.
Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi- view stereopsis evaluation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 406-413, IEEE, 2014.
Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, and Chenfanfu Jiang. Vr-gs: A physical dynamics-aware interactive gaussian splatting system in virtual reality. In ACM SIGGRAPH 2024 Conference Papers, 2024.
Hailin Jin, Stefano Soatto, and Anthony J Yezzi. Multi-view stereo reconstruction of dense shape and complex appearance. International Journal of Computer Vision, 63:175-189, 2005.
Johanna Karras, Aleksander Holynski, Ting-Chur Wang, and Ira Kemelmaaher--Shlizenee Dream- pose: Fashion image-to-video synthesis via stable diffusion. In IEEE/CVF International Conference on Computer Vision, pp. 21663-22663... IEEE, 2023.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances Neural Information Processing Systems, 35:26565-26577, 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4):1-14, 2023.
12

---

Published as a conference paper at ICLR 2025
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zene zero: Text-to-image diffusion models zero-shot video generators. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15954-15964, 2023.
Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4):1-13, 2017.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In International Conference on Machine Learning, pp. 5742-5752. PMLR, 2021.
Obin Kwon, Jeongho Park, and Songhwai Oh. Renderable neural radiance map for visual navigation. In Proceedings of the IEEE/CVÉ Conference on Computer Vision and Pattern Recognition, pp. 9099-9108, 2023.
Qingming Liu, Yuan Liu, Jiepeng Wang, Xianqiang Lv, Peng Wang, Wenping Wang, and Junhui Hou. Modgs: Dynamic gaussian splatting from causually-captured monocular videos. arXiv preprint 00434, 2444:24406.00444. 2024.
arXiv:2406.
Cheng Lu, Yuhao Zhou, Fan Bao. Jianfei Chen, Chongxuan Li, and Jun Zhu Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:221..01095, 2022b.
Dimitra Maoutsa, Sebastian Reich, and Manfred Opper. Interacting particle solutions of fokker- planck equations through gradient-log-density estimation. Entropy, 22(8):802, 2020.
Ricardo Martin- Brruala, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210-7219, 2021.
Leonard McMillan and Gary Bishop. Plenoptic modeling: an image-based based rendering system. In Proceedings of the 22nd Annual Conference Computer Graphics and Interactive Techniques, SIGGRAPH *95, pp. 39-46, New York, NY, USA, 1995. Association for Computing Machinery. ISBN 0897917014. doi: 10.1145/1183802213333 218380. 218398.
URL https:////o// //tp:::///o...// www.:/..orro..... 1145/
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18444-1848 18455, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinoy2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199. 2023.
Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A survey of structure from = Acta Numerica, 26:305-366 2017.
Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, and Alexander C Berg. Transformation- grounded image generation network for novel 3d view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3500-3509, 2017.
13

---

Published as a conference paper at ICLR 2025
Albert Pumarola, Enric Corona, Gerard Pons-Moll and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing a consistent long-term 3d scene video from single image. In CVPR, 2022.
Gernot Riegler and Vladlen Koltun. Free view synthesis. Computer
2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pp. 623-640. Springer, 2020.
Gernot Riegler and Vladlen Koltun. Stable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12216-12225, 2021.
Robin Rombach, Patrick Esser, and Björn Ommer. Geometry-free view synthesis: Transformers and no 3d priors, 2021.
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9420-9429, 2024.
Johannes L Schonberger and lan-Michael Frahm. coucture-from-mott revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4104-4113, 2016.
Steven M Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE computer society conference on computer vision and pattern recognition, volume pp. 519-528. IEEE, 2006.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Nagabhushan Somraj. warping for view synthesis DIBR, 2020. URL UtLhtttss://ttthus htt///aaaaussaannSS55..a wtwpsaaahnnnnnt//o/-eeaaaa
com/Na
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training acore-baseds generative models. Advances in neural information processing systems, 03:12438-122 12448, 2020.
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of based diffusion models. Advances in neural information processing systems, 34:1415-1428, 2021a.
Yang Song, Jascha Sohl-Dicksted Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021b.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias NieBner, et al. State of the art on neural rendering. In Computer Graphics Forum, volume 39, pp. 701-727. Wiley Online Library, 2020.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 12959-12970, 2021.
14

---

Published as a conference paper at ICLR 2025
Hung- Yu Tseng, Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Con- sistent view synthesis with pose- asse-guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16773-16783, 2023.
Vikram Voleti, Chun-Har Yao, Mark Boss, Adam Letts, David Pankratz, Dmitrii Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel multi-view synthesis and 3D gen- eration from single image using latent video diffusion. In European Conference on Computer Vision, 2024.
Guangcong Wang, Zhaoxi Chen, Chen Change Loy, and Ziwei Liu. Sparsenerf: Distilling depth rank- ing for few-shot novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9065-9076, 2023a.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. International Conference on Learning Representations, 2023b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang. Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In International Conference on Learning Representations, 2023.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20310-20320 June 2024a.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623-7633, 2023.
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21551-21561 2024b.
Jianfeng Xiang, Jiaolong Yang, Binbin Huang, and Xin Tong. 3d-aware image generation using 2d diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2383-2393 2023.
Haolin Xiong, Sairisheek Muttukuru, Rishi Upadhyay, Pradyumna Chari, and Achuta Kadambi. Sparsegs: Real-time 360 ('deg) sparse view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00206 2023.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.09411 2024b.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable
3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20331-20341, 2024c.
Meng You and Junhui Hou. Decoupling dynamic monocular videos for dynamic view synthesis. IEEE Transactions Visualization and Computer Graphics, 2024.
Meng You, Mantang Guo, Xianqiang Lyu, Hui Liu, and Junhui Hou. Learning a locally unified 3d point cloud for view synthesis. IEEE Transactions on Image Processing, 2023.
15

---

Published as a conference paper at ICLR 2025
Jason J Yu, Fereshteh Forghani, Konstantinos G Derpanis, and Marcus A Brubaker. Long-tern pho- tometric consistent novel view synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7094-7104, 2023.
Jason Zhang, Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. Ners: Neural reflectance
surfaces for sparse-view 3d reconstruction in the wild. Advances in Neural Information Processing Systems, 34:29835-29999 29847, 2021.
Jingbo Zhang. Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields. IEEE Transactions Visualization and Computer Graphics, 2024.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint 0rXiv:2204.33900 2022.
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In European conference on computer vision (ECCV), 2022.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver--3: Improved diffusion ode solver with empirical model statistics. Advances in Neural Information Processing Systems, 36, 2024.
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 286-301. Springer, 2016.
Zixin Zou, Weihao Cheng, Yan-Pei Cao, Shi-Sheng Huang, Ying Shan, and Song-Har Hai Zhang. Sparse3d: Distilling mulliview--conister diffusion for object reconstruction from sparse views. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7900-7908, 2024.
16

---

Published as a conference paper at ICLR 2025
CONTENTS
A Scene Prior-based Score-Modulation Preserves Data Manifold
17
A.I Modulated Score Is More Accurate
17
A.2 Guided Posterior Sampling of Modulated Score Is on Data Manifold
18
B Visual Comparison Details
19
C Quantitative Comparison Details
23
Visualization of X(t, Ps)
24
Warping Strategy
24
F Performance of Our Method across Different Depth Estimation Methods
25
G Comparisons with Image Inpainting Results
25
H 360 NVS Strategy
27
I Quantitative Comparison on LPIPS
27
J Mesh reconstruction
27
K Comparison result on Dycheck
27
L Ablation on different weight function
28
M Comparison with ZeroNVS
28
Comparison with Photoconsistent-NNS
29
Comparison with Sparse Gaussian on Extreme Zoom In and Out
29
A SCENE PRIOR-BASED SCORE-MODULATION PRESERVES DATA MANIFOLD
A.1 MODULATED SCORE MORE ACCURATE
In this section, first illustrate that the proposed modulation more accurate than guided sampling an inpainting-lite manner. Denote by Xu.p. the warping content for scene's 's prior on pose p.. Then, the vanilla guided sampling methods, especially the inpainting-bass method, directly utilize Xo,p. as the guidance prior. Thus, its score estimation error shown as
Ep =-: Pipp.112
|80.p
- Ri.p. = Us 1|Ap:ll
(19)
17

---

Published as a conference paper at ICLR 2025
where Ap; indicates the variations of camera pose between the given and target views. However, for our modulated score, its score estimation error EM is shown as
1
H..p + A(t,p:)
Ko.p - Mi... ||s
EM =
(20)
1 1 P()
1+ A(t,
1
(Hi.. -Pip.)) )+ +
((t,p;)
(Xo.pe -Piin Pi
(21)
1 1 Pt)
1+ A(t, P:)
A(t, P:)
||uEp. - -1.p.l12+ 1+ A(t P:)
So.p. Mi.p.
(22)
+ Pi),
(1)
X((.P.)
1+ X(t, 0s/ ds|12p1l2
(23)
(t.
By substituting the utilized solution of A(t, p.) from Eq. (18), with Q 03/1Apl12 (2o we can derive that
201 (1)
-2m Q) + /22 4u1 Q
Em S
U/ Apll2
(24)
Q + VQ2 + 4U1Q
Q + /Q² 411Q
201 (2)
-201
U' Aplla + Ep.
(25)
-Q + VQ² + 4U1Q
-Q+ + VQ² + 40iQ
2
+ Ep.
(26)
QQ + /Q2 + 4v1Q
since 0j is small value make further approximation that
-2 Q
Ep < lim
+ Er,
(27)
Q + /Q² + 4v1Q
-2v1Q
lim
+ Ep,
(28)
Q + Q +
Q/424401
-211Q
lim
+ E
(29)
-Q+ + 11
2/04
lim #1
-40, -4u1Q² + Er,
(30)
Ep,
(31)
where I indicates apply Taylor series of /Q2 + 1e142.TTThT. Q.Thus, Ex S Ep indicates that the proposed method modulates more accurate target score function with less emor.
A.2 GUIDED POSTERIOR SAMPLING OF MODULATED SCORE IS ON DATA MANIFOLD
According to Chung et al. (2023); Huang et al. (2022), we define a local tangent space as TM- for local orthogonal projection onto manifold M. with transition process
Q Rd
Rd Et pc X)(
(32)
i dx
8p4 d.
|1 14r
(1t: p.c))
(33)
Since for the Jacobin matrix Jq. space of function Qt. Thus, we have
By dz
denotes a transition which maps a vector to the tangent
18

---

Published as a conference paper at ICLR 2025
pe: 14 Jo. (2( (a)) E M, Ox
(34)
The aforementioned proof indicates that our introduced regularization, especially (DGS), only advocates moving the latent in the orthogonal manifold direction. Moreover, considering that the updating step size is relatively small 2e -2 (t) compared to the noised latent of v we have
2e 20-2 Vo(l)
2e-
20- = 226-22
(35)
(t) +1
(o(1)
V2
[(0
Thus, both the orthogonal regularization direction and small updating step make the proposed method to be safely moving on the data manifold.
B VISUAL COMPARISON DETAILS
In this section, we visually demonstrate the results as shown in Figs. B-1, B-2, and B-3. The visual result demonstrates the superiority of the proposed method.
19

---

Published as a conference paper at ICLR 2025
Figure B-1: Visual comparison of single visw-based NVS results by (a) Text2Nerf (Zhang al.. 2024), (b) 3D-aware (Xiang et al., 2023). (c) MotionCtrl (Wang et al., 2024), (d) Ours (Post). The middle view of each scene highlighted with the red rectangle refers to the input view.
20

---

Published as a conference paper at ICLR 2025
AEE
VEE
HE
Figure B-2: (a) The two input views of each scene highlighted with the red rectangle. Visual results of multiview- NVS by (b) 3D-aware (Xiang et al., 2023), (c) MotionCtrl (Wang al., 2024), (d) Ours (Post).
21

---

Published as a conference paper at ICLR 2025
badled
Pallod
Pasl
Figure B-3: Visual comparison on dynamic scene view synthesis of (a) input frames in the correspond- ing time of generated images, (b) Siermable-Gaussiar (Yang al., 2024c), (c) 4D-4404Ga Gaussian (Wu et al., 2024a). (d) 3D-aware (Xiang et al., 2023). (e) MotionCtri (Wang et al., 2024), (f) Ours (Post). 'Failed' refers the method cannot work on the condition.
22

---

Published as a conference paper at ICLR 2025
C QUANTITATIVE COMPARISON DETAILS
We give detailed quantitative comparisons of different methods as shown Tables C-1, C-2 and C-3. The results demonstrate that the proposed methods, Ours (DGS) and (Post) outperform SOTA methods in most scenes.
Table C-1: Quantitative comparison of different methods on single view synthesis. For all metrics, the lower, the better.
Scene
Metric
Sparse Gaussian
Sparse Nerf
Text2Nerf
3D-aware
MotionCtrl
Ours (DGS)
Ours (Post)
ATE
4.424
2,850
7.111
17.160
1.261
Anditorium
RPE-T
2.020
3.398
1.448
2.669
0.265
RPE-R
0.266
2.609
1.320
1.192
0.144
ATE
1.133
2.832
3.217
2,638
0.430
Barn
RPE-T
0.307
0.868
0,369
0.533
0.104
RPE-R
0.064
2.336
1.074
0.632
0.197
ATE
2.831
3.045
3,038
0.883
Castle
RPE-T
1.425
0.716
0.634
0.171
RPE-R
1.036
0.483
0.983
0.039
ATE
2.167
2.830
3,256
3.017
0.664
Church
RPE-T
0.486
0.860
0.798
0.636
0.107
RPE-R
0.093
2.476
1.553
1,000
0.176
ATE
1,264
2,830
1.941
0.944
1.023
Family
RPE-T
0.867
0,378
0,212
0.191
RPE-R
0.050
2.295
0,890
0,461
0.514
ATE
1.561
2.831
3.431
1.226
0.408
Ignatius
RPE-T
0.365
0.866
0.495
0,342
0.129
RPE-R
0.050
2.298
0.243
ATE
2.835
3,903
0,959
L.62
Pulace
RPE-T
0.904
0.517
0.221
0.192
RPE-R
0.578
0,363
0.313
0.101
2.385
Seaside
RPE-T
0.754
0.717
0.522
0.639
0.090
RPE-R
0.073
0.473
0,283
0.311
0.040
ATE
2.628
2.854
4,740
6.521
0.776
Trees
RPE-T
0.718
1.420
1.098
1.399
0.155
RPE-R
0.151
0.859
0.506
0.894
0.079
Table C-2: Quantitative comparison of different methods on sparse view synthesis. For all metrics, the lower, the better.
Scene
Metric
Spurse Gaussian
Spurse Nerf
Text2Nerf
3D-aware
MotionCtri
Ours (DGS)
Ours (Post)
ATE
1.697
2.826
2.250
0.330
caterpillar
RPE-T
0.046
0.040
0.010
0.002
RPE-R
2.337
0.680
0.305
0.032
ATE
2.813
0.721
1.988
0.597
0.033
playground
RPB-T
0.059
RPE-R
0.156
3.534
0.524
0.201
0.113
ATE
2,556
2.718
0.058
01.68
truck
RPE-T
0.142
0.088
0.010
0.002
RPE-R
0.954
0.287
0.048
XTE
7,980
2.872
33.29
7.069
scun
RPE-T
2,518
8,477
31.02
3.918
RPE-R R
0.302
1.411
2.995
0.592
48.00
4,480
scan2
RPE-T
2.609
31.60
5,902
2.211
RPE-R R
0.307
1.614
0.648
0.324
XTE
8.795
46.22
8.660
7.617
scan3
RPE-T
2.633
31.20
6.104
3.976
RPE-R
0.269
1.656
0,940
0.564
ATE
7.902
2,886
53.38
59.03
3.436
scan5
RPE-T
2.510
8.495
16.13
32.65
2.270
RPE-R
0.297
1.413
1,526
2.551
0.326
ATE
7.072
48.75
55.68
6.504
scan1.5
RPE-T
2,394
30.57
42.98
2,796
RPE-R
0.327
1.692
1.690
0.352
ATE
8.025
97.60
31.86
6.936
scan55
RPE-T
2.487
47.21
37.41
3.098
RPE-R
0.275
4,520
2.422
0.618
23

---

Published as a conference paper at ICLR 2025
Table C-3: Quantitative comparison of different methods on dynamic view synthesis. 1_ 1/.2' means that we synthesize novel views by moving the camera on the right/left circle trajectory. For all metrics, the lower, the better.
Scene
Metric
Deformable-Gamd Gaussiaa
ID-Gaussian
3D-aware
MotionCtrl
Ours (DGS)
Ours (Post)
ATE
2.834
2,817
2.847
2.746
0.619
0.619
Street
RPE-T
1.463
1.342
1.383
0.811
0.306
0.307
RPE-R
1.009
1.322
1.0340
0.906
0.217
0.218
ATE
0,486
0.819
1.470
2,842
2.842
Street_2
RPE-T
0.169
0.163
0.235
0.647
0.647
RPE-R
0.439
0.800
0.528
0.604
0.602
ATE
3,546
4.193
1.396
3,421
3.131
Kangaroo.1
RPE-T
0.979
1.347
0.778
1.501
1,482
RPE-R
0.361
2.637
0.606
0.574
0.444
ATE
3,302
2,438
4,099
5,681
5.922
4,248
Kungarvo.2 2
RPE-T
1.672
0,852
2,489
1.973
2.065
1.608
RPE-R
0.639
0,372
0.954
0,303
0.791
0,353
1.130
Train3_1
RPE-T
0.698
0.650
2.642
1,501
0.999
0,402
RPE-R
0.468
0.588
1.494
0.471
0.455
0.245
ATE
1.144
2,880
4.031
2.186
1.026
1.315
Train3_2
RPE-T
0.968
1.108
1.748
0,908
0.448
0.476
RPE-R
0.785
1.483
2.266
0.574
0.567
0.529
ATE
2.838
1.600
3,011
3.274
0.624
0.815
Train5_1
RPE-T
0.313
0.252
1.537
0.880
0.196
0.165
RPE-R
0.546
0.699
1.523
1.042
0.453
0.629
ATE
1.046
2,3516
2.708
3.3222
0.851
2,874
Train5 .2
RPE-T
0.377
0.719
RPE-R
0.734
1.686
1.347
0.666
0.599
0.653
ATE
0.521
0.548
1.195
2.783
0.774
3.328
Road_1
RPE-T
0.219
0.251
0.667
1,061
0.326
1.016
RPE-R
0.330
0.363
0.679
0.479
0.100
0.239
ATE
2.492
2.679
2.775
3.445
2.534
2.781
Road
RPE-T
0.260
0.275
0.867
0.991
0.230
0.158
RPE-R
0.564
0.579
1.214
0.828
0.099
0.091
D VISUALIZATION OF A(t, P:)
In Fig. D-4, we visualize the Ā(t, p:) of certain sequences in different NVS conditions.
020
025
86
20
022
015
8.0
40
(8)
(e)
Figure D-4: Visualization of A(1, p:) for (a) single view synthesis, (b) sparse view synthesis (c)
dynamic view synthesis.
E WARPING STRATEGY
We illustrated the warping strategies for different NVS conditions in Fig. E-5.
24

---

Published as a conference paper at ICLR 2025
Sadll Lagal
Maliiie
Figure E-5: Illustration of different warping patterns. From left right, there are single-view, monocular video, and multi-view-based NVS.
F PERFORMANCE OF OUR METHOD ACROSS DIFFERENT DEPTH ESTIMATION METHODS
We conducted experiments on our method using various depth estimation approaches, including DINOV2 (Oquab et al., 2023) and DepthAnything V2 (Yang et al., 2024b). The results, shown in Table F-4, clearly demonstrate that our method consistently performs well across different depth estimation techniques, surpassing the state-of of-the-art NVS method, MotionCtrl (Wang et al., 2024). As expected, better depth maps indeed lead to more accurate NVS, as evidenced by the comparison between DepthAnything V1 (Yang et al., 2024a) and DINOV2 Oquab et al. (2023). Furthermore, the performance of Beythiny V2 is only comparable to Anythnn V1 due to one scene that appears to be outlier. Upon removing this scene (marked with *), DepthAnythingV2 significantly outperforms DepthAnything VI. Additionally, DINOV2 achieves lower FID due to the amplification of rendering pose errors, which facilitates easier reconstruction. Fig. F-6 visualizes the results of our V1, V2, and DINOV2. These comparisons confirm the robustness of
method with DepthAnything our method with respect to depth estimation techniques, consistently delivering high performance across different depth estimation modules, including DINOV2, DepthAnything V1, and V2.
Table F-4: Quantitative comparison of our method with DepthAAnythingvl DepthAnythingy2, and V2n all metrics, the lower, the better.
Methods
FID
ATF
RPF-T
RPE-R
Motion trl (SOTA)
179.24
3.851
0.705
0.835
Ours + DepthAnythingvl
165.12
0.767
0.156
0.170
Ours DepthAnythingv2
162.896
0.831
0.110
0.170
Ours Antthnnny
163.93
0.243
0.056
0.056
Ours DinoV2
158.24
2.395
0.454
0.674
(a)
(b)
(c)
Figure F-6: Visual results of our method with (a) DepthAnythingyl, (b) DepthAnythingy2.(c)
D2n
G COMPARISONS WITH IMAGE INPAINTING RESULTS
Fig. G-7 visualizes the comparison between using an image inpainting method and our approach. Here, we use SDEdit (Meng et al., 2022) as the image inpainting method. When pir-view inpainting is used to render novel views, it becomes difficult to ensure that the synthesized views naturally consistent. As expected, the visual comparisons show that consistency across different views cannot be guaranteed when solving NVS per-view inpainting task with SDEdit.
25

---

Published as a conference paper at ICLR 2025
(b)
Figure G-7: Visual comparisons of a synthesized video sequence by (a) image inpainting method, and (b) method.
26

---

Published as a conference paper at ICLR 2025
H 360 NVS STRATEGY
For a single view input, we first apply the proposed method to rotate 120 to the left and right sides. Moreover, to achieve consistency from different directions, further process the remaining 120" in a multi-view prompt manner, i.e., we warp both side views to the central as prompt images. Note that in the first two 120 rendering processes, it's also difficult directly achieve such large range NVS. Thus, we first reconstruct 30 NVS with 24 frames. Then, we sample 12 frames from the reconstructions the first 12 prompt images for 60% NVS. Next, we can achieve frames of 120% NVS by sampling 12 prompts from 60 NVS reconstruction. When given multiple views, the proposed method achieves 360 NVS by treating two neighboring images as side views of pair, warping them towards the central as prompt images.
I QUANTITATIVE COMPARISON ON LPIPS
We give detailed quantitative comparisons on LPIPS of different methods as shown in Tables 1-5, The results demonstrate that the proposed methods, Ours (DGS) and (Post) outperform SOTA methods in most scenes.
Table I-5: Quantitative comparison LPIPS of different methods single view synthesis. For all metrics, the lower, the better.
Scene
Metric
Sparse Gaussian
Sparse Nerf
Text2Nerf
3D-aware
MotionCtul
Ours (DGS)
Ours (Post)
Auditoriu
LPIPS
0.622
0.707
0.598
0.563
0.367
Bar
LPIPS
0.428
0.643
0.443
0.427
0.438
Church
LPIPS
0.642
0.714
0.627
0.623
0.634
Family
LPIPS
0.647
0.794
0.540
0.523
0.525
Ignatius
LPIPS
0.634
0,833
0.586
0.542
0.536
Palace
LPIPS
0.765
0.627
0.577
0.554
0.549
J MESH RECONSTRUCTION
We have applied 2D Gaussian Splatting to reconstruct the mesh on the generated 360 scene. As demonstrated in Fig.J-8, our method successfully maintains geometric consistency across the gener- ated images.
Figure J-8: Mesh reconstruction of synthesized 360 NVS.
K COMPARISON RESULT ON DYCHECK
We conducted comparison experiments on three scenes from the Dycheck dataset. Table. K-6 illustrates the results of our method alongside two Gaussian-basee methods. The three scenes from Dycheck were captured using two cameras, each positioned at a considerable distance from the other. In experiments, we used monocular video from one camera input and generated video following the trajectory the other camera. This setup introduces a significant content unoverlap challenge, the input and target videos capture substantially different perspectives of the scene, making the task particularly difficult. To address potential scale inconsistencies. we utilized the depth maps provided in the dataset. Our method outperformed the 4D-Gauss approach across all
27

---

Published as a conference paper at ICLR 2025
three metrics.The Gaussian method failed to produce viable results in this challenging scenario.
Table K-6; Quantitative comparison of our method with 4D-Gaussian, Deformablc-Gausiian on Dycheck. (resp. +) means the larger (resp. smaller), the better.
Methods
PSNR
SSIM 1 LPIPS
Deformable Gaausaai Gaussian
4D-G Gaussian Ours
12.68
0.346
0.737
15.84
0.385
0.410
L ABLATION ON DIFFERENT WEIGHT FUNCTION
We conducted ablation experiments to evaluate different weight function choices, including (1) constant value of 0.5
X(t. p:) = 0.5,
(36)
linear function
Ā(t, Pt) =t
(37)
(3) an exponential function
A(t, = (t+
(38)
The results are presented in Table. L-7 and Figure. L-9. These results highlight the superiority of our weight design, as all three simple weight functions lead to decreased performance in generated image quality and trajectory accuracy, particularly when using a constant value for weighting.
Table L-7: Quantitative comparison of different weight functions. For all metrics, the lower, the better.
Methods
FID
ATE
RPE-T
RPE-R
Constant (0.5)
175.68
6.09
1.06
0.864
Linear
174.23
1.04
0.210
0.199
Exponential
174.60
1.363
0.312
0.330
Ours
165.12
0.767
0.156
0.170
(8)
(b)
(c)
(d)
Figure L-9: Visual comparison of single view-based NVS results utilizing weight function as (a)
constant, (b) Linear, (c) Exponential, (d) Ours (Post).
M COMPARISON WITH ZERONVS
We present comparison of 360-degree NVS comparison results in Figure M-10. The results highlight sigmificant differences between the approaches. ZeroNVS generates videos by adhering to a specific and relatively constrained pattern, which limits the diversity and complexity of the generated scenes. In contrast, our method demonstrates the ability to produce videos with more intricate and realistic background geometry.
28

---

Published as a conference paper at ICLR 2025
Figure M-10: Comparison on 360 NVS results between (a) Ours and (b) ZeroNVS (Sargent et al., 2024).
N COMPARISON WITH PHOTTCONSIITTENT-NSS
conducted comparative experiment on Photoconsistent-NVS, and the quantitative results are presented in Figure N-11.
Figure N-11: Visual comparison between (a) Ours and (b) Photoconsisstet-NNSS (Yu et al., 2023).
O COMPARISON WITH SPARSE GAUSSIAN ON EXTREME ZOOM IN AND OUT
We conducted a comparative experiment by performing extreme zoom-in and zoom-out operations on Sparse Gaussian (Xiong al., 2023), illustrated in Figure 0-12.
ispul
(n)
(b)
Janocias
input
(b)
CV
(4)
Figure O-12: Visual comparison between (a) Ours and (b) Sparse Gaussian (Xiong et al., 2023), where for the first two scenes both methods are inputted with the same one view. Moreover, we also input the 12 views Sparse Gaussian the third scene and keep with I scene, which is "unfair" to our method. Experimental results demonstrate the robustness and consistency of our method with large camera pose changes.
29


Reviews:
[{"review_id": "ZwF5eLIfnt", "reviewer": "Reviewer_PHjc", "summary": "This work introduces a diffusion model-based approach to achieve novel view synthesis. In particular, it leverages the depth-warped views as guidance to achieve adaptative modulation. Experiments on single-view images, multi-view images, and monocular video input-based novel view synthesis showcase the efficacy of the introduced methods.", "questions": "Kindly refer to [Weaknesses].", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The idea of using depth-warped images as guidance for novel view synthesis is reasonable.\n* It is interesting to see that the temporal consistent video diffusion model can be effectively reformulated to achieve geometrical consistent NVS in a training-free manner.\n* Experiments on several challenging settings, including 360-degree NVS from a single view, verify the significance of the introduced method.", "weaknesses": "* Accessing the geometry accuracy. For the 360-degree case, e.g., the truck, it would be better to apply mesh reconstruction on the rendered views, similar to Fig. 5(b) in latentSplat [Wewer et al. ECCV 2024]. The reconstructed mesh will provide a clearer understanding of how well the rendered views maintain correct geometry.\n\n* Pixel-aligned metrics. For the NVS task, it would be better to report comparisons with state-of-the-part methods regarding pixel-aligned metrics, e.g., PSNR and SSIM. \n\n* Discussion with feed-forward 3DGS models. It might be interesting to see comparisons with detailed analysis between the introduced methods and those feed-forward 3DGS models, e.g., pixelSplat [Charatan et al., CVPR 2024], MVSplat [Chen et al., ECCV 2024]. And it would be better to consider adding these methods to the related work for better coverage of recent NVS works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CduDSskCrz", "reviewer": "Reviewer_cA11", "summary": "This paper introduces a NVS method that leverages large pre-trained video diffusion models without additional training. The approach adaptively modulates the diffusion sampling process using input views to produce high-quality results from single or multiple views of static scenes or dynamic videos. Theoretical modeling is used to iteratively adjust the score function based on scene priors, enhancing control over the diffusion process. The modulation adapts to view pose and diffusion steps.", "questions": "For dynamic scene comparison, it is said \"For monocular video-based NVS, we downloaded nine videos from YouTube, each comprising \nframes and capturing complex scenes in both urban and natural settings.\" \n\nWhy not just following the dynamic nerf settings? They have well aligned ground-truth for measuring the reconstruction performance.\nGeneration metrics like FID are not that reliable.\n\nSoma example datasets are HyperNerf, DyCheck (https://hangg7.com/dycheck/)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is entirely training-free, meaning it directly leverages pre-trained large video diffusion models without requiring additional fine-tuning or retraining. This feature not only reduces computational demands but also makes it adaptable to a wide range of applications where time or resources for training may be limited. The flexibility of using pre-trained models enhances its practicality, allowing users to apply this method to various scenes and tasks with minimal setup.\n\n2. The generated videos maintain high visual fidelity, delivering smooth results. This quality stems from the adaptive modulation of the diffusion process, which effectively incorporates scene details and structures from the given views, ensuring that outputs are kind of realistic.\n\n3. The method is underpinned by theoretical modeling, which guides its adaptive modulation strategy. By iteratively adjusting the score function with scene priors and analyzing estimation error boundaries, the approach achieves both controlled and adaptive modulation of the diffusion process.", "weaknesses": "1. The comparison between this method and NeRF-based methods is fundamentally imbalanced. NeRF techniques incorporate an underlying 3D structure, enabling them to render any view with predictable performance, as the 3D structure informs which views are feasible and which are not. In contrast, the proposed method lacks an explicit 3D representation, limiting its view synthesis capabilities to specific views with no guarantee of consistent performance. This distinction is significant, as NeRF's inherent 3D information allows interpretable, reliable results across views, whereas this method’s output reliability is less predictable and may vary based on input views.\n\n2. To achieve a fair comparison between the proposed method and NeRF-based techniques, the authors should first reconstruct a 3D model from the output video of this method and then re-render the scene from that reconstructed model. This process would allow for a direct assessment of both methods’ rendering consistency and quality, ensuring that comparisons consider the 3D structure NeRF inherently leverages. Also, reconstruction error like PSNR should be reported.\n\n3. The video results of the proposed method exhibit visible flickering artifacts, which could substantially affect reconstruction quality and consistency. A deeper analysis is needed to assess how these artifacts impact overall reconstruction accuracy and to identify potential mitigation strategies. This might include tuning reconstruction parameters to minimize flickering, which would help improve the method’s output stability and robustness, especially for applications sensitive to temporal consistency.\n\n4. A major contribution is the derivation of the parameter $\\lambda$ in Section 4.2, which aims to minimize the estimation error upper bound in Equation 15. However, a gap remains between this upper bound and the actual estimation error represented in the left side of Equation 15. To strengthen the theoretical foundation, the authors should provide a more comprehensive analysis of how reducing the upper bound affects the actual estimation error. This could be achieved through statistical analysis and empirical evidence showing how well the method reduces estimation error in practice, thereby validating the theoretical assumptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uBgMcGHTNo", "reviewer": "Reviewer_T9eW", "summary": "1. This paper proposes a novel view synthesis pipeline without any training. The pipeline can take single or multiple views of static scenes or monocular videos of dynamic scenes as input.\n2. This paper modulates the score function with the warped input views to control the video diffusion process and generate visually pleasing results. They achieve the modulation in an adaptive fashion based on the view pose and the number of diffusion steps.\n3. They conduct extensive results on both static and dynamic scenes and show promising results with both evaluation numbers and visualizations.", "questions": "Line 415 mentions that the proposed method can achieve 360-degree NVS. Would it be possible to include a comparison with ZeroNVS [7] to better demonstrate its effectiveness?\n\n\n[7] Sargent, Kyle, et al. \"ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed adaptive modulation of the score function in the diffusion process is novel.\n2. The proposed method achieves better results in various scenarios compared to baselines.\n3. The authors provide the code with an anonymous link, ensuring the applicability of the results.", "weaknesses": "My primary concerns are with the references and experimental details:\n\n1. Some key references on diffusion-based NVS are missing [1,2,3,4,5,6,7]. Among these, [3] specifically focuses on scenes and has released its code. Is there a particular reason it was not included in the comparison?\n2. How is the synthesized view pose calculated in this paper? In Line 364, it states that 'current depth estimation algorithms struggle to derive absolute depth from a single view or monocular video, resulting in a scale gap between the synthesized and ground truth images.' When calculating pose error, does the proposed method account for this scale gap?\n\n**Minor Points:**\n\n1. In Table 1, it is stated that MotionCtrl [Wang et al., 2023b] and 3D-aware [Xiang et al., 2023] do not require training. However, as I understand, they do require fine-tuning.\n\n[1] Wu, Rundi, et al. \"Reconfusion: 3d reconstruction with diffusion priors.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n\n[2] Watson, Daniel, et al. \"Novel View Synthesis with Diffusion Models.\" *The Eleventh International Conference on Learning Representations*.\n\n[3] Yu, Jason J., et al. \"Long-term photometric consistent novel view synthesis with diffusion models.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.\n\n[4] Cai, Shengqu, et al. \"Diffdreamer: Towards consistent unsupervised single-view scene extrapolation with conditional diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[5] Tseng, Hung-Yu, et al. \"Consistent view synthesis with pose-guided diffusion models.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.\n\n[6] Chan, Eric R., et al. \"Generative novel view synthesis with 3d-aware diffusion models.\" *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YhieKHBiO4", "reviewer": "Reviewer_b97v", "summary": "The authors propose a training free novel view synthesis paradigm based on video diffusion model. Specificcally, warped depth maps are utilized and certain sampling methods are proposed to ensure high quality NVS.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The work is complete and results seem to be good quantitively.", "weaknesses": "1. The work is complete, but the novelty and the contribution are not strong enough to meet the quality of ICLR. A good training-free is not that appealing after previous works like MotionCtrl, and it highly relys on the capability of SVD. It is hard to know about it generalizability around various video generation methods. \n2. It is recommended to show \"Directly guided sampling\" and \"Posterior sampling\" clearly in the view of practical implementation. Some illustration would be appreciated.\n3. Some equations should be displayed more clearly. For instance in eq.8, I(P_0) should not be related to the pose index i, but in eq.9 it is related to i. Maybe in eq.9 the I(P_0) should be revised as I(P_i)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xEmHCxC8Tn", "reviewer": "Reviewer_QumE", "summary": "The paper presents a training-free sampling method that generates image sequences conditioned on input camera trajectories. The input view is warped onto the target views, serving as pseudo-GT variables. To condition the reverse process, the paper re-computes the predicted mean of a reverse step with an interpolation between the predicted mean of the current noisy latent and the warped samples. It further explores calculating the optimal interpolation weight and two guidance methods (replacement and gradient). Both quantitative result and qualitative results show the proposed method can generate smooth, high-fidelity image sequences.", "questions": "1. The proposed method sets the number of inference step to 100, which is quite large. Do other diffusion-based baselines also use the same number of inference step?\n2. The paper mentions the camera trajectory is estimated (L.360). Could you provide more details on how the pose metrics (ATE, RPE) are computed?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a training-free method that enables a pre-trained diffusion model to generate image sequences based on given camera trajectories.\n2. The qualitative videos demonstrate that the proposed method generates plausible outputs.", "weaknesses": "1. The paper relies on an off-the-shelf depth estimation network to warp the input image(s), which may be prone to scale inconsistencies across frames. Since the reverse process is largely dependent on the warped images, depth estimation errors could propagate to the final outputs.\n2. The paper does not explain how occluded regions are handled in depth-warping. Handling the occluded regions can be particularly challenging when the camera pose variation is large, as $\\tilde{\\mathbf{\\mu}}_{t, \\mathbf{p}_i}$ in Eq.12 may produce degenerate outputs. \n3. The camera trajectories in most qualitative results are limited to relatively small variations or 360-degree circular poses, where scale ambiguity from the depth estimation network can largely be ignored. This raises robustness of the proposed method when the camera variations are large. The paper could benefit from including more examples with larger camera trajectories.\n4. The toy experiment in Fig.2 (a) shows that the $\\mathcal{E}_D$ decreases with the diffusion reverse process. This brings another question on how the predicted means converge to the GT image (loss=0) where it is very unlikely to have sampled $\\mathbf{X}_t$ that would have led to the desired GT. I believe more details of the experiment can help understanding the paper better.\n5. The paper makes some assumptions to compute the optimal interpolation weight $\\lambda(t, \\mathbf{p}_i)$ in Sec.4.2. However, the paper does not present ablation study on the choice of the weight (except when $\\lambda(t, \\mathbf{p}_i) = \\infty$, which also shows comparable results). To validate the choice of the weight schedule, the paper could present comparisons to other naive techniques (e.g., linear, constant, exponential). Additionally, the interpolation weight depends on a set of hyperparameters $\\{ v_1, v_2, v_3 \\}$ which may require engineering effort to tune on new scenes. \n6. While the paper shows promising quantitative and qualitative results, the number of scenes used for evaluation is insufficient to validate the method's effectiveness and differs from previous state-of-the-art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)