Controlling Space and Time with Diffusion Models
Authors: Daniel Watson
Saurabh Saxena
Lala Li
Andrea Tagliasacchi
David J. Fleet
Venue: ICLR 2025 Poster (2025)
Keywords: generative
models
diffusion
novel
view
synthesis
3d
4d
scenes
Primary area: generative models
TL;DR: Generate novel views across space and time given a 1+ input images

Abstract: We present 4DiM, a cascaded diffusion model for 4D novel view synthesis (NVS), supporting generation with arbitrary camera trajectories and timestamps, in natural scenes, conditioned on one or more images. With a novel architecture and sampling procedure, we enable training on a mixture of 3D (with camera pose), 4D (pose+time) and video (time but no pose) data, which greatly improves generalization to unseen images and camera pose trajectories over prior works which generally operate in limited domains (e.g., object centric).
4DiM is the first-ever NVS method with intuitive metric-scale camera pose control enabled by our novel calibration pipeline for structure-from-motion-posed data. Experiments demonstrate that 4DiM outperforms prior 3D NVS models both in terms of 
image fidelity and pose alignment, while also enabling the generation of scene dynamics. 4DiM provides a general framework for a variety of tasks including single-image-to-3D, two-image-to-video (interpolation and extrapolation), and pose-conditioned video-to-video translation, which we illustrate qualitatively on a variety of scenes.
See https://4d-diffusion.github.io for video samples.

Published as a conference paper at ICLR 2025
CONTROLLING SPACE AND TIME WITH DIFFUSION MODELS
Daniel Watson', Saurabh Saxena', Lala Li", Andrea Tagliasacchi, David Fleet Google DeepMind
ABSTRACT
We present 4DiM, a cascaded diffusion model for 4D novel view synthesis (NVS), supporting generation with arbitrary camera trajectories and timestamps, in nat- ural scenes, conditioned on one or more images. With a novel architecture and sampling procedure, enable training
mixture of 3D (with camera pose), 4D (pose+time) and video (time but no pose) data, which greatly improves gener- alization unseen images and camera pose trajectories over prior works which generally operate in limited domains (c.g., object centric). 4DiM is the first-ever NVS method with intuitive metric-scale camera pose control enabled by our novel calibration pipeline for structure- cecttuur-fromm-eee cooo-mmtion--oseee data. Experiments demon- strate that 4DiM outperforms prior 3D NVS models both in terms of image fidelity and pose alignment, while also enabling the generation of scene dynamics. 4DiM provides general framework for variety of tasks including single-image-to-DD two- ae--inage-to-videe (interpolation and extrapolation), and pose- conditiodd video- to-video translation, which we illustrate qualitatively on variety of scenes. See https: /oop:://4d-o0 fus on. gi github. io for video samples.
Inpul
360 rolation
Input
Circular pan
Inputs
Extrapolate time
Inputs
Interpolate time
Zero-shot samples from 4DiM LLFF (Mildenhall al. 2019) and Davis (Pont- Tuset et al. 2017) given one or two input images. 4DiM excels at wide variety of generative tasks involving camera and temporal control (e.g.. free-form camera control, temporal interpolation and extrapolation). Note that 4DiM is a 32-frame model, so here we evenly sub-sample outputs. Samples are best seen as video: https: 4d- Ad-dif Frrfffusinn github.
Figure 1:
Equal contribution.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
4D generative models cast novel view synthesis (NVS) as a form of image- conditiend multi-view generation with control of camera pose and time. They extend the impressive capabillities of generative image and video models, leveraging large-scall learning to enable image generation with consistent 3D scene geometry and dynamics. As such, they promise to unlock myriad generative tasks such as novel view synthesis from single image, image to 4D generation, image(s) to video, and video to video translation. Nevertheless, designing and training 4D models that generalize well to natural scenes, with accurate control over camera pose and time, remains challenging.
While there has been interest in diffusion models for NVS (Watson et al. 2022 Liu ct al. 2023b), much of this work has focused on isolated objects with cameras located on the viewing sphere around the object. A key barrier to generalization, both to arbitrary scenes and free-form camera poses. has been the lack of rich, posed training data. The predominant datasets, i.e., RealEEstate IOK (Zhou et al. 2018) and CO3D (Reizenstein et al., (021), are restricted to narrow domains, with camera poses estimated from structure- fromuurrerooomooooonoo (SfM) that are noisy and lack metric scale information. Consequently, state-of-the-en art 3D NVS models with good generalization to date have focused on somewhat specific domains (c.g., CAT3D (Gao et al. (024) for object-centric scenes, or PNVS (Yu et al. 2023a) for indoor scenes), and they exhibit issues with pose alignment, where generated images do not coincide with the specified target viewpoints. In the case of 4D NVS, said challenges around lack of large-scale posed training data and quality of pose estimation are exacerbated. To this end, we advocate co-training with unposed video, as diverse video is available at scale and contains valuable information about the spatiotemporal regularities of the world.
To address these challenges, we introduce 4DiM, a diffusion model for NVS conditioned on (one or more) images of arbitrary scenes, camera pose, and time. 4DiM extends previous 3D NVS diffusion models in three key respects: 1) from objects to scenes; 2) to free-form camera poses specified in meaningful physical units; and 3) with simultaneous control of camera pose and time. 4DiM is trained on mixture of data sources, including posed 3D images/video and unposed video, of both indoor and outdoor scenes. We make this possible through an architecture with FiLM layers (Dumoulin) et al. (018)) that default to the identity function for missing conditioning signals (e.g.. videos without pose, or static scenes without time), which we call Masked FILM, and through multi- muulri-uuidace, which leverages this kind of training to adjust guidance weights for different conditioning signals separately. We also introduce calibrated versions of datasets posed via COMMAP (Schönberger & Frahm 2016); calibrated data makes it easier learn metric regularities in the world, like the typical sizes of everyday objects and spatial relations, and it enables one to specify camera poses in meaningful physical units for precise camera control. As we later show, training on poses with unknown scale will (unsurprisingly) make scale itself stochastic when generating samples given one input image, whereas fixing this problem yields both improved fidelity and pose alignment.
To summarize, our main contributions include:
We introduce 4DiM, a pixel-based diffusion model for novel view synthesis conditioned on one or more images of arbitrary scenes, camera pose, and time. 4DiM comprises a base model that generates 32 images at 64 x 64 and super resolution model that upsamples to 32 x 256 x 256; We devise an effective data mixture for 4D models comprising posed and unposed video of indoor and outdoor scenes, yielding compelling results on camera control, time control, and promising zero-shot results controlling both simultaneously in video-to-video translation;
We propose Masked FiLM layers to enable training with incomplete conditioning signals, and multi- muitii- which leverages such training to further improve sample quality at inference time;
We create a scale-calibrated version of RealEstate 10K to improve model fidelity and enable metric pose control;
We include extensive evaluation and comparisons prior work, including fixes to based metrics (He et al. 2024), and novel keypoint distance metric to detect the presence of dynamics.
RELATED WORK
Unlike the typical setting of Neural Radiance Fields (Mildenhall et al., 2021) (NeRF) where tens-to- hundreds of images are used as input for 3D reconstruction, pose-conditional diffusion models for NVS aim to extrapolate plausible, diverse, 3D consistent samples with as few as a single image input. Conditioning diffusion models image and relative camera pose was introduced by Watson et al.
2

---

Published as a conference paper at ICLR 2025
(2022) and Liu al. (2023b) an effective alternative to prior few- NVS methods, overcoming severe blur and floater artifacts (Sitzmann et al.) 2019; Yu et al. 2021 Niemeyer et al. 2022] Saajadi et al. )022). However, they rely on neural architectures unsuitable for jointly modeling more than two views, and consequently require heuristics such as stochastic conditioning or Markovian sampling with limited context window (Yu et al. 2023a), for which it is hard to maintain 3D consistency.
Subsequent work proposed attention mechanisms leveraging epipolar geometry to improve the 3D consistency of image-to-image diffusion models for NVS TTeng et al.) 2023), and more recently, finetuning text-to-video models with temporal attention layers or attention layers limited to overlap- ping regions (He al. 2024. Wang et al., 2023a, Bahmani et al. 2024) to model the joint distribution of generated views and their camera extrinsics. These models however still suffer from several issues: they have difficulty with static scenes due to persistent dynamics from the underlying video models; they still suffer from 3D inconsistencies and low fidelity; and they exhibit poor generalization to out-of-distribution image inputs. Alternatives have been proposed improve fidelity in multi-view diffusion models of 3D scenes, albeit sacrificing the ability to model free-form camera pose; e.g., see MVDiffusion and follow-ups (Tang et al. 2023 2024) focusing on few, specific pose trajectories.
A concurrent body of related work on 3D extraction has recently emerged following DreamFusion (Poole et al., 2022), where instead of directly training diffusion models for NVS, new techniques for sampling views parametrized Distillation Sampling (SDS) and Variational Score Distillation (Wang et al. 2024) (VSD). Here, pre-existing diffusion model acts prior that drives the generation process. This enables, for example, text-to-3D using a text-to-image diffusion model. As demonstrated by ZeroNVS (Sargent al. 2023), MVDream (Shi al. 2023), ReconFusion (Wu et al. 2023) and CAT3D (Gao et al. 2024). using& pose-conditional diffusion model offers the advantage that the diffusion model can be conditioned on the sampled viewpoint for volume rendering during score distillation or NeRF postprocessing. Still, the extension of 3D extraction methods to 4D dynamic scenes and complex view- dependent reflectance phenomena remain challenging. In contrast, geometry-free diffusion models show encouraging results at scale with camera control, time control, and even their combination (Setzer et al., (223). Large-scale video models also provide encourage evidence of 3D geometrically consistent generation without an explicit rendering model (Brooks et al. 2024a] Li et al. 2024). Our success with 4DiM further supports this perspective, showing that one can simultaneoush achieve fine-grained camera control and dynamics with multiview diffusion model alone.
NeRF with volume rendering are proposed such Score
3
4D NOVEL VIEW SYNTHESIS MODELS FROM LIMITED DATA
4DiM uses a continuous-ting time diffusion model to learn joint distribution over multiple views,
p(#C+1NN 11C PI:N, (1N)),
where EC+1:N are generated images, T1.C are conditioning images, PIay are relative camera poses (extrinsics and intrinsics) and fı:v are scalar, relative timestamps Following Ho et al. (2020) we choose loss function to be the error between the predicted and actual noise (Esimple in their paper), though we use the LI rather than L2 norm, because, like prior work (Saharia et al. 2022a
Saxena al. )023), we found that it improves sample quality. Our models use the "u-parametrizzation"
(allmans & Ho| 2022). which helps stabilize training, and we adopt the noise schedules proposed
by Kingma et (2021) D. Our current models process N 8 or 32 images at resolution 256 x 256 (N is the combined number of conditioning plus generated frames). To this end, we decompose the task into two models similar to prior work (Ho et al. 2022a b): we first generate images at 64 x 64, and then up-sample using 4x super-resolution model trained with noise conditioning augmentation (Saharia et al., 2022c). We also finetuned the 32-frame model to condition on 2 and 8 frames to enable more comparisons and example applications of 4DiM models.
Training data. While 3D assets, multi-view image data, and 4D data are limited, video data is available at scale and contains rich information about the 3D world, despite not having camera poses. One of our key propositions is thus to train 4DiM on large-scale dataset of 30M videos without pose annotations, jointly with 3D and 4D data. As shown in Section 5. video plays a key role in helping to regularize the model. The 3D datasets used to train 4DiM include ScanNet++ (Yeshwanth et al.
"4DiM does not require sequential temporal ordering as in video models as the architecture is permutation- cquivariant over frames. All N images (conditioning and generated) are processed by the diffusion model.
3

---

Published as a conference paper at ICLR 2025
Conditioning
64x
32x
Project
MEP
Time DIT
FILM identity
Attention
x16
FILM Identty
Framo DIT
FILM Identty
Project
64x
Corditioning
e, 0,0,0
Figure 2: Architecture We illustrate the 4DiM base model architecture, including choices for attention blocks and our conditioning mechanism. The 4DiM super-resolution model follows the same choices modulo minor differences to condition on low-resolution images. Outputs corresponding to the conditioning frames are always discarded from the training objective.
2023) and Matterport3D (Chang et al. 2017). which have metric scale, and more free-form camera poses compared to other common 3D datasets in the literature (e.g.. CO3D (Reizenstein al. 2021) and MVImgNet (Yu et al. 2023b)). We also use 1000 scenes from Street View with permission from Google, comprising posed panoramas with timestamps (i.e., it is a "4D" dataset). During training, we randomly sample views from Street View from the set of panorama images within K consecutive timesteps (K=5 for our 8-view models, and K=20 for 32-view models). We sample unposed videos with probability 0.3 and views from posed datasets otherwise. 3D datasets are sampled in proportion to the number of scenes in each dataset.
Calibrated RealEstate10K. One particularly rich dataset for training 3D models is RealEstate 10K (Zhou al. (018) (RE10K). It comprises 10,000 video segments of static scenes for which SfM (Schönberger & Frahm 2016) has been used to infer per-frame camera pose, but only up to an unknown length scale. The lack of metric scale makes training more difficult because metric regularities of the world are lost, and it becomes more difficult for users to specify the target camera poses camera motions in any intuitive units. This is especially problematic when conditioning single image, where scale itself otherwise becomes ambiguous. We therefore created a calibrated version of RealEstate 10K by regressing the unknown metric scale from a monocular depth estimation model (Saxena et al. 2023). (For details, see Supplementary Material A)) The resulting dataset, CREJOK, has major impact on model performance (see Section below). We follow the same procedure to calibrate the LLFF dataset (Mildenhall et al. 2019) for evaluation purposes.
Architecture. Relatively little training data has both time and camera pose annotations. Most 3D data represent static scenes, while video data rarely include camera pose. Finding a way to effectively condition on both camera pose and time in way that allows for incomplete training data is essential. We thus propose to chain Masked FiLM layers (Dumoulin et al., (2018) for (positional encodings of) diffusion noise levels, per-pixel ray origins and directions, and video timestamps. When any of these conditioning signals is missing (due to incomplete training data or random dropout for classifier-fies guidance (Ho & Salimans 2022)) the FiLM layers are designed reduce the identity function, rather than simply setting missing values to zero. This avoids the network from confusing a timestamp or ray coordinate of value zero with dropped or missing data. In practice, we replace the FiLM shift with zeros and scale with ones for masked signals. We illustrate the overall choices in Fig. 2 (For
details see Supplementary Material (
Sampling. Steering 4DiM with the correct sampling hyperparameters is essential, especially the guidance weights for classifier-free guidance (Ho & Salimans 2022) (CFG). In its usual formulation, CFG treats all conditioning variables as "one big variable". In practice, placing a different weight on each variable is important; e.g., text requires high guidance weights, but high guidance weights images can quickly lead to unwanted artifacts. We thus propose multi- guidance, where we generalize CFG to do exactly this, without making independence assumptions between conditioning variables.
4

---

Published as a conference paper at ICLR 2025
In essence, multi quitii-uiddnce generalizes 2-guidance originally proposed in InstructPix2Pix (Brooks
et al. 2023) to N variables. We start from a classifier-guide formulation (Dhariwal Nichol 2021 where we wish to sample with & conditioning signals, v1 U2,... using the score
V log ((x) + wi Va log p(v1 + Σ
wy V plgey
(2)
j=...
In the classifier-free formulation, this is equivalent
(1 - (1-w)) V logp(x)
+ Σ (wj-1 w/) V log + wn V logp(z/viaa) (3)
j=2...
If we train our model such that it drops out only Uk. or uk-1 and Up. or all of Visk. we can sample with different guidance weights w; on each conditioning variable u, 4DiM is trained to drop out conditioning signals with probability 0.1, and drops either f1.N, f1:N and PI:N. or all of 11:N. P1:N and æT.C This is natural choice, since poses and timestamps without corresponding conditioning images do not convey useful information. For best results, 4DiM uses a guidance weight 1.25 on conditioning images and weights of 2.0 on camera poses and timestamps. We include ablation study in our Supplementary Material B that demonstrates how multi-guidance provides additional knobs to push the Pareto frontiers between fidelity and dynamics / pose alingnment.
EVALUATION
Evaluating 4D generative models is challenging. Typically, methods for NVS are evaluated using generation quality. In addition, for cose-conditioned generation it is desirable to find metrics that capture 3D consistency (rendered views should be grounded in a consistent 3D scene) and pose alignment (the camera should move as expected). Furthermore, evaluating temporal conditioning requires some measure that captures the motion of dynamic content. We leverage several existing metrics and propose new ones to cover all these aspects, as described below.
Image and Video fidelity. FID (Heusel et al., 2017) is a key metric for image quality, but used in isolation it can be uninformative and a poor objective for dyper-paramett selection. For example, Saharia et al. (2022b) found that FID scores for text-to-image models are optimal with low CFG weights (Ho & Salimans 2022), but text-image alignment suffers under his setting. In what follows we report FID as well as the improved Frechet distance with DINOv2 features (FDD) (Oquab et al., 2023), which appears correlate with human judgements better than Inception features (Szegedy
et al.
2016 For video quality, we also report FVD scores (Unterthiner et al.) 2018).
The work of Yu et al. 2023a introduced the thresholded symmetric epipolar distance (TSED) evaluate 3D consistency, a more computationally efficient alternative to training separate NeRF models (Mildenhall et al. 202T) on every sample (e.g., like (Watson et al. 2022)). First, SIFT is used to obtain keypoints between pair of views. For each keypoint in the first image, we compute the epipolar line in the second image and its minimum distance to the corresponding keypoint. This is repeated for each keypoint in the second image to obtain a SED score. A hyperparameter threshold is selected, and the percentage of image pairs whose median SED is below the threshold is reported. Below, we report TSED with a threshold of 2.0 and include the TSED score of ground truth data, poses in the data noisy and do not achieve perfect score.
3D consistency.
Pose alignment (revised metric: SEM distances). We follow He al. (2024) to quantify pose alignment, i.e., by running COLMAP pose estimation on the generated views, and comparing the camera extrinsics predicted by COLMAP to the target poses. To ensure the results are reasonable metrics, however, we modify the procedure of He et al. (2024) in several ways. First, due to the inherent scale and rotational ambiguity of COLMAP, we observe that it is necessary align the estimated poses to the original ones before comparing their differences as described above. We also report the relative error in camera positions (SfMD pus) and the angular deviation of camera rotations in radians (SIMDrat) to account for scale variation across scenes. Finally, to get best results from COLMAP, we also feed it the input views. Please see our Supplementary Material for more detail. We refer this set of metrics SYM distances hereon. Like we do for TSED scores, we also report the SfM distances for the ground truth data.
Metric scale pose alignment. The aforementioned metrics for 3D consistency and pose alignment invariant to the scale of the camera poses as they rely on epipolar distances and SfM. In order to
5

---

Published as a conference paper at ICLR 2025
REIOK
REIOK
LLFF (acro-shot)
Input
PNVS
MoticnCul
ZeroNVs
4DIM
Reference
Figure 3: Qualitative comparison of 3D NVS across various diffusion models on in-distribution (RE10K) and cittof-diitribution (LLFF) datasets. The model is 8- 8-frame 4DiM model trained solely on RE10K for 3D data for fair comparison with prior work (it not our final 4DiM model). 4DiM demonstrates superior image fidelity and pose alignment compared other methods, as be seen in (1) the quality and realism of images furthest away from the input image, and (2) the location of objects and scene content alignment aligning with the ground truth images (shown the bottom row). See our website for more samples: https::/ 4d di fus ion. wwt.gg.tthh...
evaluate the metric scale alignment of 4DiM report PSNR, SSIM (Wang al. 2004) and LPIPS (Zhang et al. 2018). These reconstruction-baat based metrics are not generally suitable to measure sample quality, as generative models can produce different but plausible samples. But for the same reason (i.e., that reconstruction metrics favor content alignment), we instead find them useful as an indirect indicators of metric seale alignment; i.e., given metric-scale poses, we would like models to not over/undershoot in position and rotation.
Dynamics (new metric: keypoint distance). One common failure mode that we observed early in our work a tendency for models to copy input frames instead of generating temporal dynamics. Prior work has also noted that good FVD scores can be achieved with static videos (Ge et al. 2024). We therefore propose a new metric called keypoint distance (KD), where we compute the average motion of SIFT keypoints(Lowwe/ 2004) across image pairs with k > 10 matches. We report results generated and reference views to assess whether generated images have a similar motion distribution.
EXPERIMENTS
We consider both in-distribution and OOD evaluations for 3D NVS in our ablations and comparisons to prior work. We use the RealEstate 10K dataset (Zhou et al. 2018) common in-distribution evaluation. We use 1% of the dataset as our validation split and compute metrics on all baselines ourselves for this split, noting they might instead be advantaged training data (for PNVS, all our test data is in fact part of their training dataset). For the OOD case, the LLFF dataset (Mildenhall et al. 2019). We present main results 3D novel view synthesis conditioned on single image in Tab. and Fig. 3, comparing the capabilities of our 4DiM cascade against PhotoConsissentNNV
our test data may exist in their
Yu al. 2023a (PNVS), MotionCtrl (Wang al. 2023b).
Foor PNVS, we follow Yu et al. 2023a and use Markovian sliding window for sampling, they find it is
the stronger than stochastic conditioning (Watson al. 2022

---

Published as a conference paper at ICLR 2025
HID (t)
FDD (4)
TSEDp; (1)
(I)
SIMD,
LPIPS (J)
PSNR (t)
SSIM (1)
CRETOK Best
PNVS
51.41
1007.
0,9961 (1.000)
0.9773 (1.024)
0.3068 (0.3638)
0.3899
16.07
0.3878
MotionCtrl
43.07
370.6
0.4193 (1.000)
1,027 (1.032)
0.2549 (0.9634)
0,5003
12.74
0,2668
ZeroNVS
37.35
522.3
0,6638 (1.000)
1.049 (1.028)
0.3325 (0.3579)
0.3792
15.30
0.3617
4DIM-R
31.23
306.3
0,9974 (1.000)
1 1.023 (1.075)
0.3029 (0.34133
0.2630
18,09
0.5309
4DiM
31,96
314,9
0,9935 (1,000)
1 10088 (1.034)
0.3326 (0.3488)
0.3016
17.08
0.4628
CLEF zero shot
PNVS
185.4
2197.
0.7235 (0.9962)
0.9346 (0.8853)
0.2213 1857)
0.5969
12.04
0.1311
MotionCtrl
106.0
$31.5
0. 1515 (0.9962)
0.9148 (0.9415)
0.1366 (0.2216)
0,7016
9.722
0.06756
7eMNVS
75.95
860.2
0.2205 (0.9962)
0.9315 (0. (980))
0.2015 (0. 1947)
0.5008
12.58
0,1578
4DIM-R
63.48
353.2
0.9689 (0.9962)
0.9265 (0.8951) )
0.2011 (0. 1934)
0,5403
11.55
0.1444
4DIM
63.78
356.8
0.9167 (0.9962)
0,9131 (0.8960)
0.1838 (0.1943)
0.5415
11.58
0.1408
Table 1: Comparison to prior work in 3D NVS. We compare 8-frame 4DiM models against prior work, with one model trained solely with cRE10K (4DiM-R) for a more fair comparison, and one model trained with our full dataset mixture (4DiM). We evaluate in-distribution and zero-shot on LLFF. For 3D consistency and pose alignment, we include scores on real views (in parenthesis) to indicate plausible upper bounds (which may be imperfect as poses noisy and SfM may fail). We highlight the best result in bold, and underline results where models score better than real images. We do not report scores when SIM fails, denoted with n/a. Our models generally outperform baselines substantially across all metrics.
FID (L)
FDD (1)
TSED (t)
(L)
SIMD,.(.)
LPIPS (1)
PSNR (t)
SSIM (1)
RETO% lest
4DIM-R (RE10K)
32.40
335.9
1,000 (1.000)
1,01 (1.022)
0,3508 (0.3737)
0.3002
16.64
0.4704
4DIM-R (0k)
31.23
306.3
0.9974 (1.000)
1.023 (1.075)
0.3029 (0.3413)
0.2630
18.09
0.5309
LLFF zero-shot
4DIM-R (REJ0k)
71.07
521.2
0.7F27 (0.9962)
0.9003 (0.8876)
0.2 (0. 1915)
0.4568
12.87
0.2016
4DIM-R (cRF 10k)
63,48
353.2
0.9689 (0.9962)
0.9265 (0.8951)
0.2011 (0.4434)
0.5403
11.55
0.1444
ScanNet++ zero-shot
4DIM-R (RF.IOk
23.68
248.0
0.9512 (0.9815)
I 885 (1.85)
1.534 (1.520)
0.1938
20.74
0.6716
4DIM-R (cRE10k)
22.89
243.2
0.9685 (0.9815)
1851 (1.917)
1.541 (1.550)
0.1809
21.25
0.6952
Table 2: Ablation showing the advantage of using calibrated data (metric-scale camera poses). We compare 8-frame 4DiM models trained on cRE10K vs uncalibrated RE10K. For RE10K and LLFF, models trained on cRE10K are tested on calibrated data, and models trained on uncalibrated REIOK are tested on uncalibrated data for a fair comparison. The exception is ScanNet++, which is already scale-calibrated. We find that training on cRE BREIOK substantially improves in-domain and OOD performance both in terms of alignment and image quality. This especially clear on reference-based image metrics such as PSNR, LPIPS, SSIM (with the exception of LLFF, where scores may be too low to be significant).
FID
FDD (1)
TSED (1)
S(MD,..()
SIMD,.(J)
LPIPS (1)
PSNR
SSIM (?)
GRETOK lest
4DiM (no video)
32.45
326.5
0.9974 (1.000)
1.023 (1.029)
0.3356 (0.3692)
0.2713
0.5209
4DiM
31.96
314.9
0.9935 (1.000)
1,MB8 (1.03)
0.3326 (0.348))
0,3016
17.08
0.4628
ScanNot++ test
4DiM (no video)
24.61
250.7
0.9615 (0.98 15)
I. 1.89 (1.764)
1.466 (1.43))
0.1824
21.34
0.6969
4DIM
23.48
223.7
i/a (0.9815)
1.706 (1 (88)
1.550 (1.503)
0.1965
20.67
0.6434
cLLFF zero-shot
4DiM (no video)
64.34
401.4 .4
0.9886 (0.9962)
0.9157 (0.9067)
0.1872 (0.1965)
0.5437
11.45
0.1430
4DiM
63.78
356.8
0.9167 (0.9962)
0,9131 (0.8960)
0,1838 (0.1943)
0,5415
11.58
0.1438
Table 3: Ablation showing the advantage of co-training with video data. We train an identical 8-frame 4DiM model without video data to reveal the net-positive impact of co-training with video. Co-training with video data improves fidelity and generalization. Qualitatively, this is evident in 4DiM's ability to extrapolate realistic content.
and the diffusion model prepared by ZeroNVS (Sargent et al. 2023). These are the strongest 3D NVS diffusion models for natural scenes with code and checkpoints availabll. For LLFF, we load the input trajectory of views in order, subsample frames evenly, and then generate 7 views given a single input image. RealEstate 10K trajectories are much longer, so following PNVS, we subsample with a stride of 10. We use the 7-view NVS task conditioned on a single image, in order to avoid weakening the baselines: MotionCtrl can only predict up to 14 frames, and PNVS performance degrades with the length of the sequence as it is an image-to-imanl
model. We trained versions of 4DiM that process 8-frames to this end (as opposed to using the main 32 32-frame 4DiM model and subsampling the output) for a more apples-to hoples-to-applee to-apless comparison. Quantitative metrics are computed on 128 scenes from Fitate 10K test split, and on all 44 scenes in LLFF.
7

---

Published as a conference paper at ICLR 2025
REIOK
CREIOK
Reference
RE
cRETOK
Reference
RETOK
cRETOK
Reference
Figure 4: Qualitative comparison of 4DiM trained on calibrated vs uncalibrated RealEstate The reference images show the scene that should be generated given the target camera pose. The model trained on calibrated data predicts scenes close to the ground truth, while the model trained on uncalibrated data often predicts scenes that have "overshot" along the trajectories. We include colored dotted lines on the furthest view from the conditioning input image to help illustrate the (mis)alignment of scene content (e.g., doorways, trees, stairs, etc.) of the generated samples when compared to the reference images.
We find that 4DiM achieves superior results in image quality (FID, FDD, FVD) by a wide margin compared to all the baselines, and generally outperforms them in reconstruction-baasee basdd metrics (PSNR, SSIM, LPIPS) with the exception of ZeroNVS on the LLFF dataset. As much prior work has shown (Saharia et al. 2022c Watson al. 2022 reconstruction-basee metrics penalize generated samples that may be plausible and tend to favor blurry images (for instance, diffusion architectures trained purely regression models tend outperform their generative counterparts in these metrics). Indeed, ZeroNVS generally produces samples with signficant blur (more qualitative comparisons are included in our Supplementary Material Gp. Qualitatively, we find that MotionCtrl does not suffer similarly from blur, but has trouble aligning with the conditioning target poses, as is evident in Figure 3 We also observe that PNVS exhibits more artifacts, with a substantial drop in quality in the zero-shot setting. Surprisingly. PNVS achieves the best TSED in LIFFF but upon closer inspection, we find that the number of SIFT keypoint matches is ~3x less than for 4DiM when computing TSED scores. This suggests potential improvements for TSED which we leave for future work, such as higher threshold for the number of minimum keypoint matches, or requiring more spatial coverage of keypoints to discard examples for which the matches too localized.
5.1 3D DATASETS SIGNIFICANTLY AFFECT NVS QUALITY
One of the most critical ingredients for 3D NVS, especially in the out-of-distribution setting, the training dataset. We ablate this in three ways:
More diverse 3D data is helpful. Table [ shows that adding more diverse 3D data beyond RealEstatel the training mixture improves pose alignment (SfM distances) on LLFF (zero-shot). The evaluation suggests a very small loss in fidelity when using the full mixture, which is expected since the mixture more diverse and includes both indoor and outdoor scenes.
Scale-calibrated data resolves ambiguity. To determine the impact of consistent. metric scale training data, we compare model trained with on the original RE10k data against one with calibrated metric scale poses. Neither model includes the other 3D data sources in their training data used to train our best 4DiM model. Quantitative results on 3D NVS from a single conditioning image are shown in Tab. 2 (including zero-shot performance on LLFF and ScanNet++), and qualitative results in Fig. We find that models trained on uncalibrated data often overshoot or undershoot when scale is ambiguous, and that models trained on calibraabb data solve this problem.
Large-scale video data improves generalization. We also show the importance of video as a source of training data. Table 3 shows results comparing 8-frame 4DiM trained on final data mixture to an otherwise identical model but trained without video. We find that including video data improves fidelity and generalization ability which expected since available 3D/4D data is not diverse. The improvement is significantly more apparent qualitatively (see Supplementary Material | for side-by-soe by-s samples), where is clear that without video geometry becomes much worse.
8

---

Published as a conference paper at ICLR 2025
Input
Street
Stitched using gamma adjustment homography warp
Generated
Figure 5: Stitching panoramas is non-trivial due exposure differences, which is hard to fix with simple gamma/gain adjustment (middle row). Here we show novel views generated by 4DiM conditioned on input frames covering a 360 field of view (bottom row). To use our 8-frame-input model, conditioning frames are randomly replicated to achieve the desired arity. (Note this is a subset of the 24 generated frames by 4DiM.)
30M video dataset test split
FID (L)
FDD (L)
FVD (1)
KD (f)
4DiM (I-frame conditioned)
62.73
355.4
934.0
2.214 (11.64)
4DiM (2-frame conditioned)
56.92
334.7
809.8
2.820 (11.51)
4DiM (8-frame conditioned) 43.34
320.7
400.2
3.181 (6.269)
Table 4: Comparing different numbers of input frames for video extrapolation. With 32-frame 4DiM models, we find that increasing the number of conditioning frames greatly improves dynamics. Note that, because this is an extrapolation task with constant video frame rate, the true keypoint distance decreases with more conditioning frames (i.e., less frames generate).
5.2 EMERGENCE OF TEMPORAL DYNAMICS
Early our experiments, we observed that video from single image can often yield samples with little motion. While this can be remedied by using more guidance weight on timestamps than on the conditioning images (see our Supplementary Material B). the seall--mage-conitttnn remains difficult. Nevertheless, with two or more input frames, 4DiM can successfully interpolate and extrapolate video and generate very rich dynamics. We quantitatively illustrate these results through keypoint distance metric (along with other standard metrics) in Tab. 4 We also include quantitative and qualitative comparisons to prior frame interpolation methods in our Supplementary Material H We suspect that future work including additional controls (e.g., text) models like 4DiM, as well as larger scale, will likely play a key role in breaking modes for temporal dynamics, given that sett-maag--t-videe has been successful for large models (Brooks al. 2024b) and the same large-scale efforts report that at smaller scales dynamics quickly worsen (e.g., see SORA samples with 16x less compute).
case
5.3 MULTIFRAME CONDITIONING
While we (and most baselines) focus on single frame conditioning, using more than one frame unlocks interesting capabilities. Here we show two applications. First, we consider 4DiM on the task of panorama stitching (rendering novel viewpoints), given discrete images that collectively cover a full 360 FOV (see Fig.5p. For reference gamma adjustment (Brown & Lowel (007). We find that outputs from 4DiM have higher fidelity than this baseline, and they avoid artifacts like those arising from incorrect exposure adjustment.
stitch the images using naive homography warp and

---

Published as a conference paper at ICLR 2025
Input
Street View
Warped input
Generated
Figure Novel views conditioned on a space-time trajectory on the Street View dataset. We condition on consecutive time steps from the front-left camera and generate views at the same time steps with camera facing the front. We find that the hallucinated regions (right half of the images) are consistent across space-time, Warped input images are provided for reference. We include more samples for video-to-video translation tt/ttps: /top::://dd-aa fiusiiiffoooo weciiinn..tttii iitbbbii..
# conditioning
Street View panoramas
Matterport3D 360"
panoramas SSIM (1)
frames
FID (L)
FDD (J)
PSNR (t)
SSIM (?)
LPIPS (L)
FID (3)
FDD (L)
PSNR (f)
LPIPS (T)
I FOV)
119.7
1601
12.16
0.2925
0.5438
86.41
1080
9.366
0,1638
0.6467
6 (360' FOV))
37.43
354.1
17.76
0.5249
0.2663
24.86
239.4
15.64
0.4974
0.2987
Table 5: Quantitative performance on generating panos in the extrapolation (number of conditioning
frames is 1) and interpolation (number of conditioning frames is 6) regime. Using more conditioning frames improves image fidelity which is to be expected. See Fig. for samples.
We finally consider the task of rendering existing space-time trajectory from novel view- points (i.e., pose-conditional video-to-video translation). Fig. G we condition on images from a single camera at 8 consecutive frames from Street View sequence and render differ- ent views yielding 3D-consistent hallucinated regions in the generated images. In our website fnttps://4n 4d lollddiiffuue diffusion gi thub io), we include many more samples video-to-video translation to demonstrate this works well in non-street scenes. We show that 4DiM can be used to stabilize the camera in input video, and also move it along novel trajectory when the input video may feature a different camera motion.
DISCUSSION AND FUTURE WORK
To the best of our knowledge, 4DiM the first model capable of generating multiple, approximately consistent views over simultancous camera and time control from as few as a single input image. 4DiM achieves state-of-the-arr pose alignment and much better generalization compared prior work on 3D NVS. This new class of models opens the door to myriad downstream applications, some of which we demonstrate, e.g., changing camera pose in videos and seamlessly stitching panoramas. While still not perfect, 4DiM also achieves previously unseen quality in extremely challenging camera trajectories such as 360 rotation from single image the wild, indoor or outdoor.
Limitations and future work. Results with 4DiM are promising, but there is room for improvement, with more calibrated 3D/4D data and larger models, with which the improved capacity is expected to improve image fidelity, 360 camera rotation, and dynamics with single-frame conditioning.
Societal impact. Like generative image and video models, it is important to develop and deploy models responsibly and with care. 4DiM is trained largely data of scenes without people (or anonymized where present), and does not condition on text prompts, which mitigates many of the safety issues that might otherwise arise.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin- Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et Vd3d: Taming large video diffusion transformers for 3d camera control. arXiv preprint arXiv:2407. 2edi::2477.1788.... 2024.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning follow image editing instructions. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 18392-18402, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufci Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024a. URL aattps://openat www.ppanaa.c.ememeeoooommoo/oeee coom//rsearch www.ssttosaaaaaaa
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024b. URL
com/ research/
generatin -odeiiooooomessseeoo as wos-worldn morrld- simulators
Matthew A. Brown and David G. Lowe. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, 74:59-73 2007. URL https://api
semanticscholase ort. wttp/Corrssii:52222222
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 170.00668, 2017.
Duolikun Danier, Fan Zhang, and David Bull. Ldmvfi: Video frame interpolation with latent diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1472-1480. 2024.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 317) 2018.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Bruain, Pratul P. Srinivasan, Jonathan T. Barron, and Ben Poole*, Cat3d: Create anything in 3d with multi-view diffusion models. arXiv, 2024.
songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Yan Zhu, and Jia- Ban Huang, On the content bias in fr' Techet video distance. arXiv preprint arXiv:2404.1239) 2024.
Justin Gilmer, Andrea Schioppa, and Jeremy Cohen. Intriguing properties of transformer training instabilities, 2023. To appear.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cam- eractrl: Enabling camera control for text-to-vides generation. arXiv preprint 244 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12595 2022.
11

---

Published as a conference paper at ICLR 2025
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1446840-6551 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022b.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision, pp. 624-642. Springer, 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, Advances in neural information processing systems, 34:21696-21700 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1e.. 22800, 2014.
Xuanyi Li, Daquan Zhou, Chenxu Zhang, Shaodong Wei, Qibin Hou, and Ming-Ming Cheng, Sora generates videos with stunning geometrical consistency. arXiv preprint arXiv:2402.17403, 2024.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprint arXiv:2310.0188 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. 7ero-1-to-3: shro one image 3d object. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9298-93000 2023b.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021.
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5480-5490, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.07199. 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Horunng, and Luc Van Gool. The 2017 davis challenge on vidco object segmentation. arXiv preprint arXiw:1704.00675, 2017.
Ben Poole. Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
12

---

Published as a conference paper at ICLR 2025
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking. Storage and Analysis, pp. 1-16. IEEE, 2020.
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. In European Conference on Computer Vision, pp. 250-266. Springer, 2022.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901-10911, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medicul image computing and compuler-ussisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 234 231.241. Springer, 2015.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic -ext-to-image diffusion models with deep language understanding. Advances neural information processing systems, 11.36479-364999 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713 4726, 2022c.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučié, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation transformer:
Geometry-fiee free novel view synthesis through set-latent scene representations. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6229-6238, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from single real image. arXiv preprini arXiv:2310. 11..1...999...... 2023.
Saurabh Saxena, Junhwa Hur, Charles Herrmann, Deqing Sun, and David Fleet. Zero-shot metric depth with a ffeddoofvvew conditioned diffusion model. arXiv:2312.133555 2023.
Johannes Lutz Schönberger and Jan-Michael Frahm. Structueefrom-Mootion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Maximilian Seitzer, Sjoerd van Steenkiste, Thomas Kipf, Klaus Greff, and Mchdi SM Sajjadi. Dyst: Towards dynamic neural scene representations on real-world videos. arXiv preprint 10.::2330.06022) 2023.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 165 2023.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-structure-awe awwree neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
13

---

Published as a conference paper at ICLR 2025
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-awerr diffusion. arXiv, 2023.
Shitao Tang, Jiacheng Chen, Dilin Wang. Chengzhou Tang. Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction, 2024.
Hung- Yu Tseng Qinbo Li, Changil Kim, Suhib Alsisan, Jia-Bin Huang, and Johannes Kopf. Con- sistent view synthesis with pose-guides diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16773-16783 2023.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.017177 2018.
Zhengyi Wang, Cheng Lu, Yikai Wang. Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu, Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641, 2023a
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionetrl: unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.036411 2023b.
Daniel Watson, William Chan, Ricardo Martin Burtala,, Jonathan Ho, Andrea Tagliasacchi, and Mo- hammad Norouzi. Novel view synthesis with diffusion models. arXiv preprint arXiv:2210.04628, 2022.
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P. Srinivasan, Dor Verbin, Jonathan T. Barron, Ben Poole, and Aleksander Holynski. Reconfusion: 3d reconstruction with diffusion priors. arXiv, 2023.
Yifeng Xiong, Haoyu Ma Shanlin Sun, Kun Han, and Xiaohui Xie. Light field diffusion for single-view novel view synthesis. arXiv preprint arXiv:2309. //25, 2023.
Chandan Yeshwanth, Kuch-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one few images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4577-4577, 2021.
Jason Yu, Fereshteh Forghani, Konstantinos G Derpanis, and Marcus Brubaker. Long-term photo- metric consistent novel view synthesis with diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7071-7081. IEEE, 2023a.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9150-9161, 2023b.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2025
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 1 1805.09817, 2018.
15

---

Published as a conference paper at ICLR 2025
Diffusion Models for 4D Novel View Synthesis Supplementary Material
This supplementary material expands upon the main paper, DDiffusion Models for 4D Novel View Synthesis', by providing in-depth details and analyses. We first delve into the scale calibration process for the RealEstate 10K dataset and the scene filtering methodology (Section A).. We also discuss the effect of multi-guie gulltigguddncee (Section B Next, we provide a comprehensive overview of the 4DiM architecture, including its UVIT backbone, transformer blocks, and camera ray conditioning (Section C). We then detail the training methodology and compute cost (Section D) and elaborate on the evaluation metries used (Section E Foffess details of the implementation of baseline models presents a qualitative comparison of 360 scene generation between 4DiM and ZeroNVS. Finally, we showcase the capabilities of 4DiM with extensive collection of generated samples (Section For more complete collection of samples, please also visit hatthhtt:///eee /aseee///dddfffsuuaeeee wwa.fussiongiitutteee
from previous work, and Section G
A MORE DETAILS ON CALIBRATING AND FILTERING REALESTATEIOK
To calibrate RealEstate 10K, we predict single length scale per scene with the help of recent zero-shot model for monocular depth (Saxena et al. 2023) that predicts metric depth from RGB and FOV. For each image we compute metric depth using the FOV provided by COLMAP. The length-scale is then obtained by regressing the 3D points obtained by COLMAP to metric depth at image locations to which the COLMAP points project. An L.I loss provides robustness to outliers in the depth map estimated by the metric depth models. The mean and variance of the per-frame scales yields per-sequence estimator. We use the variance as a simple measure confidence to identify scenes for which the scale estimate may not be reliable, discarding ~30% of scenes with the highest variance. Table shows the importance of filtering out scenes with less reliable calibration. The calibrated dataset will be made public to facilitate quantitative comparisons.
(L)
FDD (J)
(V)
TSED (7)
SIMD,..(I)
S/MD..())
LPIPS (.)
PSNR (1)
SSIM (1)
GRETOK
filtering)
(1,099)
16.19
4DMMR (wi filtering)
31,23
306.3
195.1
0.9974 (1.000)
1,023 (1,075)
0.3029 (0.3413)
0.2630
18.09
0.5349
Nean:Nett zcro-shii
4DMM-R fittriing)
24.35
232.0
143.1
w/a (0.9815)
18811 (1.774)
1.639 (1.512)
0.1990
20.15
0.6436
filtering)
130.6
(0.9815)
(1.177
(1.550)
0.1809
21,25
L. zern-shot
4DiM-R (no fiteriing)
62.55
369.5
848.5
0.9167 (0.9962)
0.9214 (0 (0.8844)
0:2194 (0.
0,5412
11.48
0.88
4DIM-R (wi' filtering)
63.48
353.2
841.6
0.9639 (0,9962)
0,9265 (0.8951)
0.2011 (0.1933)
0.5403
11.55
0.1444
Table 6: Data filtering ablation. We show the importance of discarding scenes where scale calibra- tion is least reliable. Beyond the clear impact on fidelity, metric scale alignment itself improves by wide margin. This is quantitatively shown through the reference-basee image metrics (LPIPS, PSNR, SSIM) which favor content alignment, as discussed Section 4
B PUSHING PARETO FRONTS WITH MULTI-GUIDANCE
To assess the effectiveness of multi-guidance, we conducted a two-stage evaluation. First, we evaluated samples generated with various 'standard' guidance weights, where all conditioning variables (image, pose, and timestamp) were treated uniformly (illustrated in blue in Figure 7 Through qualitative and quantitative analysis, we identified the optimal standard guidance weight. Next, while holding this image guidance weight constant, we systematically varied the pose or timestamp guidance weights (shown in red). This procedure was applied to two tasks: single-image 3D generation using the cRE10K dataset, and video extrapolation from the first two frames using the DAVIS dataset. As demonstrated in Figure our findings indicate that for video extrapolation, slightly stronger timestamp guidance enhances performance in both dynamics and FID. For 3D generation, a slightly stronger pose guidance improves pose alignment (measured by TSED) with minimal impact on FID.
16

---

Published as a conference paper at ICLR 2025
Hc10k 1 image to
Davis 2-image to-videa (extrapolation)
#0W-3.0
6M-10
21.5
30.5
30.5
@w-1.
110
29.5
00-
WM-1.125
2.
30- 2.002.0
gw-1 2.
20.5
1-15
1.125
28.5
64
40N-10
keypaint
difference
Figure 7: Multi-guidance ablation. Left: Pareto plot between FID and TSED (with threshold 1) for the single image to 3D task on the RealEstate 10K dataset. Right: Pareto plot between FID and keypoint distance for the video extrapolation task from the two initial input frames on the Davis dataset. First, shown in blue, standard guidance, denoted with "gw", is swept. Iee., all conditioning variables have the same guidance weight. Then, shown in red, pose guidance (left) and timestamp guidance (right), denoted "pgw" and "tgw" respectively, are varied starting from the Pareto-optimal configuration in the initial sweep, leaving the guidance weights of other variables fixed. We find that overemphasizing pose guidance only helps marginally. likely because even without guidance the worst scores for pose alignment are already near near-perfect (2 98% of keypoint matches are less than pixel away from the epipolar line). And in the case of timestamp guidance, we find that this dramatically improves the presence of temporal dynamics generated videos with very small trade on FII) score.
C FURTHER DETAILS ON NEURAL ARCHITECTURE
UVIT backbone. As hinted by Figure 2 4DiM uses UVIT architecture following Hoogeboom et al. (202)). Compared to the commonly employed UNet architecture (Ronneberger et al.] 2013)). UVIT uses transformer backbone at the bottleneck resolution with no convolutions, leading to improved accelerator utilization. Information across frames mixes exclusively in temporal attention blocks, following (Ho et al.] (0226). We find that this key to keep computational costs within reasonable limits compared to, say. 3D attention (Shi et al., 2023). Because the temporal attention blocks have a limited sequence length (32 frames), we insert them all UViT resolutions. To mitigate the quadratic cost of attention, we use per-frame self-attention blocks only at the 16x16 bottleneck resolution where the sequence length is the product of the current resolution's 's dimensions. And further minimize the amount of activations stored for backpropagation, we use only two UNet blocks when the resolution at 32x or beyond, with hidden sizes (channels) decreasing in powers of two to a minimum of 256. Convolutional kernels at the 32x resolution have 3x3 kernels, and we increase the kernel size additively by 2 for higher resolutions to a maximum of 7x7. Our base model uses 16 transformer blocks at the bottleneck resolution of hidden size 128 with 16 heads, and our super-resolution model uses 8 transformer blocks of hidden size 128 with 8 heads.
Transformer block design. Our transformer blocks combine helpful choices from several prior work. We use parallel attention and MLP blocks following Dehghani et al. (2023), as we found in our early experiments that this leads to slightly better hardware utilization while achieving almost identical sample quality. We also inject conditioning information in a similar fashion to Peebles & Xie (2023), though we use fewer conditioning blocks due the use of the parallel attention + MLP. We additionally emply query-key normalization following the findings of Gilmer et al. for improved training stability, but with RMSNorm (Zhang & Sennrich 2019) for simplicity.
Conditioning, We use the same positional encodings for diffusion noise levels and relative times- tamps as (Saharia et al., 2022b). For relative poses, we follow 3DiM (Watson et al., 2022). ) and condition generation with per-pixel ray origins and directions, as originally proposed by SKT (Sajjadi jet al. 2022) in a regressive setting. Xiong et al. (0033) has compared this choice to conditioning via encoded extrinsics and focal lengths a-la Zero- 1 (Liu et al. 2023b), finding it advantageous. While a thorough study of different encodings is missing in the literature, we hypothesize that rays natural choice they encode camera intrinsics in way that is independent of the target resolution, yet gives the network precise, pixel-Level information about what contents of the scene are visible and which outside the field of view and therefore require the model to extrapolate. Unlike
17

---

Published as a conference paper at ICLR 2025
Plucker coordinates, which parametrize lines in 3D space, rays additionally preserve information about camera positions, which is key to deal with occlusions in the underlying 3D scene.
D FURTHER DETAILS ON COMPUTE, TRAINING TIME, AND SHARDING
We train 8-frame 4DiM models for ~ 1M steps. Using 64 TPU v5e chips, we achieve throughput of approximately 1 step per second with a batch size of 128. All 8-frame models we train (i.e. base and super-resolution models) use this batch size, and we train all models with an Adam (Kingma & Ba 2014) learning rate of 0.0001 linearly warmed up for the first 10,000 training steps, which we found was the best peak value in early sweeps. No learning rate decay is used. We follow Ho et al.) (2020) and keep exponential moving average of the parameters with decay rate of 0.9999 to use at inference time for improved sample quality.
Available HBM is maximized with various strategies: first, we use 6 activations (but still use float32 weights to avoid instabilities). We also use ESDP (i.e., zero-redundancy sharding (Rajbhandari ct al. (2200) with delayed and rematerialized all-gathers). We then finetune this model to its final 32-frame version. This strategy allows us to leverage large batch-size pre-training, which is not possible with too many frames because the amount of activations stored for backpropagation scales linearly respect the number of frames per training example. The model is finetuned with the same number of chips, albeit only for 50,000 steps and batch size of 32. This allows to shard the frames of the video and use more than one chip per example at batch size 1. When using temporal attention, we simply all gather the keys and values and keep the queries sharded over frames (one of the key insights from Ring Attention (Liu et al. (2023a)). though we don't t decompose the computation of attention further we find we have sufficient HBM to fully parallelize over all- aal-ggatereed keys and values. FSDP is also enabled on the frame axis to maximize HBM savings so the number of shards truly the number of chips (as opposed to the number of batch-parallel towers). This is possible because frame sharding requires an all-reduce by mean the loss over the frame axis. Identically to zero-redundancy sharding, this can be broken down into a reduce-scatter followed by an all-gather.
E FURTHER DETAILS ON PROPOSED METRICS
We follow Yu al. 2023a) and compute TSED only for contiguous pairs in each view trajectory, and discard sequences where we find less than 10 two-view keypoint matches. We use threshold of 2.0 all our experiments. For our proposed SfM distances, in order to get scale- inver-inv metric. we first align the camera positions predicted by COLMAP by relativizing the predicted and original poses with respect to the first conditioning frame. This should resolve rotation ambiguity. Then, we resolve scale ambiguity by analytically solving for the least squares optimization problem that aligns the original positions to re-scaled COLMAP camera positions. Finally, we report the relative error in positions under the L2 norm, i.e., we normalize by the norm of the original camera positions.
F FURTHER DETAILS ON BASELINES FROM PRIOR WORK
For PNVS, the conditioning image is the largest square center crop of the original conditioning frame, resized 256 256 pixels. This is the same preprocessing procedure used by all our 4DiM models. PNYS generates output frames sequentially (one at a time), using 2000 denoising steps for each frame. All 4DiM samples are produced with 256 denoising steps.
For MotionCtrl, we use the checkpoint based on Stable Diffusion, which generates 14 frames conditioning on one image and 14 poses. We set the sampling hyper-parameter speed to use 128 ddim denoising steps, and keep the other hyper-paun parameters as default values in the released script from MotionCtrl. To avoid any loss of quality due to an -uttt---istiibuttinn resolution or aspect ratio, we use the resolution 576 x 1024, i.e. the same resolution as MotionCtrl was trained on. RE10K images already have the desired aspect ratio. LLFF images have resolution 756 x 1008; we are thus forced to crop them to 567 1008 so they have the same aspect ratio MotionCtrl was trained with. For comparability with 4DiM and PNVS, we postprocess MotionCtrl samples at 567 x 1008 by taking the largest center crop and then resizing to 256 x 256. Note that, in order to preserve the aspect ratio of LLFF, the resulting square outputs are for a smaller region than 4DiM and PNVS, hence the gray padding for MotionCtrl LLFF samples in Figure
18

---

Published as a conference paper at ICLR 2025
Input
ZeroNYS
4DMM
ZeroNVS
4DIM
4DMM ZeroNVS
Figure 8: Qualitative comparison of 360 generation between 4DiM and ZeroNVS. 4DiM achieves superior sharpness and richer content.
For ZeroNVS, Both input and output images are 256 x 256. We use default hyper-parameters as in the open sourced configuration, except the 'scene scale' parameter used for dealing with length scale ambiguity. We sweep over scale in [0.7, 1.0, 1.2, 1.1,1,5,,5 1.5, 2.0, 3.0, 5.0] and choose 3.0 for REI0K and 1.5 for LLFF
its diffusion model evaluate novel view synthesis REIOK and LLFF.
G QUALITATIVE COMPARISON OF 360 GENERATION VS ZERONVS
employ the ZeroNVS pipeline with score distillation sampling (SDS) to generate 360 views conditioned single image. For each scene, we use the default seene-generation configuration as open sourced by ZeroNVS, as adjusting camera camera_distance and elevation angle yield worse and incoherent generation. We adjust the scale parameter for each scene, and use 1.0, 3.0, 3.0 for the three scenes in Figure 8 Due to the computationally intensive nature of 360 generation with ZeroNVS (2-3 hours per scene), exhaustive hyperparameter tuning is impractical. We also observe another limitation: volume rendering will fail if the camera positions are not allowed to vary (or even if camera_ camera_distance distance is too small, e.g., 0.5), i.e., it only employ rotation. This occurs because there is no parallax and it is therefore not possible to infer depth/density (rotation alone only warps pixels). Figure presents comparison between 4DiM and ZeroNVS outputs, showing the superior sharpness and richer content achieved by 4DiM.
feasiable keep the camera position fixed and
H COMPARISONS OF 4DIM VS PRIOR FRAME INTERPOLATION METHODS
We compare 4DiM against prior works on the task frame interpolation (generating a video with large amounts of motion given the first and last frames). We the Davis dataset the benchmark and consider generative baseline, LDMVFI (Danier et al.) 2024), as well as the well-known (but non-generative) works FILM (Reda al. 2022) and RIFE (Huang et al.) 2022)
Below, we report FID scores of the middle frame for all methods, in order to consider the most challenging outputs and evaluate with the same number of samples despite that 4DiM generates 4x
19

---

Published as a conference paper at ICLR 2025
more frames. We take the immediately previous frame to the middle frame in the case where model generates an even number of frames.
Method
FID
RIFE
57.68
FILM
68.88
LDMVFI
56.28
4DiM
52.89
Input views
FILM
RIFE
LDMVFI
4DiM
Figure 9: Comparison against frame interpolation baselines.
Qualitatively (Fig. 9. we find that 4DiM competitive against all of these baselines, and does much better job when the amount of motion is large. Non-generative baselines suffer from severe artifacts in these cases.
I ABLATION FOR CO-TRAINING WITH VIDEO DATA
In our main evaluations, we find that 4DiM trained without video performs slightly better on in- distribution evaluations (e.g., RealEstate 10K). This is not surprising much of the capacity of the model now needs to be spent to model a more diverse data distribution. Still, to validate our claim that co-training with video is of paramount importance, is necessary consider -fit--ddistiiutioon qualitative comparisons, Here it becomes evident that model trained without video performs much worse. In particular, such model does not produce plausible geometry for unknown objects (Fig. T0 whereas the model trained with large-scale video data does not suffer from this issue.
Input image
view vidco)
Figure 10: Ablation for co-training with video. We find this is key for geometric understanding in the wild. The model trained w/o video data infers incorrect geometry frequently in the LLFF dataset.
20

---

Published as a conference paper at ICLR 2025
J ADDITIONAL 4DIM SAMPLES
We include more
4DiM samples below, though we strongly encourage the reader to
browse our website: https: //4d https::/4d-diffuii.oo ceiffusiinngithub.... io
Ingut
Output
Figure 11: More 4DiM samples from the RealEsttaee00 dataset.
21

---

Published as a conference paper at ICLR 2025
Circalur Pun
Imput
Input
Trunslate Backward
STOP
6106
Figure 12: More 4DiM samples with custom trajectories from the LLFF dataset.
22

---

Published as a conference paper at ICLR 2025
Input
Extrapolation
Input
Interpolation
Figure 13: More 4DiM samples of videos generated from 2 input images.
23

---

Published as a conference paper at ICLR 2025
Input
Octput handling reflections from screen, Boor, and water.
Imput
Output handling light, Fog. main
Figure 14: 4DiM can handle challenging scenarios such as reflection, light rays, fog, rain, etc.
24


Reviews:
[{"review_id": "rN9G2tfUNP", "reviewer": "Reviewer_3Lp3", "summary": "The paper presents \"4DiM\", a novel cascaded diffusion model aimed at 4D novel view synthesis (NVS) and able to generate scenes that can be viewed from arbitrary camera trajectories and timestamps. The model envolves several techniques to achieve this goal, including a super resolution model, Masked FiLM layers and different CFG params for conditioning variables. The model is trained on a combination of posed and 3D, 4D and video data, including RealEstate10K calibrated with a monocular depth estimation model. The design of training dataset improve the fidelity of generation and generalization ability to unseen images and camera pose trajectories. The proposed model is evaluated on in-distribution and out-of-distribution datasets with multiple metrics, some of which are proposed by the authors.", "questions": "There are some questions that need to be clarified:\n- The sampling efficiency and cost of the proposed model is not clear. How long does it take to generate a video compared to other methods? How much GPU memory is needed?\n- In appendix F, the authors mention the gray padding for MotionCtrl LLFF samples in Figure 3. I don't understand the purpose of \"in order to preserve the aspect ratio of LLFF\" since the sampling resolution is 567\\*1008. Why is it necessary to add gray padding as the sampling resolution is already higher than 256\\*256? Does the gray padding affect the metrics of the model?\n- gw, tgw and pgw in appendix Figure 7 are not explained in the main text.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The task of unposed 4D novel view synthesis is a challenging one, and the paper presents a novel approach to tackle it. Compared to previous works, the proposed model has two strengths:\n- The sampling method is well designed. The multi-guidance sampling enables controls on images; camera poses, and timestamps.\n- The dataset combination enhances the generalization ability of the model. The authors also give experimental results to explain the effectiveness of the dataset.\n- The authors propose novel metrics such as keypoint distance to measure temporal dynamics and improved TSED for 3D consistency, which are well-suited to evaluate the complex requirements of 4D NVS.\nMoreover, the ablation study is well-designed, and the results are well-explained. The paper is well-written and easy to follow.", "weaknesses": "Although the paper has several strengths, there are some weaknesses that need to be addressed:\n- The sampling resolution is not high enough. Compared with SOTA methods, ViewCrafter generates videos with higher resolution (576x1024), while the proposed method only generates 256x256 resolution videos even with a super resolution model. Is it possible to generate higher resolution videos with shorter sequence length or smaller training batch size?\n- The discussion of 360-degree videos is not enough. The model is not evaluated on 360-degree datasets, MipNeRF360 for example, which is a common task in NVS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZVo9WmeTc", "reviewer": "Reviewer_m6rk", "summary": "This paper proposes 4DiM, a cascaded diffusion model for 3D and 4D novel view synthesis. By a specially designed network architecture and sampling procedure, 4DiM enables joint training with posed and unposed video data, which results in better generalization and output fidelity. This also allows joint camera pose and time control in 4D novel view synthesis, making 4DiM a general model for various tasks like image-to-3D, two-image-to-video (interpolation and extrapolation), and pose-conditioned video-to-video translation. Qualitative and quantitative results show that the 4DiM performs favorably against priors art on static and dynamic scene generation.", "questions": "Q1: I’m wondering if the authors have any explanation for why adding unposed video data lead to worse performance on certain metrics (see S3).\n\nQ2: Following W2, do the authors have any ideas on how to improve the texture fidelity in these cases?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1: General model\n4DiM enables joint training with posed and unposed video data, which not only improves the generalization and output fidelity, but also allows joint camera pose and time control in 4D novel view synthesis. It is shown to perform well on various tasks such as image-to-3D, two-image-to-video (interpolation and extrapolation), and pose-conditioned video-to-video translation.\n\nS2: Better 3D consistency and pose alignment\nBoth qualitative and quantitative results show a consistent improvement over prior methods in terms of 3D consistency and camera pose alignment. Although it is a bit surprising that joint training with unposed video data leads to some worse metrics on in-domain data like cRE10k and ScanNet++ (Table 3).\n\nS3: Good writing\nThe paper is well-written and easy to follow. The ablation studies also give good insights on where the performance gain comes from.", "weaknesses": "W1: Motion realism and temporal consistency\nWhile the model shows great 3D consistency and pose alignment, the fidelity and temporal consistency of dynamic objects seems less impressive. In most dynamic scene results, the moving objects (cars, tires, animals, etc) either have unrealistic motion or temporal artifacts. This can probably be improved by including dynamic object data like Objaverse for training. It would also be good to show more results on single-image-to-4D.\n\nW2: Cartoonish texture\nSome results (especially in single-image-to-4D and long trajectory generation) have quite cartoonish texture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLzZNFOg7Q", "reviewer": "Reviewer_ossm", "summary": "The paper introduce a new video generative model that has both camera and time control. This is achieved by carefully assembling static/dynamic training data and a model design that help leverage missing condition signals (e.g., camera poses / time variation). The qualitative results are promising, showing good disentanglement between camera and time. The quantitative result is better than existing methods on RE10k/LLFF on novel view synthesis and camera controllability.", "questions": "- The paper compared to MotionCtrl and found it has trouble following pose controls. Is there a hypothesis for this observation? Is it due to lack of metric scale training poses, or are there other underlying factors? What happens if MotionCtrl is trained on cRE10K that has metric scale poses?\n- What is the difference between the proposed multi-guidance and InstructPix2Pix [B] Eq 3.\n- Will the data/model be open-sourced?\n- Related works. [A] can be discussed as the setup is close although they do not claim time control.\n\n[A] VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control\n[B] InstructPix2Pix Learning to Follow Image Editing Instructions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- They introduce masked FiLM layers, which avoids misusing 0 conditions (1) during conditioning signal dropout and (2) due to missing data.\n- They introduce cRealEstate10K dataset to unify the scale of camera extrinsics, which reduces ambiguities during training and leads to more reliable camera control.\n- The qualitative results are promising, showing good disentanglement between camera and time. \n- The quantitative result is better than existing methods on RE10k/LLFF.", "weaknesses": "- It's unclear whether the problem should be called 4D NVS. One common interpretation of 4D is 3D+time. Under this interpretation, I don't think the proposed model can do 4D. With camera control, it can do 3D. With time control, it can do 2D+time. However, combining both is not equivalent to 3D+time. 3D+time would require generating multiple videos that are synchronized in time, or multiple \"frozen\" time videos with different t.\n- Data: The model is trained on mostly static data. The only source with non-static content is 1000 Google street views. How diverse is the data, and how much dynamic contents are there?  Does it include human motion and scene motion (e.g., tree, cars)?\n- Result/eval: The reported metrics are mostly for 3D NVS. The only metric related to dynamics is keypoint distance, which however, only assess how large the motions are, but not how good the motions are. Additionally, the evaluation protocol is not clear enough -- is camera parameters the same for all the frameds when generating the videos? Otherwise, the camera motion would produce large keypoint distance.\n- Disentanglement. The ability of disentangling camera motion and scene dynamics is not fully tested. It would help to report metrics that quantitatively measures the disentanglement. For example, whether fixing camera and changing t would introduce background motion; whether fixing t and changing camera will introduce object motion (e.g., through epipolar geometry).\n- Some claims can be supported by experiments. \n  - The benefit of masked FiLM layers.\n  - In the supplement, \"Unlike Plucker coordinates, rays additionally preserve camera positions which is key to deal with occlusions in the underlying 3D scene.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zTBFRiaepw", "reviewer": "Reviewer_SN35", "summary": "The paper proposes a two-stage diffusion model that generates image sequences conditioned on camera pose and time. To address the lack of pose-aware image sequence datasets, the paper trains the model on a mixture of datasets (posed and unposed) with additional data preprocessing steps and a modified architecture. The qualitative results demonstrate that the model achieves better generalizability across both in-domain and out-of-domain scenes with better alignment with the input camera parameters.", "questions": "1. It appears that 4DiM is trained from scratch rather than fine-tuned on a pre-trained diffusion model. Is there a particular reason for this choice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The qualitative results look promising, showing generalizability to new scene types and arbitrary camera trajectories while outperforming the baselines.\n2. The paper incorporates different sources of dataset to achieve generalizability of the model. \n3. The paper proposes a model architecture, Masked FiLM, to incorporate CFG training with incomplete conditional inputs.\n4. The evaluation metrics comprehensively assess various aspects of a pose-aware generative model.", "weaknesses": "1. In Tab.1, 4DiM shows discrepancy in reconstruction-based metrics (LPIPS, PSNR, SSIM) between in-domain (RE10K) and out-of-domain (LLFF) samples. While the authors mention that the reconstruction metrics favor the blurry images generated by the baseline method, this implies that the proposed method cannot accurately reconstruct images at specific camera poses. This can be seen as contradicting the claims in the main text, such as \"precise camera control\" and \"accurate control over camera pose.\" Similar trend is observed in Tab.2. The reasoning and explanations for the results on out-of-domain data could be presented in more detail. \n\n2. Training on a mixed dataset is not a novel approach and has been proposed in previous works on different domains: MiDaS for depth estimation and dust3r in 3D reconstruction. Highlighting the key differences could help the readers to better understand the novel contributions.\n\n3. Comparisons to previous works on multi-frame conditioning could be strengthened. When more than a single input image is given, one can consider generating sequence of images using frame-to-video generative models (e.g., FILM) or autoregressive models (e.g., Show-O). Similarly, comparisons to previous works in image-to-panorama generation (e.g., MVDiffusion, PanoDiff) are not presented. The effectiveness of the proposed method would be more convincing with the addition of relevant baselines for each application.\n\n4. The ablation study in Table 3 shows that the effect of co-training with video is not significant, and qualitative comparisons between the two cases are not presented (\"see Supplementary Material H for more samples\"). The authors could provide the more explanations for interpreting the ablation study. \n\nSome missing relevant works: Multi-Guidance (e.g., Instruct-Pix2Pix, StyleCrafter), Training on a Mixture of Datasets (eg., MiDaS, dust3r).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)