DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos
Authors: Wen-Hsuan Chu
Lei Ke
Katerina Fragkiadaki
Venue: NeurIPS 2024 poster (2024)
Keywords: 4D Scene Generation;  Video-to-4D Generation
Primary area: machine_vision
TL;DR: We propose DreamScene4D, the first video-to-4D scene generation approach to produce realistic 4D scene representation from real-world multi-object videos.

Abstract: View-predictive generative models provide strong priors for lifting object-centric images and videos into 3D and 4D through rendering and score distillation objectives. A question then remains: what about lifting complete multi-object dynamic scenes? There are two challenges in this direction: First, rendering error gradients are often insufficient to recover fast object motion, and second,  view predictive generative models work much better for objects than whole scenes, so, score distillation objectives cannot currently be applied at the scene level directly. We present DreamScene4D, the first approach to generate 3D dynamic scenes of multiple objects from monocular videos via 360-degree novel view synthesis. Our key insight is a "decompose-recompose" approach that factorizes the video scene into the background and object tracks, while also factorizing object motion into 3 components: object-centric deformation, object-to-world-frame transformation, and camera motion. Such decomposition permits rendering error gradients and object view-predictive models to recover object 3D completions and deformations while bounding box tracks guide the large object movements in the scene. We show extensive results on challenging DAVIS, Kubric, and self-captured videos with quantitative comparisons and a user preference study. Besides 4D scene generation, DreamScene4D obtains accurate 2D persistent point track by projecting the inferred 3D trajectories to 2D. We will release our code and hope our work will stimulate more research on fine-grained 4D understanding from videos.

DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos
Wen-Hsuan Chul, Lei Kei, Katerina Fragkiadaki Carnegie Mellon University {wenhsuac, leik, katef cmu.
attps://dreamscant/ hit.eeeeeeeeg...g........ io/
Abstract
View-pre -ew-predictive generative models provide strong priors for lifting object-centric images and videos into 3D and 4D through rendering and score distillation objec- tives. A question then remains: what about lifting complete multi-object dynamic scenes? There are two challenges in this direction: First, rendering error gradients are often insufficient recover fast object motion, and second, view predictive generative models work much better for objects than whole scenes, so, score distil- lation objectives cannot currently be applied at the scene level directly. We present DreamScene4D, the first approach to generate 3D dynamic scenes of multiple objects from monocular videos via 360 novel view synthesis. Our key insight is a cettooppos--rccopponr background and object tracks, while also factorizing object motion into com- ponents: object-centric deformation, object-to- woo-tooo-ood-aame transformation, and camera motion. Such decomposition permits rendering error gradients and object view-predictive models recover object 3D completions and deformations while bounding box tracks guide the large object movements in the scene. We show extensive results on challenging DAVIS, Kubric, and self-captured videos with quantitative comparisons and a user preference study. Besides 4D scene genera- tion, DreamScene4D obtains accurate 2D persistent point track by projecting the inferred 3D trajectories to 2D. We will release our code and hope our work will stimulate more research on mne-graieed 4D understanding from videos.
approach that factorizes the video scene into the
Introduction
Videos are the result of entities moving and interacting in 3D space and over time, captured from a moving camera. Inferring the dynamic 4D scene from video projections in terms of complete 3D object reconstructions and their 3D motions across seen and unseen camera views is challenging problem in computer vision. It has multiple important applications, such 3D object and scene state tracking for robot perception [16] 36), action recognition, visual imitation, digital content creation/simuultiin and augmented reality.
Video-to-4D a highly under-constrained problem since multiple generation hypotheses project to the same video observations. Existing 4D reconstruction works 339 33 23 28
6
25 mainly focus representation
on the visible part of the scene contained in the video by learning a differentiable
that is often a neural field [31 a set of 3D Gaussians 200 with temporal deformation. What about the unobserved views of the dynamic 3D scene? Existing 4D generation works utilize generative models to constrain the appearance of objects in unseen views through score distillation losses.
TD [46] 2 53 24 I or image-to-4D (64, 43
58
66 setups take a single text prompt or 14
image input to create 4D object. Several works
43
32
61 62 explore the video-to-4D
Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input Frames
Reference View Rendering
Novel View
Novel View
Figure 1: DreamScene4D extends video-to-4D generation to multi-object videos with fast motion.
We present rendered images and the corresponding motions from diverse viewpoints at different timesteps using real-world DAVIS [37 videos with multiple objects and large motions.
setup, but these methods predominantly focus on videos containing a single object with minor 3D deformations, where the object deforms in place without large motion in the 3D scene. This focus arises because current generative models perform significantly better at predicting novel views of individual objects (26) than multi-object scenes. Consequently, score distillation objectives for 3D object lifting are difficult to apply directly at a scene level. Also, optimization difficulty arises when neural fields 3D Gaussians are trained to model large temporal deformations directly. This limits their practical real-world usage where input videos depicting real- complex scenes containing multiple dynamic objects with fast motions, as illustrated in Figure 4
In this paper, we propose DreamScene4D, the first video-to-4D scene generation approach to produce realistic 4D scene representation from a complex multi-object video with large object motion or deformation. To 360 synthesize novel views for multiple objects of the scene, DreamScene4D proposes a seccompos--reomppsse background scene, where each is completed across occlusions and viewpoints, then recomposed to estimate relative scales and rigid thjeel-to-world transformations in each frame using monocular depth guidance, so all objects are placed back in a common coordinate system.
strategy. A video is first decomposed into objects and the
To handle fast-moving objects, DreamScene4D factorizes the 3D motion of the static object Gaussians into 3 components: camera motion, 2) object-centric deformations, and 3) an object-centric world frame transformation. This factorization greatly improves the stability of the motion optimization process by leveraging powerful object trackers to handle large motions and allowing view- predictive generative models to receive object-centric inputs that are in distribution. The camera motion estimated by re-rendering the static background Gaussians match the video frames.
We show the view renderings at various timesteps and diverse viewpoints of DreamScene4D using challenging monocular videos from DAVIS (37 in Figure [ DreamScene4D achieves significant improvements compared to the existing SOTA video-to-4D generation approaches (43) 17] on DAVIS, Kubric [15), and our self-captured videos with fast moving objects (Figures 4.. To evaluate the quality of the learned Gaussian motions, we measure the 2D endpoint error (EPE) of the inferred 3D motion trajectories across occlusions and show that our approach produces accurate and persistent point trajectories in both visible views and synthesized novel views.
Related Work
Video-to-4D Reconstruction Dynamic 3D reconstruction extends statie reconstruction to dynamic scenes with the goal of 3D lifting the visible parts of the video. Dynamic NeRF-based

---

methods B3 33 23 27 5 extend NeRF [1 to dynamic scenes, typically using grid or voxel- or learning a deformation field 6 [3] that models the dynamic portions of an object or scene. Dynamic Gaussian Splatting 30 extends 3D Gaussian Splatring 20). where scenes are represented as 4D Gaussians and show faster convergence than NeRF-based approaches. However, these 4D scene reconstruction works [30 51 59 typically take videos where the camera has a large number of multi-view angles, instead of a general monocular video input. This necessitates precise calibration of multiple cameras and constrains their potential real- world applicability. Different from these works [4 [11] on mostly reconstructing the visible regions of the dynamic scene, DreamScene4D can 360 synthesize novel views for multiple objects of the scene, including the unobserved regions in the video.
based representations 28
Video-to-4D Generation In contrast to 4D reconstruction works, this line of research is most related by attempting to complete and 3D reconstruct a video scene across both visible and unseen (virtual) viewpoints. Existing text to image to 4D generation works 43 14
32
61
62] typically use score
distillation sampling (SDS) (38 to supply constraints in unseen viewpoints order to synthesize
full 4D representations of objects from single text [46 2 24 image [58] 66], or combination of both 664] prompts. They first map the text prompt or image prompt to a synthetic video, then lift the latter using deformable 3D differentiable NeRFs 23 or set of Gaussians [S]] representation. Existing
video-to-4D generation works [43 17 14 32
62 usually simplify the input video by assuming non-occluded and mloimov object while real-world videos with multiple dynamic objects inevitably contain occlusions. Owing to our proposed scene decoupling and motion factorization schemes, DreamScene4D is the first approach to generate complicated 4D scenes and synthesize their arbitrary novel views by taking real-world videos of multi-ohject scenes.
61
3 Approach
To generate dynamic 4D scenes of multiple objects from a monocular video input, we propose DreamScene4D, which takes Gaussian Splatting [10 as the 4D scene representation and leverages powerful foundation models to generalize to diverse zero-shot settings.
3.1 Background: Generative 3D Gaussian Splatting
Gaussian Splatting 20 represents a scene with a set of 3D Gaussians. Each Gaussian is defined by its centroid, scale, rotation, opacity, and color, represented spherical harmonics (SH) coefficients.
Generative 3D Gaussian Splatting via Score Distillation Sampling Score Distillation Sampling (SDS) (38) is widely used for text-to-3D or magge-to-31) tasks by leveraging a diffusion prior for optimizing 3D Gaussians to synthesize novel views. For 3D object generation, DreamGaussian [49] uses Zero-I-to- 1-to- (26). which takes reference view and relative camera pose input and generates plausible images for the target viewpoint, for single frame 2D-to-3D lifting. The 3D Gaussians the input reference view I are optimized by rendering loss and an SDS loss 38
V Ca E.TT,.P
OΦ
where is the timestep indices, w(r) is a weighting function for denoising timestep T, o (-) represents the Gaussian rendering function, 7 is the rendered image, eg (-) is the predicted noise from Zero-l- to-3, and is the added noise. We take the superscript p to represent an arbitrary camera pose.
3.2 DreamScene4D
We propose a cectooppss--rcomppos principle to handle complex multi-object scenes, As in Fi given a monocular video of multiple objects, we first segment and track 44 19 g 10 each 2D object and recover the appearance of the occluded regions (Section (....I. Next, we decompose the scene into multiple amodal objects and use SDS 38 with diffusion priors to obtain 3D Gaussian representation for each object (Section 3.2.2). To handle large object motions, we optimize the deformation of 3D Gaussians under various constraints and factorize the motion into three components (Figure3): the object-centric motion, an object-centric to world frame transformation, and the camera motion (Section 3.2.3) This greatly improves the stability and quality of the Gaussian
3

---

Decompased and Completed ObjectuGG
Predicted Depth
4D Scene Represemtation
Camera Motion
3 Co.
Inpul Frames
Object Mation
Dopth Based Compostion
Video Scene Decomposttin and Obj-Cantric Lifting
Motion Factorization
(c) 4D Scane Composttion
Figure 2: Method overview for DreamScene4D: (a) We first decompose and amodally complete each object and the background in the video sequence and use DreamGaussian 49 to obtain static 3D Gaussian representation. (b) Next, we factorize and optimize the motion of each object track independently, detailed in Figure 3 (c) Finally, use the estimated monocular depth to recompose the independently optimized 4D Gaussians into one unified coordinate frame.
optimization and allows new-predictive image generative models to operate under in in-distribution object-centric settings. Finally, we compose each individually optimized object to form a complete 4D scene representation using monocular depth guidance (Section 3.2.4..
3.2.1 Video Scene Decomposition
Instead of taking the video scene as whole, we first adopt mask trackers 44
segment 19
10
and track objects in the monocular video when GT object masks are not provided. From the monocular video and associated object tracks, we amodally complete each object track before lifting to 3D as in Figure [ To achieve zero-shot object appearance recovery for occluded regions of individual
object tracks, we build off of inpainting diffusion models [45] and extend it videos for amodal video completion. We provide the details of amodal video completion in the appendix.
3.2.2 Object-Centric 3D Lifting from World Frame
After decomposing the scene into individual object tracks, we use Gaussians Splatting 20 with SDS loss [38 49 lift them to 3D. Since novel-view generative models (26) trained on Objaverse (II]
are inherently object-centric, we take a different manner to 3D lifting. Instead of directly using the first frame of the original video, where the object areas may be small and centered, we create new object-centric frame I by cropping the object using its bounding box and re-scaling it. Then, we
optimize the static 3D Gaussians with both the RGB rendering on I and the SDS loss [38 in Eq
3.2.3 Modeling Complex 3D Motions via Motion Factorization
To estimate the motion of the first-frame lifted 3D Gaussians (G one solution like DreamGaus- sian4D [B] is to model the object dynamics by optimizing the deformation of the 3D Gaussians directly in the world frame. However, this approach falls short in videos with large object motion, as
the rendering loss yields minimal gradients until there is an overlap between the deformed Gaussians in the re-rendered frames and the objects in the video frames. Large motions thousands of 31) Gaussians also increase the training difficulty of the lightweight deformation network [13 6)
Thus, we propose to decompose the motion into three components and independently model them: 1) object-centric motion, modeled using learnable deformation network; 2) the object-centric to world frame transformation, represented by set of 3D displacements vectors and scaling factors; and 3) camera motion, represented by set of camera pose changes. Once optimized, the three components can be composed to form the object motion observed in the video.
Object-Centric Motion Optimization The deformation of the 3D Gaussians includes a set of learnable parameters for each Gaussian: 1) a 3D position for each timestep 14, = (jurt Nilt, pız)) 2) 3D rotation for each timestep, represented by a quaternion R. (qWe, qı,, q3t, gzt), and 3)

---

Gouzsians
Figure 3: 3D Motion Fac- torization. The 3D motion is decomposed into 3 compo- nents: 1) the object-centric de- formation, 2) the camera mo- tion, and 3) the object-centric to- fo-world frame transformation. After optimization, they can be composed to form the orig- inal object motion observed in the video.
Syree Fucas Frigh
Gausslans
Object-Cantric Deformtion
3D Oiflusion Prior
Buu?
SDS
Motion
Render
Lrigh
Objecc-Cantric World Frame Trenafarmaion
Imput Frames
3D scale for each timestep st (sz,, syt, s21). The RGB (spherical harmonics) and opacity of the Gaussians are shared across all timesteps and copied from the first-frame 3D Gaussians.
To compute the 3D) object motion in the object-centric frames, we take the cropped and scaled objects in the individual frames I, forming a new set of frames ī for each object. Following DreanGaussian4DD 45 we adopt K-plane [3] based deformation network Do(G ()) predict the 10-D deformation parameters (14, Re, 8t) for each object per timestep. We denote the rendered image at timestep under the camera pose p as if, and optimize Da using the SDS loss in Eq. as well as the rendering loss between f and īद for each frame under the reference camera pose r.
Since 3D Gaussians can freely move within uniformly-colored regions without penalties, the rendering and SDS loss are often insufficient for capturing accurate motion, especially for regions with near- uniform colors. Thus, we additionally introduce flow rendering loss ftow which is the masked LI loss between the rendered optical flow of the Gaussians and the flow predicted by an off-the-shelf optical flow estimator [54). The flow rendering loss only applies to the confident masked regions that pass a simple forward- backwadd flow consistency check.
Physical Prior on Object-Centrie Motion Object motion in the real world follows set of physics laws, which can be used to constrain the Gaussian deformations further. For example, objects usually maintain a similar size in temporally neighboring frames. Thus, we incorporate scale regularization
loss Lycate
Σl=
||s sell1. where penalize large Gaussian scale changes.
To preserve the local rigidity during deformations, we apply a loss Ligid to penalize changes the relative 3D distance and orientation between neighboring Gaussians following (30] We disallow
pruning and densification of the Gaussians when optimizing for deformations like 30
Object-to-world Frame Transformation We compute the translation A, = (Ar.t, Ay,t Ag.)) and scaling factor s that warps the Gaussians from the object-centric frame the world frame. The 2D bounding-box-baat
cropping and scaling (Sec (.e. 3.2.2) from the original frames to the object-centric frames can be represented as affine warp, which we use to compute and initialize Az, Ay,t, and sy for each object in each frame. 4., is initialized to O. We then adopt the rendering loss on the original frames Le instead of center-cropped frames I, to fine-tune A, with a low learning rate.
To further improve the alignment between renderings and the video frames, is essential to con- sider the perceptual parallax difference. This arises when altering the object's s 3D position while maintaining fixed camera perspective, resulting in subtle changes in rendered object parts. Thus, we compose the individually optimized motion components and jointly fine-tune the deformation network Dg and affine displacement A, using the rendering loss. This refinement process, conducted over a limited number of iterations, helps mitigate the parallax effect as shown in Figure [1 of the appendix.
Camera Motion Estimation We leverage differentiable Gaussian Splatting rendering to jointly reconstruct the 3D static video background and estimate camera motions. Taking multi-frame
inpainted background images Ih2 as input, we first use an off-the-shelf algorithm (50) to initialize the background Gaussians and relative camera rotation and translation {Re, T.} between frame 1 and frame I. However, the camera motion can only be estimated up to an unknown scale (60) as
there is no metric depth usage. Therefore, we also estimate a scaling term 3 for T,. Concretely, from
5

---

the background Gaussians G and {Ra, Tel, we find the 5 that minimizes the rendering loss of the background in subsequent frames:
Log
1 + T
o , R2, 8,Ir)
(2)
Empirically, optimizing a separate 3c per frame [3] yields better results by allowing the renderer to compensate for erroneous camera pose predictions.
3.2.4 4D Scene Composition with Monocular Depth Guidance
Given the individually optimized 4D Gaussians, we recompose them into unified coordinate frame to form a coherent 4D scene. As illustrated in Step (c) of Figure 2 this requires determining the depth and scale for each object along camera rays.
Concretely, we use an off-the-shell depth estimator [56] to compute the depth of each object and the background and exploit the relative depth relationships guide the composition. We randomly pick an object as the "reference" object and estimate the relative depth scale k between the reference object and all other objects. Then, the original positions μ! and scales s of the 3D Gaussians for the objects are scaled along the camera rays given this initialized scaling factor k: 14 C" (C" jue) * k and sy = 8t k. where (" represents the position of the camera. Finally, we compose and render the depth map of the reference and scaled object, and minimize the affine-invariant LI loss [56] 42 between the rendered and predicted depth map to optimize each object's scaling factor k:
d, d - d.
d; t(d) (dd)
Ldepth
HW
Here, dž and d, are the scaled and shifted versions of the rendered depth di and predicted depth d, t(d) is defined as the reference object's median depth and odd) is defined as the difference between the 90% and 10% quantile of the reference object. The two depth maps are normalized separately using their own t(d) and (d). Once obtain the scaling factor & for each object, can easily place and re-compose the individual objects a common coordinate frame. The Gaussians can then be rendered jointly to form scene-level 4D representation, as shown in Figure
4 Experiments
Datasets While there exist datasets used in previous video-to-4d generation works [17, they only consist of a small number of single-object synthetic videos with small amounts of motion. Thus, we evaluate the performance of DreamScene4D on more challenging multi-object video datasets, including DAVIS (37), Kubrie [15), and some self-captured videos with large object motion. We select a subset of 30 challenging real-world videos from DAVIS (37]. consisting of multi-object monocular videos with various amounts of motion. We further incorporate the labeled point trajectories from DAP-Vid-DAVEL [12] to evaluate the accuracy of the learned Gaussian deformations. In addition, we generated 50 multi-object videos from the Kubric [5] simulator, which provides challenging scenarios where objects can be small or off-center with fast motion.
Evaluation Metrics The quality of 4D generation can be measured in two aspects: the view rendering quality of the generated 3D geometry of the scene, and the accuracy of the 3D motion. For the former, follow previous works [17 45 and report the CLIP [4 and LPIPS (3 scores between novel-view rendered frames and the reference frame, and compute its average score per video. These metrics allow us to assess the semantic similarity between rendered and reference frames. We also conducted user study to evaluate the 4D generation quality for the DAVIS videos using two-way voting compare each baseline with our method, where 50% 50% indicates equal preference.
The accuracy of the estimated motion can be evaluated by measuring the End Point Error (EPE) of the projected 3D trajectories. For Kubric, we report the mean EPE separately for fully visible points and points that undergo occlusion. For DAVIS, we report the mean and median EPE [65), as the annotations only exist for visible points.
Implementation Details We run our experiments on one 40GB A100 GPU. We crop and scale the individual objects to around 65% of the image size for object lifting. For static 3D Gaussian

---

Consisten14D
Wio MF (DeamCaussiantD)
Wio MF CheanGausianne VSD)
DreamScenedi (Ours)
ConsistemAD
DreaGGuzsaaaD
Wio ME: Wo VSD
(Dream DreamiGaussiani CaursiandD VSD)
Wio MF
DreamSconed)
Figure 4: Video 4D Comparisons. We render the Gaussians at various timesteps and camera views. We denote Motion Factorization as MF and Video Scene Decomposition as VSD. Our method produces consistent and faithful renders for fast-moving objects, while DreamGaussian4D 4 (2nd row) and Consistent4D [17] (Ist row) produce distorted 3D geometry, blurring, or broken artifacts. Refer to our Supp. Materials for extensive qualitative comparisons.
optimization, we optimize for 1000 iterations with a batch size of 16. For optimizing the dynamic components, optimize for 100 times the number of frames with batch size of 10. More implementation and running time details are provided in the appendix.
4.1 Video to 4D Scene Generation
Baselines We consider the following baselines and ablated versions of our model:
(1) Consistent4D [17), recent state-of-the ahe method for 4D generation from monocular videos that fits dynamic NeRFs per video using rendering losses and score distillation.
(2) DreamGaussian4D 43 which uses dynamic 3D Gaussian Splatting like us for 4D generation
from videos, but does not use any video decomposition motion factorization as DreamScene4D..
This is most related to our method.
7

---

Table 1: Video to 4D Scene Generation Comparisons. We report the CLIP and LPIPS scores in Kubric [5] and DAVIS 37. For user preference, A% B% denotes that A% of the users prefer the baseline while B% prefer ours in two-way voting. We denote methods with Video Scene
Decomposition as VSD and methods with Motion Factorization as MF.
DAVIS
Kubric
Method
VSD
MF
CLIP 1
LPIPS 4
User Pref.
CLIP 1
LPIPS 1
Consistent4D 17
82, 14
0.141
28.3% / 71.7%
80.46
0.117
DreamGaussian4D 43
X
X
77.81
0.181
22.1% / 77.9%
73,45
0,146
DreamGaussian4D w/ VSD
7
X
81.39
0.169
30.4% / 69.6%
79.83
0.122
DreamScene4D (Ours) w/o L fiow w/o Erigid and Lecate
85.09
0.152
85.53
0.112
84.94
0.152
86.41
0.113
83.24
0.153
84.07
0.115
(3) DreamGausiandD+VSDD (Video Scene Decomposition). We augment DrcamGaussian4D with VSD, where we segment every object before 4D lifting, and recompose them. The main difference between this stronger variant and our DreamScene4D is the lack of motion factorization.
(4) DreamScene4D ablations on losses. We also ablate without flow losses and regularization losses.
4D Generation Results on DAVIS & Kubric We present the 4D generation quality comparison in T where our proposed Video Scene Decomposition (VSD) and Motion Factorization (MF) schemes greatly improve the CLIP and L.IPS score compared to the input reference images. From the user study, we can also observe that DreamScene4D is generally preferred over each baseline. Compared to the baselines, these significant improvements are mainly due to our proposed motion factorization, which enables the SDS loss to perform in object-centric manner while reducing the training difficulty for the lightweight Gaussian deformation network in predicting large object motions. We also show qualitative comparisons of 4D generation multi-object videos and videos with large motion in Figure 4 where both variants of DreamGaussinn4D [43] and Consistent4D [17] tend to produce distorted 3D geometry, faulty motion, or broken artifacts of objects. This highlights the applicability of DreamScene4D to handle weal-world complex videos.
4D Generation Results on Self- Celf-Captured Videos We also captured some monocular videos with fast object motion using smartphone to test the robustness of DreamScene4D,, where objects can be off-center and are subject to motion blur. We present qualitative results of the rendered 4D Gaussians in the right half of Figure 4 Even under more casual video capturing settings with large motion blur, DreamScene4D can still provide temporally consistent 4D scene generation results while the baselines generate blurry results or contain broken artifacts of the objects.
4.2 4D Gaussian Motion Accuracy
Baselines and Ablations Design To evaluate the accuracy of the 4D Gaussian motion, we consider DreamGaussian-D] [43] as the baseline, since extracting motion from NeRF-based methods [I7) is highly non-trine non-trivial. In addition, we compare against PIPS++ [65] and CoTracker [18], two fully- supervised methods explicitly trained for point-tracking, serving upper bounds for performance.
4D Motion Accuracy in Video Reference Views In Table 2 we tabulate the motion accuracy com- parison, where DreamScene4D achieves significantly lower EPE than the baseline DreamGaussian4D on both the DAVIS and Kubric datasets. We noted that conventional baselines often fail when objects are positioned near the edges of the video frame or undergo large motion. Interestingly, the motion accuracy of DreamScene4D outperforms PIPS+ (55, despite never being trained on point tracking data, as in Figure 6 This is due to the strong object priors of DreamScene4D, as the Gaussians adhere to remaining on the same object it generates and their motion is often strongly correlated. 4D Motion Results
Generated Novel Views An advantage of representing the scene using 4D Gaussians is being able to obtain motion trajectories in arbitrary camera views, which we visualize in Figure 1 and Figure & in the appendix. DreamScene4D both generate a 4D scene with consistent appearance across views and produce temporally coherent motion trajectories in novel views.
4.3 Reliance on External Depth Estimation
DreamScene4D uses the estimated monocular depth to infer the relative depth relationships/scales of each independently optimized 4D Gaussians and recompose them into one unified coordinate frame. To provide more insights into the robustness of depth estimation errors. replaced the
8

---

Table 2: Gaussian Motion Accuracy. We report the EPE in Kubric [15 and DAVIS 37 12] We denote methods with our Video Scene Decomposition in column VSD and methods with 3D Motion Factorization in column MF. Note that CoTracker is trained on Kubric.
DAVIS
Kubric
Method
VSD
MF
Mean HPR
Median EPR +
EPE (vis)
RPR (OCc). +
(a) Not trained on point tracking data
Baseline: DreamGiaussian4D 43
x
26.65
6.98
101.79
120.95
w/ VSD
x
20.95
6.72
85.27
92.42
DreamScene4D (Ours)
8.56
4.24
14.30
18.31
w/o fim
10.91
3.83
18.54
24.51
w/o Erigid and Ceoate
10.29
4.78
16.21
22.29
(b) Trained point tracking data
PIPS++
19.61
5.36
16.72
29.65
CoTracker
18
7.20
2.08
2.51
6.75
T=0
T=t1
T=12
T=t3
Cloud
Point Cloud
Point Cloud Vissa Rzation (Side View
Figure 5: Grouping visualization of Gaussians. The grouping of the point cloud visualized as colored point clouds from different camera views. The spatial relationships between objects are preserved after the composition.
Depth-Anything model with MIDAS (4) weaker depth prediction model, well as the newly released Depth-Anything v2 (57]. Furthermore, we added random noise to perturb the outputs of Depth-Anything v1 in various magnitudes.
Since depth estimation is only used for scene composition, we conducted experiments on DAVIS videos only to emphasize the differences and summarize the results in Table 3 While different depth predictions result in objects being placed in slightly different scene locations, we note that existing SOTA depth prediction models (such as Depth Anything series) meet the requirements in most cases, and the rendered quality of the scene will not deteriorate much as long as the relative depth ordering of the objects is correct, which even holds when we add small amount of noise to the predicted depth (second row) or use a weaker model like MiDAS (fourth row).
4.4 Limitations
Despite the exciting progress and results presented in the paper, several limitations still exist: (1) The SDS prior fails generalize videos captured from a camera with steep elevation angles. (2) Scene composition may fall into local suboptimas if the rendered depth of the lifted 3D objects is not well

---

Table 3: Generation Results Using Different Depth Estimators. We report the PSNR and LPIPS on multi-object DAVIS videos using different depth estimators.
Method
PSNR t
LPIPS
Depth-Anything vl (original)
83.71
0.169
Depth-Anything v1 + Noise (10%)
83.67
0.171
Anyth-nn v1 Noise (25%)
83.48
0.174
MiDAS v3.1
83.34
0.176
Depth- Annthin v2
83.76
0.169
PIPS++
CoTracker
Ours
Figure 6: Motion Comparisons. The 2D projected motion of Gaussians accurately aligns with dynamic human motion trajectory in the video, where the point trajectories estimated by PIPS+ (65) tend to get "stuck" in the background wall. For CoTracker [18), partial point trajectories are mixed up, where some points the chest region (yellow/green) ending up in the head area (red).
aligned with the estimated depth. (3) Despite the inpainting, the Gaussians are still under-constraine when heavy occlusions happen, and artifacts may occur. (4) Our runtime scales linearly with the number of objects and can be slow for complex videos. Addressing these limitations by pursuing more data-da triven ways for video 4D generation direct avenue of our future work.
Conclusion
We presented DreamScene4D, the first video-to-4D scene generation work to generate dynamic 3D scenes across occlusions, large object motions, and unseen viewpoints with both temporal and spatial consistency from multi-object monocular videos. DreamScene4D relies on decomposing the video scene into the background and individual object trajectories, and factorizes object motion to facilitate its estimation through pixel and motion rendering, even under large object displacements. We tested DreamScene4D on popular video datasets like DAVIS, Kubric, and challenging self-captured videos. DreamScene4D infers not only accurate 3D point motion in the visible reference view but also provides robust motion tracks in synthesized novel views.
Acknowledgments and Disclosure of Funding
This work is supported by TRI, an NSF CAREER award, and AFOSR FA9550-23-1-0257 award.
10

---

References
[1] Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. To4d: Trajectorycoondiioneedd text-to-4d generation. arXiv preprint arXiv:2403.17920, 2024.
[2]
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generaaton using hybrid score distillation sampling. arXiv preprint arXiv:2311. 1.79844, 2023.
[3] Wenjing Bian. Zirui Wang, Kejie Li, Ma-Wang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising
neural radiance field with no pose prior. In CVPR, 2023.
[4] Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3. 1-a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307. 140.00007.4440, 2023.
[5] Marcel Büsching, Josef Bengtson, David Nilsson, and Marten Björkman. Flowibr: Leveraging pre-training for efficient neural image-based rendering of dynamic scenes. arXiv preprint 0rXiv:2309.05411 2023.
[6] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR, 2023.
Duygu Ccylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Vidco editing using image diffusion. In ICCV, 2023.
[8] Ya-Lian LiLning Chang, Zhe Yu Liu, Kuan- Ying Lee, and Winston Hsu. Learnable gated temporal shift module for deep video inpainting. BMVC, 2019.
[9] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. In ECCV, 2022.
[10] Wen-Hsuan Chu, Adam W Harley, Pavel Tokmakov, Achal Dave, Leonidas Guibas, and Katerina Fragki- adaki. Zero-shot open-vocabulary tracking with large pre-trained models. ICRA, 2024.
[11] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
[12] Carl Daersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in a video. In Neur!PS, 2022.
[13] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
[14] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and Ulrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint arXiv:2403. 123365, 2024.
[15] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In CVPR, 2022.
[16] Eric Heiden, Ziang Liu, Vibhav Vineet, Erwin Coumans, and Gaurav S Sukhatme. Inferring articulated rigid body dynamics from rgbd video. In IROS, 2022.
[17] Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 {'deg] dynamic object generation from monocular video. In ICLR, 2024.
[18] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07633. 2023.
[19] Lei Ke, Mingqiao Ye, Martin Danclljan, Yifan Liu, Yu- Yu-Wing Tai, Chi-Keung Tang, and Fisher Yu. Segment anything in high quality. NeurlPS, 2023.
[20] Bernhard Kerbl. Georgios Kopanas. Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, -1 2023.
[21] Levon Khachatryan, Andranik Movsisyan. Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In ICCV, 2023.
[22] Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchronized joint diffusions. NeurlPS, 2023.
[23] Tianye Li, Mira Slavcheva. Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim. Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
[24] Huan Ling, Seung Wook Kim. Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint 1377:3331..13663 2023.
[25] Lingjee Liu, Jiatan Gu, Kyaw 7aw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advonces in Neural Information Processing Systems, 13:15651-15663, 2020.
[26] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1- 1-to-3: 3: shot one image to 3d object. In ICCV, 2023.
11

---

[27]
Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf. Changil Kim, Yung- Yu Chuang. Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In CVPR. 2023.
[28]
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv: 19060051 2019.
[29] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
[30] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
[31] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[32] Zijie Pan, Zeyu Yang. Xiatian Zhu, and Li Zhang. Fast dynamic 3d object generation from single-view video, arXiv preprint arXiv 2401.08742, 2024.
[33] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Bauain- Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[34] Keunhong Park. Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla. and Steven M. Seitz. Hypernerf: A nigher-dimensionee topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dee 2021.
representation for
[35] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-ing image translation. In SIGGRAPH. 2023.
[36] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. Sfv: Reinforcement learning of physical skills from videos. ACM Transactions On Graphics (TOG), 37(6):1-14, 2018.
[37] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Hornung and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint N0Giv/770.00065 2017.
[38] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ICLR, 2023.
[39] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance
fields for dynamic scenes. In CVPR, 2021.
[40] Chenyang Qi. Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang. Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. ICCV, 2023.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[42] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 44(3), 2022.
[43] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Drcamgaussian4d: Generative 4d gaussian splatting. arXiv preprint 1xXi::23/2.17145 2023.
[44] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chea, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint 2rX1 14159, 2024.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffiusion models. In CVPR, 2022.
[46] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Turii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint arXiv::301. 12200 2023.
[47] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR, 2021.
[48] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov. Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. resolution-robus large mask inpainting with fourier convolutions. In WACV, 2022.
[49] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. ICLR, 2024.
[50] Shuzhe Wang. Vincent Leroy. Yohann Cahon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric
3d vision made easy. In CVPR, 2024.
[51] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024. [52] Jay Zhangjie Wu, Yixiao Ge. Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi. Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023.
[53] Dejia Xu, Hanwen Liang. Neel P Bhatt, Hezhen Hu, Hanxue Liang. Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: LIm-guided compositional 4d scene generation. arXiv preprint arXiv:2403.16999 2024.
12

---

[54] Haofei Xu, Jing Zhang, Jianfei Cai. Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In CVPR, 2022.
[55] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-to-sequence video object segmentation. In ECCV,
2018.
[56] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
[57] Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. TrXiv:2406009414 2024.
[58] Qitong Yang. Mingtao Feng. Zijie Wu, Shijie Sun, Weisheng Dong, Yaonan Wang, and Ajmal Mian.
Beyond skeletons: Integrative latent mapping for coherent 4d sequence generation. arXiv preprint
wrXiv2243.132258 2024.
[59] Ziyi Yang, Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint 1xXi:22309.31011. 2023.
[60] Vickie Ye. Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos the wild. CVPR, 2023.
[61] Yuyang Yin, Dejia Xu, Zhangyang Wang. Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.1722 2023.
[62]
Yifei Zeng. Yanqin Jiang. Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao
Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. arXiv preprint arXiv:2403. 14939, 2024.
[63] Richard Zhang, Phillip Isola, Alexei Efros, Shechuman,
Oliver Wang. The unreasonable
effectiveness of deep features as perceptual metric. In CVPR, 2018.
[64] Yuyang Zhao, Zhiwen Yan, Enze Xie. Lanqing Hong. Zhenguo Li, and Gim Hee Lee. 12imate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311. 14603, 2023.
[65] Yang Zheng. Adam W Harley, Bokui Shen. Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: A large-scale synthetic dataset for long-tern point tracking. In ICCV, 2023.
[66] Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text-and guided 4d scene generation. arXiv preprint arXi::3311.16544 2023.
13

---

A Appendix Supplemental Material
In the supplementary materials, we provide the details for video amodal completion, more imple- mentation details of our DreamScene4D, and some qualitative and quantitative evaluations of the amodal completion. For more qualitative video-to-4D generation evaluations, we suggest looking at the videos in the website
A.1 Video Amodal Completion
We build off SD-Inpaint 45 and adapt it for video amodal completion by making two modifications
to the inference process without further fine-tuning.
Self-Attention A common technique for extending Stable Diffusion-based models for video generation editing inflates the spatial self-attention layers additionally attend across frames without changing the pre-trained weights [52 21 7 [4). Similar to [7]. we inject tokens from
adjacent frames during self-attention enhance inpainting consistency. Specifically, the self-attention operation can be denoted as:
Q = Wo K = WK [.1-1, 24, 2+11,, V = Wv 24:
(4)
where [-] represents concatenation, 2, is the latent representation of frame t, and Wo. WK. and Wy denote the (frozen) projection matrices that project inputs queries, keys, and values.
Latent Consistency Guidance While inflating the self-attention layers allows the diffusion model attend to and denoise multiple frames simultaneously, it does not ensure that the inpainted video frames are temporally consistent. To solve this issue, we take inspiration from previous works that perform test-time optimization while denoising for structured image editing [35] and panorama generation [22) and explicitly enforce the latents during denoising to be consistent.
Concretely, follow two-step process for each denoising step for noisy latent 2 denoising timestep 7 to latent z -r- For each noisy latent 27 at frame f, we compute the fully denoised latent 2f and its corresponding image I directly in one step. To encourage the latents of multiple frames to become semantically similar, we freeze the network and only update 2T:
TT VfCe
where 7) determines the size of the gradient step and L. is a similarity loss, i.e., CLIP feature loss or SSIM between pairs of It. After this latent optimization step, we take ST and predict the added noise ET using the diffusion model to compute -T as:
OGe
a1-1
+ /1 at a
(6)
where de is the noise scaling factor defined in DDIM
47
A.2 More Implementation Details
Deformation Network. The deformation network uses Hexplane [6] backbone representation with 2-layer MLP head on top predict the required outputs. In our evaluations, the resolution of the Hexplanes is [64, 64, 64, 25] for (x, y, z, () to ensure fair comparisons with the baselines. For longer videos (more than 32 frames), we set the resolution to [64, 64, 64, 0.87) for (ar, y, z, L), where 7 is the number of frames. We found that the network is generally quite robust to the temporal resolution of the Hexplane grid.
we set different learning rates for different Gaussian parameters. We use the same set of hyperparameters as Dream- Gaussian and use a learning rate that decays from le to -5 for the position, a static learning rate of 0.01 for the spherical harmonics, 0.05 for the opacity, and 53 for the scale and rotation. The learning rate of the Hexplane grid is set to 6.4e while the learning rate the MLP prediction heads is set to -3 During joint fine-tuning of the deformation network and the object-centric to world frame transformations, we set the learning rate the original value. We use the AdamW optimizer for all our optimization processes.
Learning Rate. Following DreamGaussian 49 and DreamGaussian4D
14

---

Thenk 5 vidaao whhic ahow oniggia Plaae
add setils
Select option
Preter
Profer (B)
Equally Preferred
Oricinal Video
Video (A)
Video
User survey interface. A GUI example of what Amazon Turk worker would see part of the user preference study.
Figure 7:
the densification in the image-to-3D step is applied for Gaussians with accumulated gradient larger than 0.5 and max scaling smaller than 0.05. Gaussians with an opacity value less than 0.01 or max scaling larger than 0.05 are also pruned. This is done every 100 optimization step. Densification and pruning are both disabled during motion optimization.
Densification and Pruning. Following 44 43
Running Time. As mentioned in the main text, we perform 1000 optimization steps for the static 3D Gaussian splatting process, while the deformation optimization takes 100 T optimization steps, where T is the number of frames. The joint fine-tuning process is conducted over 100 steps. While many videos converge faster, we found that videos with more complex objects and motion require more optimization steps. On a 40GB A100 GPU, the static 3D lifting process takes around 5.5 minutes, and the 4D lifting process takes around 17 minutes for video of 16 frames per object. For comparisons with the baselines, please refer to Table
Table 4: Running time comparisons. Since the running time of DreamScene4D scales w.r.t the number of objects in the video, we present the results separately for videos with 1, 2, and objects from DAVIS, denoted by the 3 entries in each field (1 obj/2 objs/3 objs).
Method
CLIP
LPIPS
Time (GPU hrs)
FPS (Hz)
Memory (GB)
Consistent4D
82.14
0.141
0.8
4.9
26.8
DreanGaussian4D
77.81
0.181
0,44hr
76.7
22.8
DreamScene4D 85.09
0.152
0.27hr/0.5hhr/0.8hhr
76.1/72.4/68.7
24.7
Evaluation Settings. In video-to-4D evaluations, render from the following combination of (elevation, azimuth) angles: (0, 45), (0, -45), (45, 0), (-45, 0). These novel view renders are then compared with the reference view at each timestep to obtain the CLIP and LPIPS scores. The scores are then averaged across all views and timesteps for the final score.
User Preference Study. For the user study, take the 30 DAVIS videos and produce smooth orbital render video by varying the azimuth angle while rendering the deforming object(s). We use Amazon Turk to outsource evaluations the 30 DAVIS videos for each baseline, including Dream- 43 + Video Scene Decomposition (VSD), Consistent4D 17
Gaussian4D 43 DreamGausian4DD and our DreamScene4D.. Each set of videos is reviewed by 30 workers with HIT rate of over 95% for a total of 2700 answers collected. The whole user preference study takes about 97s per question and 72.8h working hours in total. We manually filtered out workers who submitted the same answer for all the videos and assigned new ones during the collection process until the desired number of answers had been collected.
The full instruction given is as follows:
Please read the instructions and check the videos carefully.
There are videos that show an orbit view of the original video. Please choose the orbiting video that looks more realistic and better represents the original video
15

---

Reference View
Novel Vie
Novel View
Figure 8: Gaussian Motion Visualizations. We visualize the Gaussian trajectories in the reference view corresponding to the video as well as in multiple novel views. The rendered Gaussians are sampled independently for each view. DreamScene4D can produce accurate motion in different camera poses w/o explicit point trajectory supervision.
you. The options (A) and (B) correspond to the two given orbit videos. If you think that both are of the same quality, please select Equally Preferred.
judge the quality of the videos, consider the following points:
1. Do the objects in the orbit video correspond to the original video?
2. Does the video look geometrically correct (e.g. not overly flat) when the camera
is orbiting?
3. Are there any visual artifacts (e.g. floaters, weird textures) during the orbit?
Please ignore the background in the original video.
GUI sample of survey question also provided in Figure for reference.
A.3 Additional Results
4D Motion Visualizations in Novel Views Since DreamScene4D represents the scene using 4D Gaus- sians, it is able to obtain motion trajectories in arbitrary camera views, in Figure DreamScene4D can both generate a 4D scene with consistent appearance across views and produce temporally coherent motion trajectories.
Video Amodal Completion To ablate our extensions to SD-Inpaint for video amodal completion, we randomly select 120 videos from YoutubeVOS [55] and generate random occlusion masks in the video 48 8). We compare against Repaint 22 and Inpaint 45 for video amodal completion.
Both baseline methods are based Stable Diffusion 45. Repaint alters the reverse diffusion iterations by sampling the unmasked regions of the image. SD-Inpaint, on the other hand, finetunes Stable Diffusion for free-form inpainting. We also ablate the performance of our proposed amodal complction approach without the inflated spatiotemporal self-attention (denoted as STSA) and consistency guidance. We summarize the results in Table and show some visual comparisons in Figure 9 Our modification achieves more consistent and accurate video completion than image inpainting approaches by leveraging temporal information during the denoising process. Note that these techniques complement other video completion approaches since DreamScene4D mainly focuses video-to-4D scene generation.
We show an example the rendered Gaussians before and after performing the joint optimization for the deformation network and the object-centric world frame transformations Fi We can see that small amount of joint fine-tuning steps helps alleviate the parallax effect and better aligns the rendered Gaussians to the input video frames.
Mitigating Parallax Effects via Joint Optimization
16

---

Oocluded Frames
SD-inpaint
Ours wio STSA
wio Guidance
Ours
Ours
Figure 9: Video Amodal Completion Comparisons. Spatiotemporal self-attention and Consistency Guidance both help to preserve the identity consistency of the inpainted objects.
Table 5: Video Amodal Completion Evaluations. We report the PSNR, LPIPS, and Temporal Consistency (TC) measured using CLIP similarity in randomly masked YoutubeVOS 55
videos.
PSNR t (masked)
Method
PSNR t
LPIPS
TC t
Repaint 29
20.76
14.04
0.23
91.18
SID-Inpaint 45
21.07
14.35
0.23
91.72
DreamScene4D (Ours)
22.27
16.09
0.22
93,40
w/o STSA
21.56
15.31
0.23
92.58
w/o Guidance
21.71
15.20
0.23
92.91
A.4 Failure Cases
We additionally show some failure cases corresponding to the limitations documented in the main text in Figure Based on our observations, the inpainting is very unstable during heavy occlusions. We believe that instead of solely relying rendering losses for the occluded regions, incorporating some form of semantic guidance loss (e.g. CLIP feature loss) might be a promising direction.
A.5 Broader Impact
Our approach is deeply connected to VR/AR applications and can potentially provide 3D meshes and dense 3D trajectories for robot manipulation. While our method does not generate or modify the original video, it is still possible for users to use generative models with malicious intent, and then apply our approach for video-to-4D lifting. The potential negative impact can be avoided by applying preventative measures in generative models and rejecting the video input if violations are found.
A.6
DAVIS Split
We list the DAVIS video names that were used perform evaluations:
bear, blackswan, bmx bmx-bumps, box.booo-----o foxing-fisheye fisheye, car shados cows, crossing,
17

---

Original Video
Orbital View
Original Video
Orbital View
Figure 10: Failure Cases. We show representative failure cases. The first case (top 2 rows) is due inpainting failures (circled in blue), where the inpainted frames flickering objects when rendered. The second case (bottom 2 rows) arises from poor depth predictions, which leads to composition errors. The two humans are placed too close to the truck, making the scale proportions of the objects seem unnatural (i.c. the truck is too small).
of high quality, leading
connt-Finetuning Finnt-Finetunin
Figure 11: Mitigating the parallax effect. A small amount of joint fine-tuning steps can help mitigate the parallax effect and align the rendered Gaussians the input video frames.
hikeeishhhhhhikkekee hockey, kid-foob lab-coat, Jonaa,, parkour, ,igs,, rallye, rhino, oollertiie...boooggiii...oooeoeo wwwotblccccccceeeeeeeeeeeoeee ana.rerrrggrraraooodddd.rrddr mnoooaard,strolllee train
18

---

For bmx- bumps, longboard, srooter-black, and scooter-grat gray, we merge the mask of the human and the other objects into one as they move together for the entire video (e.g. person riding bike or scooter).
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction clearly document the scope and contributions of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the limitations section in the main text. We also show some failure cases corresponding to these limitations the supplementary materials.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer:
[NA]
Justification:
We do not claim or present theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss the details of the method and evaluation in the experiments section in the main text and the supplementary materials. The code will also be made public in the future.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: We do not include our code in the submission. However, it will be made public upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
ollicy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccppblle wwwppppguuuiiieeeee /otic///uuiee//oeeuuuiiiiiii SooodeSubiissi. ssiiiioiiccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer the experiments section and the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
Does not apply to our evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to the implementation details subsection in the main text and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: sectps://neurips.t cc http:pp.....pp.ccccccccccccaaaa.
Answer: [Yes]
Justification: We have carefully reviewed the Code of Ethics and confirm that the research conforms to
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer the supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Does not apply, since our paper does not generate new things, but instead converts existing videos to 4D representations.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our code is written by ourselves, while the data comes from existing datasets or simulators, which are cited in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We include the details of our human evaluations in the supplementary materials.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Does not apply to the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "NUvva6vPqY", "reviewer": "Reviewer_EEWc", "summary": "1. The paper claims that their methodology represents the first approach capable of generating verisimilar four-dimensional scene representations derived from complex multi-object video sequences exhibiting substantial motion. This approach purportedly enables 360-degree novel view synthesis and precise motion estimation in both observed and unobserved perspectives.\n\n2. To validate their approach, the researchers conduct evaluations of DreamScene4D using challenging datasets, including DAVIS, Kubric, and proprietary video footage featuring rapidly moving objects. Their findings indicate enhanced performance compared to extant methodologies, specifically in terms of novel view synthesis quality and motion estimation accuracy.", "questions": "Please see the weaknesses.", "limitations": "1. Limited generalization capability of the SDS (Scene Decomposition System) when applied to video footage captured from acute camera angles.\n\n2. Potential for suboptimal scene composition in cases where the rendered depth of reconstructed 3D objects fails to align precisely with the estimated depth map.\n\n3. Presence of visual artifacts stemming from under-constrained Gaussian representations during instances of significant occlusion, despite the implementation of inpainting techniques.\n\n4. Linear scaling of computational runtime in relation to the quantity of objects, potentially resulting in reduced efficiency for videos of high complexity.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The \"decompose-recompose\" strategy represents an innovative approach to addressing the complexities of multi-object scenes in video-to-4D generation. This method circumvents the limitations of previous approaches, which encountered difficulties with multiple objects, by employing a three-step process: decomposing the scene into individual objects and background, independently elevating these elements to 3D representations, and subsequently recomposing them.\n\n2. Quantitative analyses reveal substantial enhancements over contemporary state-of-the-art baselines in two key areas: novel view synthesis quality, as assessed by CLIP and LPIPS metrics, and motion accuracy, quantified by End Point Error. The conducted ablation studies effectively elucidate the contributions of various components within the method, notably the video scene decomposition and motion factorization, thereby providing insight into their respective impacts on overall performance.", "weaknesses": "1. Insufficient analysis of computational complexity: While the authors note that the algorithm's runtime exhibits linear scaling with respect to the number of objects, the manuscript lacks a comprehensive examination of its computational demands. Moreover, there is an absence of comparative analysis vis-à-vis alternative methodologies in terms of computational efficiency. Given the intricate nature of the proposed approach, a more rigorous assessment of its computational costs would significantly enhance the paper's contribution to the field.\n\n2. Reliance on external depth estimation: The proposed methodology incorporates an extrinsic depth estimation algorithm for scene composition. However, the manuscript provides limited discourse on the potential ramifications of depth estimation inaccuracies on the final output. A thorough investigation into the method's robustness to depth estimation errors would substantially augment the study's validity and applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing DreamScene4D as an innovative approach that addresses the difficulties of multi-object scenarios and achieves significant improvements over current state-of-the-art baselines in view synthesis and motion accuracy. Here we respond to your insightful comments and questions.\n\n> **Q1: Insufficient analysis of computational complexity.**\n\nWe mention that \"the algorithm's runtime exhibits linear scaling with respect to the number of objects\" in the limitation section (Sec 4.3, L283-L284), and also discuss the run time of DreamScene4D in lines L490-L495 of the Appendix (A.2, L490-L495), where \"On a 40GB A100 GPU, the static 3D lifting process takes around 5.5 minutes, and the 4D lifting process takes around 17 minutes for a video of 16 frames per object.\"\n\nPer your request, to provide a more comprehensive comparison, we include the optimization time and rendering efficiency, measured in hours and FPS respectively, for DreamScene4D and the baselines, in addition to the original metrics presented in Table 1 of the paper. Since our method has linear runtime complexity w.r.t. to the number of objects in the scene, we present the results separately for videos with 1, 2, and 3 objects from DAVIS, denoted by the 3 entries in each field (1 obj/2 objs/3 objs). We will include this analysis in the revised version of the paper.\n\n| Method          | CLIP&uarr;  | LPIPS&darr;    | Optimization Time (GPU hrs) | FPS (Hz) | GPU Memory (GB) |\n|-----------------|:------:|:---------:|:---------------------------:|:--------:|:---------------:|\n| Consistent4D    | 82.14  |  0.141    |            0.81hr           |   4.9    |        26.8      |\n| DreamGaussian4D | 77.81  |  0.181    |            0.44hr           |   76.7   |        22.8      |\n| DreamScene4D (**ours**)    | 85.09  |  0.152    |     0.27hr/0.53hr/0.81hr    |   76.1/72.4/68.7   |        24.7      |\n\nAs motion factorization provides a strong prior on the general deformation of the objects, DreamScene4D can converge faster with fewer steps than the more conservative experiments reported in the paper without a loss in quality. However, as the number of object(s) increases, we can observe the trade-off where the individual optimization of decomposed objects results in higher training time but  yields much better view synthesis quality. We leave it to future work to further accelerate the inference time of our model without loss in performance.\n\n> **Q2. Reliance on external depth estimation.**\n\nDreamScene4D uses the estimated monocular depth to infer the relative depth relationships/scales of each independently optimized 4D Gaussians and recompose them into one unified coordinate frame. To provide more insights into the robustness of depth estimation errors, we conducted additional experiments:\n\n1) We replaced the Depth-Anything model with MiDAS, a weaker depth prediction model, as well as the newly released Depth-Anything v2.\n2) We added random noise to perturb the outputs of Depth-Anything v1 in various magnitudes.\n\nSince depth estimation is only used for scene composition, we conducted experiments on **multi-object DAVIS videos only** to emphasize the differences and summarize the results below. Note that this is different from the original evaluations in the paper which also contains some single-object videos from DAVIS. We will also include this analysis in the revised version of the paper.\n\n| Method                         | CLIP&uarr; | LPIPS&darr; | \n|--------------------------------|:----:|:-----:|\n| Depth-Anything v1 (**original**)   | 83.71| 0.169 | \n| Depth-Anything v1 + Noise (10%)| 83.67| 0.171 | \n| Depth-Anything v1 + Noise (25%)| 83.48| 0.174 | \n| MiDAS v3.1                     | 83.34| 0.176 | \n| Depth-Anything v2              | 83.76| 0.169 | \n\nWhile different depth predictions result in objects being placed in slightly different scene locations, we note that existing SOTA depth prediction models (such as Depth Anything series) meet the requirements in most cases, and the rendered quality of the 4D scene will not deteriorate much as long as the _relative depth ordering_ of the objects is correct, which even holds when we add a small amount of noise to the predicted depth (second row) or use a weaker model like MiDAS (fourth row).\n\nFor videos/scenarios requiring precise depth placement for 4D lifting, the captured depth from RGBD cameras (widely equipped on iPhones) can also be used. Also, we expect the SOTA monocular depth models will further improve the robustness by scaling data/model.\n\n> **Q3. Limitation disclosure.**\n\nWe summarized the limitations of our paper in Section 4.3 and extensively discussed it in Appendix A.4 section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QiELjZ44Qn", "reviewer": "Reviewer_onS9", "summary": "The paper introduces a novel framework for reconstructing multi-object scenes from monocular videos by factorizing motion into three components: object-centric deformation,  object-to-world-frame transformation, and camera motion. This method enhances stability in motion optimization and effectively captures large-scale motions. Initially, the method segments the video into foreground objects and background. These elements are then lifted into a static 3D Gaussian representation using the DreamGaussian method. Subsequently, the motion of each object is independently optimized and finally, the optimized 4D Gaussians are recombined into a unified scene using estimated monocular depth, aligning the model effectively within the global scene context.", "questions": "See weakness section.", "limitations": "The author has made a commendable effort to acknowledge the technical limitations of their method by showcasing failure cases in A.4.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The \"decompose-recompose\" approach effectively addresses the challenge of generating dynamic multi-object 4D scenes from monocular videos.\n- The experiments shows solid evidence that factorization of object motion improves motion optimization.\n- The writing well written. The main ideas and concepts are mostly well explained and articulated throughout.", "weaknesses": "- Table 1 the ablation study on L_{flow} does not justify the need for this component, the metrics for the DreamScene4D and w/o L_{flow }are close, could you provide some examples to justify the need of this loss? \n\n-While the object-to-world frame transformation process described includes computing the translation and scaling factors that warp the Gaussians from the object-centric frame to the world frame, there are still noticeable artifacts and misalignment of objects in the scene as shown in the supplemental results. Can the authors provide more details on the potential causes of these artifacts and misalignments? Additionally, what steps or adjustments could be implemented to address and reduce these issues, ensuring better alignment and artifact-free renderings?\n\n- The method seems to rely on monocular depth estimation. How does the performance degrade with less accurate depth data, and are there any strategies to mitigate this dependency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging that our paper is novel, effective with solid evidence and well-written. Here we respond to your insightful comments and questions.\n\n> **Q1. In Table 1, the ablation study on $L_{flow}$ does not justify the need for this component.**\n\nWe measure the 4D generation performance of DreamScene4D both in **view synthesis** (Tab.1 of the paper) and Gaussian **motion accuracy** (Tab.2 of the paper). While the flow loss $L_{flow}$ does not directly contribute to the quality of the rendered images, it drastically improves the accuracy of the 3D Gaussians movement in Tab.2 (paper), where the endpoint error (EPE) of the inferred 3D trajectories is reduced obviously from 10.91 to 8.56 on DAVIS, and 18.54 to 14.3 on Kubric.\n\n> **Q2. Can the authors provide more details on the potential causes of these artifacts and misalignments?**\n\nWe aim to handle real-world complex videos with multi-object movement. As we discussed in the Limitations (Sec 4.3 of paper) and Failure Cases (Sec A.4 of Appendix) section, although DreamScene4D produces decent 4D generation results for most videos, there are still three potential causes of artifacts of misalignment errors:\n\n1. Occlusions and image completion failures. When the object is heavily occluded and small, it is  difficult to accurately inpaint the missing parts of the object. Thus, the input may only contain a small part of the object, and the Gaussians will not attempt to explain the missing parts, as showcased in the top half of Figure 9 (Sec A.4 of Appendix). This can potentially be fixed by reducing the reliance on photometric RGBA rendering losses and incorporating semantic losses such as using CLIP to measure the similarity of the rendered object in frame t with other frames.\n2. Large monocular depth prediction errors. This happens when the errors of the depth estimation cause the depth ordering of the objects to be wrong, resulting in the scene composition process placing an object that should be in the back in front of something else, as showcased in the bottom half of Figure 9. Please refer to our response to Q3 for a more detailed analysis.\n3. Parallax differences. When warping the object to a different location in the scene, different parts of the object are rendered by the camera (e.g. the side of the object becomes visible when we move an object from the center to the left). We found empirically that doing a couple of joint-finetuning steps can mitigate this issue, as we discussed in the paper and showcased in Figure 10 (Appendix).\n\n> **Q3. How does the performance degrade with less accurate depth data, and are there any strategies to mitigate this dependency?**\n\nDreamScene4D uses the estimated monocular depth to infer the relative depth relationships/scales of each independently optimized 4D Gaussians and recompose them into one unified coordinate frame. To provide more insights into the robustness of depth estimation errors, we conducted the additional experiments:\n\n1) We replaced the Depth-Anything model with MiDAS, a weaker depth prediction model, as well as the newly released Depth-Anything v2.\n2) We added random noise to perturb the outputs of Depth-Anything v1 in various magnitudes.\n\nSince the depth estimation is only used for scene composition, we conducted experiments on **multi-object DAVIS videos only** to emphasize the differences and summarize the results below. Note that this is different from the original evaluations in the paper which also contains some single-object videos from DAVIS. We will also include this analysis in the revised version of the paper.\n\n| Method                         | CLIP&uarr; | LPIPS&darr; | \n|--------------------------------|:----:|:-----:|\n| Depth-Anything v1 (**original**)   | 83.71| 0.169 | \n| Depth-Anything v1 + Noise (10%)| 83.67| 0.171 | \n| Depth-Anything v1 + Noise (25%)| 83.48| 0.174 | \n| MiDAS v3.1                     | 83.34| 0.176 | \n| Depth-Anything v2              | 83.76| 0.169 | \n\nWhile different depth predictions will result in objects being placed in slightly different scene locations, we note that existing SOTA depth prediction models (such as Depth Anything series) meet the requirements in most cases, and the rendered quality of the 4D scene will not deteriorate much as long as the _relative depth ordering_ of the objects is correct, which even holds when we add a small amount of noise to the predicted depth (second row) or use a weaker model like MiDAS (fourth row).\n\nFor videos/scenarios requiring precise depth placement for 4D lifting, the captured depth from RGBD cameras (widely equipped on iPhones) can also be used or we can leverage the depth post-optimization method like RCVD [a] to further improve the initial depth prediction. Besides, we expect the SOTA monocular depth models will further improve its robustness by scaling data/model.\n\n[a] Robust Consistent Video Depth Estimation. CVPR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zhkJjwYN2A", "reviewer": "Reviewer_4pTB", "summary": "DreamScene4D is a novel approach to generating 3D dynamic scenes of multiple objects from monocular videos using 360° novel view synthesis. The method employs a \"decompose-recompose\" strategy, segmenting the video scene into background and object tracks and further decomposing object motion into object-centric deformation, object-to-world-frame transformation, and camera motion. This decomposition enables effective recovery of object 3D completions and deformations, guided by bounding box tracks for large object movements. DreamScene4D demonstrates extensive results on challenging datasets like DAVIS, Kubric, and self-captured videos, providing quantitative comparisons and user preference studies. The approach also achieves accurate 2D persistent point tracking by projecting inferred 3D trajectories into 2D. The release of the code aims to stimulate further research in fine-grained 4D understanding from videos.", "questions": "While the authors effectively demonstrate their method, the paper lacks some intermediate visualizations, such as the 3D Gaussian visualizations for Figure 2. Additionally, there are potential bad cases involving multiple objects that need further exploration and clarification. These aspects are crucial for thoroughly understanding and validating the proposed approach.", "limitations": "The authors do not discuss the limitations in the manuscript.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper introduces a novel \"decompose-recompose\" strategy, effectively handling complex multi-object scenes and fast object motion by robustly recovering object 3D completions and deformations.\n\n2.\tThe paper demonstrates comprehensive experimental validation on challenging datasets such as DAVIS, Kubric, and self-captured videos, showcasing its applicability to real-world scenarios and superiority over existing methods through strong quantitative comparisons and user preference studies.\n\n3.\tThe paper is well-written and well-structured, making it easy to follow and understand the key ideas and methodologies.", "weaknesses": "1.\tFigure 2 appears to show artifacts in the decomposed background. The paper should clarify how the video scene decomposition separates the background and objects, as this is crucial for the model's performance. \n\n2.\tThe paper does not explain how temporal consistency is maintained between object motion and camera motion. Ensuring synchronized motion is essential for the accuracy of the 4D scene generation.\n\n3.\tThe paper lacks comparisons with relevant baselines such as Animate124[1] and 4Diffusion[2]. It should also address how the model handles scenarios with many objects or objects appearing at different times.\n\n[1] Zhao Y, Yan Z, Xie E, et al. Animate124: Animating one image to 4d dynamic scene[J]. arXiv preprint arXiv:2311.14603, 2023.\n\n[2] Zhang H, Chen X, Wang Y, et al. 4Diffusion: Multi-view Video Diffusion Model for 4D Generation[J]. arXiv preprint arXiv:2405.20674, 2024.\n\n4.\tThe model relies on an off-the-shelf depth estimator to compute the depth of each object and the background for 4D scene composition. However, the effectiveness of this guidance in dense object scenarios is questionable and needs further evaluation.\n\n5.\tThe paper should include visualizations of the 3D Gaussians from Figure 2 to provide a clearer understanding of the spatial representation and validate the model's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging DreamScene4D being a novel approach with comprehensive experimental validation and superiority over existing methods, and for the writing being well-structured. Here we respond to your insightful comments and questions.\n\n> **Q1. How does video scene decomposition separates background and objects?**\n\nWe take off-the-shelf segmenters (GroundingDINO + SAM) and mask trackers (XMem, DEVA [a]) to segment and track the objects. The background image simply contains the remaining parts of the video frame. We then use a modified version of SD-Inpaint (Sec A.1 of Appendix) to amodally complete (inpaint) the background image. We will update Sec 3.2.1 of the paper to clarify this.\n\nWe note that the small artifacts in Figure 2 result from the background inpainting process. We expect that there will be fewer artifacts using better image inpainting models (e.g., SDXL-Inpainting [b]) for background inpainting. \n\n[a] Tracking Anything with Decoupled Video Segmentation. ICCV, 2023.\n\n[b] stable-diffusion-xl-1.0-inpainting-0.1\n\n\n> **Q2. The paper does not explain how temporal consistency is maintained between object motion and camera motion.**\n\nThe inferred camera and object motion trajectories are temporally aligned by the reconstruction process (Sec 3.2.3 of paper), where the composition of the camera and object motion is grounded on the RGB/flow rendering loss between each rendered frame and the ground truth frame sequentially. This ensures that both camera and object motions faithfully reveal the temporal movement / deformation observed in the original video. \n\n> **Q3.1. Comparisons with Animate124[a] and 4Diffusion[b].**\n\nThanks for pointing out these two concurrent related works, and will discuss and cite them in the Related Works (paper). However, please note that Animate124 and 4Diffusion do not tackle in-the-wild multi-object videos with large motion. **Also, 4Diffusion[b] was released after the NeurIPS submission deadline, while Animate124 [a] is reported with worse performance than DreamGaussian4D and is proposed as an image-to-4D method instead of video-to-4D**, as described in Table 2 and Figure 7 of [c].  We show our model significantly outperforms DreamGaussian4D in Tables 1 and 2. We also cited Animate124 in related work. Per the reviewer's suggestion, we additionally ran experiments on DAVIS using Animate124 by adapting it to condition on given video frames, which we show here **(the results for other baselines are from Tab 1 of Paper)**:\n\n| Method|CLIP&uarr;|LPIPS&darr; | \n|-|:-:|:-:|\n| Consistent4D|82.14|0.141| \n| Animate124| 74.17| 0.206 | \n| DreamGaussian4D | 77.81 | 0.181| \n| DreamScene4D (**ours**)| 85.09| 0.152| \n\n[a] Animate124: Animating one image to 4d dynamic scene. arXiv 2023.\n\n[b] 4Diffusion: Multi-view Video Diffusion Model for 4D Generation. arXiv 2024.\n\n[c] Dreamgaussian4d: Generative 4d gaussian splatting. arXiv 2023.\n\n> **Q3.2. How does the model handle many objects appearing at different times?**\n\nDreamScene4D employs a multi-object segmentation tracker (Grounded SAM + XMem or DEVA) to infer 2D mask tracks for each object. It can track objects that appear and disappear arbitrarily throughout the video. We then extract the time intervals during which objects appear or disappear from the inferred video masks. Consequently, we can optimize for each object independently during its visible intervals and then recompose the 4D Gaussians at the appropriate timesteps. This approach highlights DreamScene4D's advantage, as the existing baselines cannot handle objects that appear in the middle of a video by deforming the 3D representations from the initial frame.\n\n> **Q4. Effectiveness of depth guidance in dense object scenarios needs further evaluation.**\n\nDreamScene4D uses the estimated monocular depth to recompose the independently optimized 4D Gaussians into one unified coordinate frame. To provide insights on the effectiveness of depth guidance in dense object scenarios, we report the CLIP and LPIPS on multi-object DAVIS videos as below:\n\n| Num objects in video | CLIP&uarr; | LPIPS&darr; | \n|-|:-:|:-:|\n| 2 objects| 83.87| 0.149 | \n| 3 objects| 83.23| 0.182 | \n| 5 objects| 84.15| 0.167 | \n| All multi-obj videos| 83.71| 0.169 | \n\nWhile we agree that depth prediction is harder in dense object scenes, existing SOTA depth prediction models meet the requirements in most cases as we only require the _relative depth ordering_ of the objects to be correct. Thus, having more objects does not necessarily mean the results deteriorate.\n\nWe also conducted the additional experiments on **multi-object DAVIS videos only** to provide insights into the robustness of depth estimation error:\n\n1) We swap out the Depth-Anything model with MiDAS, a weaker depth estimation model, as well as the newly released Depth-Anything v2.\n2) We added random noise to perturb the outputs of Depth-Anything v1 in various magnitudes.\n\n|Method|CLIP&uarr;|LPIPS&darr;| \n|-|:-:|:-:|\n|Depth-Anything v1 (**original**)|83.71|0.169| \n|Depth-Anything v1 + Noise (10%)|83.67|0.171| \n|Depth-Anything v1 + Noise (25%)|83.48|0.174| \n|MiDAS v3.1|83.34|0.176| \n|Depth-Anything v2| 83.76|0.169| \n\nWhile different depth predictions will result in objects being placed in slightly different scene locations, the _relative depth ordering_ of the objects largely stays the same, so the impacts on the performance is not large. We will also include these analysis in the revised version of the paper.\n\n> **Q5. Intermediate visualizations for Gaussians.**\n\nThanks for the suggestion, we have added this in Figure 1 of our one-page rebuttal PDF, and will include it in the revised paper.\n\n> **Q6. The authors do not discuss the limitations in the manuscript.**\n\nThis is incorrect. We'd like to kindly remind the reviewer that we discussed the limitations in Section 4.3 (paper, L279-L285), as well as provided detailed failure case analysis in Sec A.4 (Appendix). We will extend the current limitation section by merging Sec A.4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Ac5213hFv", "reviewer": "Reviewer_TmCf", "summary": "This work introduces DreamScene4D, a pioneering approach for generating realistic 4D scene representations from complex multi-object videos with significant object motions. It proposes a \"decompose-recompose\" strategy, where a video is first decomposed into individual objects and the background scene. Each component is then completed across occlusions and viewpoints. To manage fast-moving objects, DreamScene4D factorizes 3D motion into three components: camera motion, object-centric deformations, and an object-centric to world frame transformation. The components are recomposed using monocular depth guidance to estimate relative scales and rigid object-to-world transformations, placing all objects within a common coordinate system. DreamScene4D's effectiveness is demonstrated using challenging monocular videos from the DAVIS dataset. It shows significant improvements over existing state-of-the-art video-to-4D generation methods on the DAVIS, Kubric, and self-captured videos with fast-moving objects.", "questions": "1. For 4D scene composition, in addition to the depth, how to estimate the height/width location of the component?\n2. How many GS points are used to represent each component? I think the appearance quality can be further improved by increasing the number of points.\n2. For LPIPS metric evaluated on the DAVIS dataset, could you explain the reasons for better performance of Consistent4D over Dreamscene4D?", "limitations": "1. Due to the segmentation and impainting operations from the source video in the decomposition stage, we can observe some artifacts in the constructed 3D assets. This is a tricky problem associated with the methodology.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, and particularly the figures, clearly illustrate the system. \n2. The idea is novel and straightforward, and the methodologies are effective and reasonable.\n3. The factorization of 3D motions into three components, i.e. camera motion, object-centric deformations, and an object-centric to world frame transformation offers clear advantages over prior arts to model dramatic motions.\n4. The comparisons with previous methods and the ablation studies are comprehensive, and the demonstrated improvements are convincing.", "weaknesses": "1. This method is composed of multiple steps and requires much human involvement.\n2. For the recomposition stage, the estimation of scales and locations of multiple components is sensitive to the performance of the depth estimation model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging that our paper is novel, well-written and effective with comprehensive comparisons. Here we respond to your insightful comments and questions.\n\n> **Q1. This method is composed of multiple steps and requires much human involvement.**\n\nThe statement “our method requires much human involvement” is incorrect. Although composed of multiple steps in the pipeline, our method is fully automatic and runs automatically using a single script without human intervention. We use off-the-shelf object segmenters (Grounded-SAM to detect and segment objects), inpainting model (adapted SD-Inpaint), and MoT trackers (DEVA) to decompose a scene into objects and background, track each one, and then lift them to 4D. \n\n> **Q2. The estimation of scales and locations of multiple components is sensitive to the performance of the depth estimation model.**\n\nDreamScene4D uses the estimated monocular depth to infer the relative depth relationships/scales of each independently optimized 4D Gaussians and recompose them into one unified coordinate frame. To provide more insights into the robustness of depth estimation during the recomposition, we further conducted the additional experiments analysis:\n\n1) We swap out the Depth-Anything model with MiDAS, a weaker depth prediction model, as well as the newly released Depth-Anything v2.\n2) We added random noise to perturb the outputs of Depth-Anything v1 in various magnitudes.\n\nSince the depth estimation is used for scene composition, we conduct experiments on **multi-object DAVIS videos only** and summarize the results below. We will also include this analysis in the revised version of the paper.\n\n| Method| CLIP&uarr; | LPIPS&darr; |\n|-|:-:|:-:|\n| Depth-Anything v1 (**original**) | 83.71| 0.169 |\n| Depth-Anything v1 + Noise (10%)| 83.67| 0.171 |\n| Depth-Anything v1 + Noise (25%)| 83.48| 0.174 |\n| MiDAS v3.1| 83.34| 0.176 |\n| Depth-Anything v2| 83.76| 0.169 |\n\nWhile different depth predictions will result in objects being placed in slightly different scene locations, we note that existing SOTA depth prediction models (such as Depth Anything series) meet the requirements in most cases, and the rendered quality of the 4D scene will not deteriorate much as long as the _relative depth ordering_ of the objects is correct, which even holds when we add a small amount of noise to the predicted depth (second row) or use a weaker model like MiDAS (fourth row).\n\n> **Q3. How to estimate the height/width location of the component for 4D scene composition?**\n\nTo estimate the 3D location of each object component, we first compute its 2D translation in the image-space (i.e. offset from the center of the image) as $(\\Delta x,\\, \\Delta y)$, then use the camera projection matrix $P = K[R|t]$ of the rendering camera to unproject this 2D translation to 3D, where $K$ is the camera intrinsics, $R$ is the rotation aspect of the extrinsics, and $T$ is the translation aspect of the extrinsics.\n\nIn our formulation with a monocular video, we define the coordinate frame of the camera space corresponding to the video and world space to be in the same orientation by setting $R$ to the identity matrix and solve the 3D translation from the origin directly: $(\\Delta X,\\, \\Delta Y) = (\\frac{\\Delta x \\cdot d}{f_x},\\, \\frac{\\Delta y \\cdot d}{f_y})$, where $d$ is a scaling factor that is determined by the depth, and $f_x$ and $f_y$ denoting the focal length of the rendering camera in the $x$ and $y$ axis. A similar process can be used for solving the 3D translation under different camera models.\n\n> **Q4. How many GS points are used to represent each component?**\n\nWe represent each object by a maximum of 20k Gaussians, while the static background is represented by a maximum of 200k Gaussians. We agree that the number of Gaussians has an effect on the quality of the 4D representations and we have compiled a table to ablate this hyperparameter:\n\n| # Gaussians | CLIP&uarr; | LPIPS&darr; | \n|-|:-:|:-:|\n| 10k| 84.57| 0.156 | \n| 20k (**Original**)| 85.09| 0.152 | \n| 25k| 85.14| 0.151 | \n\nWe observe the performance does improve with more Gaussians, however, it also results in an increased amount of compute and VRAM cost. We noticed that at around 20k Gaussians per object, there exists a wall of diminishing returns, which was how the hyperparameter was chosen in the experiments.\n\n>**Q5. For LPIPS metric evaluated on the DAVIS dataset, the reasons for better performance of Consistent4D over Dreamscene4D?**\n\nWe noticed that the LPIPS metric will ignore a lot of optimization artifacts like **floaters** which are more common in Consistent4D (Figure 4 of paper), as the backbone in LPIPS is trained for ImageNet Classification. Besides, current SOTA NeRF implementations (as used in Consistent4D) are also slightly better at NVS when the queried novel view change is large for in-the-wild videos compared to GS [1], so in the LPIPS metric, Consistent4D has a slight advantage as its main disadvantage (optimization artifacts) is largely ignored.\n\n[1] \"Radsplat: Radiance field-informed gaussian splatting for robust real-time rendering with 900+ fps.\" arXiv 2024.\n\n> **Q6. Limitation disclosure.**\n\nDreamScene4D handles real-world complex videos with multi-object movement. We summarized the paper's limitations in Sec. 4.3, as well as detailed failure case analysis in Sec. A.4. When the object is heavily occluded and small, the segmentation or inpainting operations may fail and the Gaussians will not attempt to explain the missing parts, as showcased in the top half of Figure 9. This can potentially be fixed by reducing the reliance on photometric RGBA rendering losses and incorporating semantic losses such as CLIP to measure the similarity of the rendered object in current frame with other frames. We also expect fewer artifacts using better diffusion-based inpainting models (e.g. SDXL-Inpainting). We will extend the current limitation section (paper) by merging failure cases discussion (A.4).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the positive and insightful comments from all four reviewers. We are glad that they found our paper "novel / innovative", "effective with comprehensive experiments" and "well-written". We will release our code to facilitate future research in fine-grained 4D understanding from videos.

We address all questions in separate responses to each reviewer, and will incorporate all valuable feedback into our paper.

Meta-review: Reviewers find the paper novel, effective, and the presentation well-structured. 
They also find the proposed "decompose-recompose" strategy innovative and the experimental validation comprehensive, demonstrating the method's superiority over existing approaches. 
While there were some initial concerns regarding computational complexity, reliance on external depth estimation, and handling of specific scenarios, the authors' detailed responses and additional experiments effectively addressed these issues. 
The overall consensus leans towards recognizing the paper's significant contribution to 4D scene generation from monocular videos.
Hence, my decision is to accept the paper.
Decision: Accept (poster)