DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos
Authors: Chieh Hubert Lin
Zhaoyang Lv
Songyin Wu
Zhen Xu
Thu Nguyen-Phuoc
Hung-Yu Tseng
Julian Straub
Numair Khan
Lei Xiao
Ming-Hsuan Yang
Yuheng Ren
Richard Newcombe
Zhao Dong
Zhengqin Li
Venue: NeurIPS 2025 poster ()
Keywords: Large reconstruction model
LRM
dynamic
Primary area: deep_learning
TL;DR: The first feed-forward reconstruction model for scene-level deformable 3DGS from a monocular video

Abstract: We introduce the Deformable Gaussian Splats Large Reconstruction Model (DGS-LRM), the first feed-forward method predicting deformable 3D Gaussian splats from a monocular posed video of any dynamic scene. Feed-forward scene reconstruction has gained significant attention for its ability to rapidly create digital replicas of real-world environments. However, most existing models are limited to static scenes and fail to reconstruct the motion of moving objects. Developing a feed-forward model for dynamic scene reconstruction poses significant challenges, including the scarcity of training data and the need for appropriate 3D representations and training paradigms. To address these challenges, we introduce several key technical contributions: an enhanced large-scale synthetic dataset with ground-truth multi-view videos and dense 3D scene flow supervision; a per-pixel deformable 3D Gaussian representation that is easy to learn, supports high-quality dynamic view synthesis, and enables long-range 3D tracking; and a large transformer network that achieves real-time, generalizable dynamic scene reconstruction. Extensive qualitative and quantitative experiments demonstrate that DGS-LRM achieves dynamic scene reconstruction quality comparable to optimization-based methods, while significantly outperforming the state-of-the-art predictive dynamic reconstruction method on real-world examples. Its predicted physically grounded 3D deformation is accurate and can be readily adapted for long-range 3D tracking tasks, achieving performance on par with state-of-the-art monocular video 3D tracking methods.

DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos
Chieh Hubert Lin 12
Zhaoyang Lv
Songyin Wu
Zhen Xu'
Thu Nguyen-Phuoc" Ming-Hsuad Hsgan Yang
Hung-Yu Tseng'
Julian Straub
Numair Khan
Lei Xiao
Yuheng Ren
Richard Newcombe
Zhao Dong'
Zhengqin Li
Meta
UC Merced
UC Santa Barbara
DGS-LRM (infer 0.6 sec)
Posed M Monocular Video 2FS
Deformable 3DGS Soene Flow
Our proposed Deformable Gaussian Splats Large Reconstruction Model (DGS-LRM) takes posed monocular videos as input and predicts deformable 3D Gaussians in single feedforward pass. Our approach enables both novel view synthesis and 2D/3D tracking, illustrated the right.
Figure 1:
Abstract
We introduce the Deformable Gaussian Splats Large Reconstruction Model (DGS- LRM), the first feed-forward method predicting deformable 3D Gaussian splats from a monocular posed video of any dynamic scene. Feed-forward scene recon- struction has gained significant attention for its ability to rapidly create digital replicas of real-world environments. However, most existing models are limited to static scenes and fail reconstruct the motion of moving objects. Developing a feed-forward model for dynamic scene reconstruction poses significant challenges, including the scarcity of training data and the need for appropriate 3D representa- tions and training paradigms. To address these challenges, we introduce several key technical contributions: an enhanced large-scale synthetic dataset with ground-trutt multi-view videos and dense 3D scene flow supervision; per-pixel deformable 3D Gaussian representation that easy to learn, supports high-quality dynamic view synthesis, and enables Long-range 3D tracking; and a large transformer network that achieves real-time, generalizable dynamic scene reconstruction. Qualitative and quantitative experiments demonstrate that DGS-LRM achieves dynamic scene reconstruction quality comparable to optimization- based methods, while signifi- cantly outperforming the previous predictive dynamic reconstruction method real-world examples. Its predicted physically grounded 3D deformation accurate and be readily adapted for long-range 3D tracking tasks, achieving performance on par with state-of-the-art monocular video 3D tracking methods.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Reconstructing a dynamic scene from monocular video, recovering accurate geometry, appear- ance, and motion, remains a significant challenge in computer vision and graphics. This task has numerous applications, including visualization, augmented/virtual reality (AR/VR), and robotics. Recent advances in this domain [46, 83, 97] have been largely driven by the development of neural representations, such as neural radiance fields [55] and 3D Gaussian splats [38], as well as deep priors for specific scene attributes like depth [85, 42, 48, 95, 66] and flow [75. 76, 37J. These methods tackle dynamic scene reconstruction by optimizing particular scene representations using densely captured images, integrating various deep priors to provide robust regularizations. Although latest methods using variants of Gaussian splatting [38] representation achieve real-time rendering [97, 88, 23, 83], the optimization process is often cime-consuming and computationally expensive, limiting their practical applicability.
Recent generalizable 3D feed-forward networks [85, 100, 30, 102, 94] directly predicts 3D rep- resentations from sparse-view image inputs, achieving speeds several orders of magnitude faster than previous perimization-basee methods. Inspired by the success of foundation models in nat- ural language processing [65] and 2D computer vision [19, 39], recent works explore training transformer-based [77] networks on large-scale 3D [14, 15] or video datasets to enable generalizable 3D reconstruction. However, prior efforts assume static scenes, leaving the challenge of handling dynamic scenes and accurately predicting motion still unsolved.
In this paper, we present DGS-LRM, the first feed-forward transformer designed to predict deformable 3D Gaussians from a posed monocular video. The predicted deformable 3D Gaussians can render novel view and scene flow of dynamic scenes. We introduce three key technical innovations to address this challenge.
First, employ per-pixel deformable 3D Gaussians represent predicted dynamic scenes, extending the recent success in feed-forward static 3D Gaussians prediction methods [102, 94, 10]. For each pixel in the target frame, method predicts its corresponding 3D Gaussian splat and 3D) scene flow from its current timestamp to all other timestamps in the input video, enabling high-quality novel view synthesis through warping 3D Gaussians across frames. Additionally, our representation is robust to occlusions and discontinuities. The predicted 3D scene flow can be chained together using sliding windows, achieving performance on par with the state-of-the-art 3D tracking method [90].
Second, motivated by the recent trend in training LRM with pure synthetic data [91, 35], we utilize multi-view synthetic data with ground-truth 3D scene flow as a primary supervision during training. This contrasts with prior scene-level feed-forward reconstruction models, which are typically trained on monocular video or RGB-D images. However, using pure photometric supervision
monocular
video has ambiguities in motion and geometry predictions for dynamic scenes. We create a customized large-scale dataset using Kubric [26]. featuring multi-view renderings paired with per-pixel 3D scene flow. Our results show that training on such a large-scale multi-view view synthetic dataset significantly enhances reconstruction quality and enables good generalization on real-world data as well.
Third, we employ temporal tokenization [4, 62] to compress input videos into compact, small pixel cubes. Unlike GS-LRM [102], which tokenizes each frame independently. our approach is computationally efficient and scales well for both training and inference. Additionally, we incorporate discretely sampled temporally distant reference frames as inputs to leverage frames with larger camera baselines, following [49], which effectively reduces geometric ambiguities.
Our best model achieves real-time inference on an A100 GPU while maintaining reconstruction quality on par with optimization-bant bassd deformable Gaussian splatting methods [97], which require hours of computation and several pre-trained models for initialization. Compared to the existing predictive dynamic object reconstruction method [67], DGS-LRM demonstrates significantly better generalization ability on real-world examples, improving PSNR by 3 points on [25]. Moreover, DGS-LRMM accurately reconstructs object motion. Our predicted 3D scene flow delivers competitive quantitative performance [90] on standard point tracking benchmarks [26. 107]. More qualitative results from DAVIS [60] further demonstrate high-quality novel-view synthesis and accurate flow prediction, shown in Figure 1.
2 Related Work
Dynamic reconstruction and view synthesis Early dynamic scene reconstruction methods [3, 33, 20, 57, 108] primarily focus on non-rigid mesh reconstruction, typically requiring RGBD images

---

as inputs. Video depth prediction methods predict consistent depth maps across video frames [40, 54, 105, 104] by integrating monocular depth priors with strong hand-crafted regularizations. Both of these approaches concentrate on geometry and are not equipped to support realistic novel view synthesis. Recent advancements in dynamic view synthesis have shown significant progress by incorporating novel neural representations, such as neural radiance fields [55] and 3D Gaussian splats [38], into dynamic scene reconstruction. Most of these methods require multi-view videos as inputs [52. 7, 9. 24, 44, 47, 72, 73, 79], which considerably limits their practical applicability. Several methods [51, 22, 23, 45, 58, 59, 78, 88, 89, 97, 96, 83, 29] target the more challenging task of monocular dynamic view synthesis. While these methods remarkably improve view synthesis quality, they heavily rely on geometry and motion priors from pre-trained models, alongside meticulously designed time-consuming optimization processes to achieve state-of- state-of-the-art reconstruction quality. On the contrary, DGS-LRM aims to learn priors from a single large transformer model, enabling efficient, generalizable dynamic scene reconstruction in feed-forward manner. In particular, PGDVS achieves high-quality novel view synthesis on the DyCheck [25] benchmark while significantly reducing reconstruction time compared to many prior optimization-basse methods. It leverages off-the-shelf depth and optical flow estimators for initialization. Uniquely, it renders novel views at different timestamps via image-space warping and aggregates results using dynamic object masks, which produce sharp appearances. However, the method still requires hours to build its representation, and in some challenging cases with complex motion, warping and masking errors can cause temporal flickering artifacts. In our work, we focus on an end-to-ens feedforward representation that can deliver similar outputs while being orders of magnitude faster in inference.
Feed-forward reconstruction Many feed-forward methods can predict neural 3D representations for novel view synthesis or geometry reconstruction. Early methods [11, 99, 81, 92] often struggle to match the reconstruction quality of optimization-
methods and, therefore, require fine-tuning [92, 11] to enhance their results. Recently, many works [85, 42, 13] have significantly increased network capacity by employing large transformer networks for better 3D reconstruction. Among these, large reconstruction models (LRMs) [30, 43, 93, 87, 80, 102] represent family of approaches that achieve state-of-ther -or-te--rttttttt-- novel view synthesis quality from sparse, posed, or unposed images. The latest LRM methods [102, 94, 98] predict pixel-aligned 3D Gaussians for realistic static scene-level view synthesis but fail to handle dynamic objects. Several concurrent works aim to reconstruct dynamic scene geometry [100, 48, 84] but not appearance and motion. The closest existing method [67] predicts time- dependent 3D Gaussians to reconstruct dynamic objects. However, we observe that it generalizes significantly worse on real-world examples, even when the dynamic objects are segmented out in the inputs. This may be due to the limited motion and appearance diversity in its training data and its 3D representation, which fails to model accurate scene flow. [49] achieves better generalization by training on self-curated internet videos, but it still does not reconstruct accurate scene flow as our model.
Flow and tracking The majority of tracking methods aim to find correspondences in 2D) image space. Classical optical flow methods [2, 6, 5, 32, 18, 75] estimate dense 21) pixel motion between two consecutive frames and, therefore, are not suitable for long-range tracking. Feature tracking methods [1, 16, 53, 69] can track pixels over long ranges, but only handle sparse points. Several efforts [71, 68] have been made to combine the merits of the two to achieve dense long-range 2D tracking, either by concatenating consecutive 2D flows through test-time optimization [56, 82) or by relying on data-driven approaches [17, 28, 37J. Another line of research estimates correspondences in 3D space to mitigate issues caused by 3D-2D projection and leverage better regularization. Most of these approaches require RGBD images or point clouds as imputs [27, 50, 63, 86, 34, 64, 74, 76] or test-time optimization [31, 46. 45] to jointly reconstruct geometry. Spatia/Tracker [90] is the closest state-of-the tf-te-----hhrrtt work that predicts dense 3D scene flow from monocular video. Experiments show that DGS-LRM achieves comparable tracking accuracy while offering more versatile framework that supports high-quality novel view synthesis.
3 Methodology
In Sec. 3.1, we introduce the network architecture and the convention of DGS-LRM. Sec. 3.2 outlines key training details. Sec. 3.3 describes synthetic training data.
3.1 DGS-LRM
Figure illustrates the inputs, transformer network architecture, and the predicted 3D representation of DGS-LRM, which we will discuss in detail.
3

---

DOS-LUM
Dedirmaaia CS
Prodicted
Dral-Vee Samphng
Readi tog Stperttioo
Un Indoommc
Debted Carbyry
Figure 2: DGS-LRM overview. We first concatenate multi-viww videos with a time- Plucker ray and tokenize them using the spatial-temporal tokenizer. Then, the transformers take the sampled time tokens as input and predict per-pixel deformable Gaussians with 3D scene flow. During training, we rendered multi-view synthetic videos using Kubric. We draw a dual-view ground-truth in each sample at the same timestamp, rendering views, depth, and scene flows.
Network inputs. Given monocular video sequence I (oo IN3 3 with per-frame camera parameters C = ( {Co..... Cv }. DGS-LRM aims to predict a deformable Gaussian splatting (DGS) reconstruction of the visible environment. Following LRM [30], we encode camera calibrations as Plücker rays, where each C₁ E C pairs with an image 1; E I with the same resolution (H x W). Each pixel p of Cn is -ivennnimenioonal vector containing a Plücker ray and a timestamp. The timestamp is the temporal index of the frame within the input time window (o..... N), and normalized to [o, 1].
Processing a large number of video frames using standardized self-attention transformer requires prohibitively large GPU VRAM. However, limiting the window size N can reduce the camera baselines among input frames, making more challenging for the model reconstruct accurate geometry. To address this issue, we follow BTTimer [49] and additionally introduce optional reference frames { Ro ... Rx} (omitting the corresponding Plücker rays for clarity). These reference frames aim to sample more views with larger camera baselines, which provides additional clues for geometry reconstruction. Consequently, we sample R temporally distant from I, and we do not predict per-pixel deformable 3D Gaussians for these frames.
Deformable Gaussian Splats. Given posed multiview images, DGS-LRM predicts a set of deformable Gaussian parameters by
G = C,)
where G = (Go, Gri,,,,,..,Ca GN). Here, I is the tempo-
ral downsampling rate, and Gz. is a set of deformable 3D Gaussians that are pixel-aligned with keyframe x 1 (O, In G..,. each pixel p contains a sin- gle Gaussian splat Ip. parametrized by 1-channel depth dz. 3-channel RGB colors, 4-channel quaternion rotation, 3 3-channel scale, 1-channel opacity, and a set of 3-channel deformation vectors f,- {fo. fi,..., , /N } to warp Gaus- sian splat Sp into the target timestamp n E (0,... N.. Our deformation vectors only model translation. We observe that deforming rotation will not further improve reconstruction quality, possibly because our per-pixel Gaussian representation is dense enough. Deforming opacity, on the other hand, allows the model to fake dynamic appearances without predicting accurate motion. To render a novel view image at timestamp n, we warp every Gaussian splats gp in G using the corresponding deformation vector fo to form a new set of 3D Gaussian W,, for rendering
Figure 3: A visualization of pixel- aligned deformable 3D Gaussians are used for novel view synthesis at arbitrary timestamp n.
W = UU warp(0p: fp)
(2)
A visualization of the warping process is shown in Figure 3.
Architecture. Our model consists of three components: an input tokenizer, a transformer, and output MIPss to project the results. The input images (both I and R) and their corresponding Plücker rays are first channel-concateenated together. We use temporal tokenization, inspired by the temporal autoencoder from MovieGen [62]. Instead of patchifying each image patch with s x

---

pixels, we consider video as volume and patchify a cube across spatial and temporal domains with s X s x pixels. This cube turned into token through a simple linear layer. We use l 1= 4 in practice, reducing the number of tokens by four times, which significantly accelerates both training and inference. In practice, the model not trainable and expensive to infer without temporal tokenization. We use standard transformer architecture [77] with 24 multi-head self-attention blocks. We additionally add weight normalization [70] to all parameters and observe that it can stabilize the training. Finally, we use two-layer MLPs to project the tokens into deformable GS parameters.
3.2 Training
Photometric losses Similar to prior works [102, 94], DGS-LRM can be trained end-to-end with photometric losses through differentiable rasterization. We sample set of ground-truth images == (To...... TQ) with corresponding camera poses C C1.-.., and define fg E {I,..., N as their timestamps. To produce a predicted rendering f we first deform the predicted 3D Gaussians to timestamp ny as Wn, then rasterize it to the output camera pose Cg with
I rasterize(Wnn: R,)
(3)
We then use MSE loss and perceptual loss [103] to supervise the rendered image.
Lise Σ MSE(I,, 1,); Lipips
Σ LPIPS(( 1n)
Output view selection and dual-view supervision. Static scene LRMs [30, 102] sample intermediate video frames from monocular videos as training data, which can lead to geometry and motion ambiguity when training with dynamic monocular videos. To remove this ambiguity, we use time- synchronized multi-view videos to train DGS-LRM.
We start with sampling two video sequences; one serves as the input I, while we sample K frames from the other sequence as R. For output view supervision, we empirically found that sampling two views at the same timestamp provides significantly better training convergence. We name this strategy dual-view sampling and show its effectiveness in Figure 7. With dual-view sampling, we sample two different training views for each timestamp from two synchronized video sequences, which retrieves Q/2 frames from each video for each training step. To ensure the inputs and outputs of DGS-LRM have sufficient covisibility, we allow one of the output sequences to overlap with either the input or the reference sequence.
Depth and scene flow supervisions. We apply depth supervision Laepth to improve surface geometry
G dedpph ΣΣ
11d dpll1-
(5)
Here, directly supervise with pixel-aligned ground-truth depth values dp for key frame {0, 22.....,N) which we observe to be more effective compared to supervising rendered depth maps at output views. Simply combining rendering loss and depth loss is still not enough for accurate motion reconstruction. We often observe discontinuities in the predicted deformation trajectories. Sometimes, the 3D Gaussian splats are even moved outside the camera frustum. Consequently, we introduce the flow loss Lnow to regularize the point deformation. Similar to the depth loss, for every key frame, we supervise our deformation vector f, with ground-truth deformation every timestamp. The flow loss is written as
Lioww ΣΣ fflli.
(6)
How to obtain ground-truth deformation vectors f, will be detailed in Sec. 3.3. Total loss. Finally, the whole network is end-to-end trained with the total objective
Luceal Linse + Aipips Lipips + Adepth Liephh + Aroow Laow
(7)
where the X's are weighting factors of the objectives. We use Appips = 0.5, Adepth = 10, and Agoo = 10.
Scene normalization. We found that inconsistent scene scales in training and inference can cause instability and generalization issues. We use a normalization approach similar MegaSaM [48] for both training and inference. We first use monocular metric depth estimator [61] to identify the scale of the scene and then normalize the scene scale so that the disparity of the 20th depth percentile is equal to 2. Such a scene scale is applied to camera poses, ground-truth depths, and scene flows.
5

---

3.3 Training Data Creation
Due to limited real-world posed monocular videos that contain diverse dynamics with sufficient ground truths (3D scene flow, multi-view images) to address the motion ambiguity, we primarily train our DGS-LRM on self-generated Kubric [26] dataset. We also explored combining it with various real-world videos for training but did not observe major improvements.
Customized Kubric. We follow the MOVi-E setting, using the Kubric engine [26] to create synthetic dynamic scenes. These scenes contain diverse objects being tossed around, simulated by a physics engine. To ensure that our video is closer to real-world monocular dynamic videos [25], we decrease the default maximum camera trajectory length from 8 meters to 0.5 meters to improve the model's s generalization ability. We generate the dataset with synchronized cameras. We first sample one camera, then sample 3 other cameras relative to the first one, with distances ranging from 4 to 16 meters. All these cameras share the same look-at points so that they can have sufficient co-visibility. To further reduce the sim-to-real domain gap, we apply additional domain randomizations. For each scene, we add motion blur to one of the cameras and make sure the camera is never being sampled as the output supervision but only as the input. We also sample varying focal lengths for each camera, ranging from 25mm to 55mm.
Scene Flow Extraction. We extract ground-truth 3D scene flow following MegaSaM [48]. The Kubric engine supports rendering per-pixel object coordinates, which specify the location where the ray cast from the camera intersects the object surface. Moreover, Kubric also records the per-frame object trajectory and rotations during physies simulation. Given that Kubric only contains rigid objects,
retrieve the 3D scene flow for every pixel at every timestamp by combining the 2 attributes together.
However, 3D scene flow has O(N x M x P 3) space complexity. In practice, using N = 24, M == 6, and P 512 512 results in 0.84 GB data for one camera. Loading a batch of such data in each iteration would exhaust the memory and data I/O, Fortunately, we observe that the scene flow is significantly sparse, where the majority points in the scene are stationary. By storing 3D scene flow as sparse tensors, we reduce the memory and I/O cost by 80%.
3.4 Flow Chaining
Our DGS-LRM is trained to handle short video clips (Is). Theoretically. it can be trained to handle longer videos, but this is computationally expensive in practice. However, we can chain multiple sequences of scene flows together to achieve long-range range tracking, which we will detail below.
Given two sequences with predicted scene flows, the flow chaining first set the end frame of the first video as the first frame of the second video. We deform the two independent deformable GS into the same timestamp. Then, find nearest neighbor for each scene flow temporally chain two flows into one. The nearest neighbor is measured by two distance quantities: the distance between the deformed GS, and the direction similarity between the momentary scene flow. For the first video, the momentary scene flow is the relative scene flow between the last two timestamps; for the second video, is the relative scene flow between the first two timestamps. The two distance quantities result in a six-value vector, and we measure the distance between these vectors with simple LI distance. We use Faiss [21] to efficiently compute such distance with GPU support. After computing all pair-wise distances between the scene flows from the first video and the scene flows from the second video, we index the nearest neighbor and temporally chain each pair of flows into one.
However, as some points in one video may be completely invisible in the second video (such as moving outside the frustum), some flows in one video may have valid match in the other video. We use a threshold to filter out such cases and concatenate these unmatched scene flows with zero values, meaning the points remain stationary after it loses track. These lost tracked scene flows are still evaluated when we compare them with the 3D tracking methods in Table 2. We also provide an idealized evaluation where these unmatched scene flows are excluded from the evaluation, marked as Flow Valid (FV) in Table 2.
Experiments
Hyperparametees. We train our method with 64 H100 GPUs with 80GB VRAM. For all variants of DGS-LRM, we use N = 24 input frames with temporal sampling rate 1 = 4, which results in keyframes after temporal tokenization. We use K 4 for reference views and set the number of output views per scene to Q 8. For training efficiency, we first train the model at 256

---

Table I: Monocular Dynamic View Syn- thesis on DyCheck [25]. DGS-LRM outper- forms LRM-based L4GM [67], and com- novel-view
Table 2: 3D Tracking on PointOdyssey [107]. DGS- LRM reconstructed 3D deformation field is compa- rable to state-of-the-ar 3D tracking methods. FC is flow chaining that combines scene flows from mul- tiple segments. FV is fully visible, which evaluates only the tracking points not occluded for more than 24 frames (DGS- LRM input length).
parable to optimization-basee synthesis methods with substantially faster reconstruction time. DynMask applies dy- namic mask to evaluate the foreground only.
Method
Frames PSNR
ATE (L) 6p.1 (1) 6 (1)
Method
Time
DynMask mPSNR (1) mLPIPS (1)
Chained RAFT3D [76]
120
NIA
0.70
0.12
0.25
D3DGS 197]
1-3 hours
11.92
0.66
Lifted CoTracker [37]
120
NIA
0.77
0.51
0.64
PGDVS [106] 3 hours
15.88
0.34
SpatiaTTrakker [90]
0.22
0.59
Ours
0.495 see
14.89
0.42
Ours (FC)
120
27.77
0.21
0.57
0.68
1.4GM [67]
4.8 sec
5.84
0.67
Ours (Native)
24
27.77
0.11
0.72
0,84
Ours
0.495 see
11.97
0.51
Ours (FC FV)
120
27.77
0.15
0.64
0.75
Time
Inputs
DADGS
PGDVS
LAGM
Ours
GT
Inputs
D3DGS
PGDVS
L4GM
GT
Figure 4: DyCheck iPhone dataset. Our DGS-LRM outperforms D3DGS [97] and does have warping artifacts as in PGDVS [106]. Both methods fail to recover the geometry and the repetitive motion of the windmill, We mask out the zero covisible regions with black pixels.
resolution and then fine-tune it at 512x512 resolution. We render the Kubric dataset according to these two setups and create 40,000 scenes (each with 4 synchronized cameras) for both resolutions. For the first stage of training, we use batch size of 15 per GPU, train for 40k iterations with learning rate of 4e 4, and then decay to le - 6 with cosine learning rate scheduler. For the second stage, we use batch size of 8 per GPU, train for 20k iterations with a learning rate of 1e 4, and then decay to 1e 6 with cosine learning rate scheduler. For both stages, for 500 iterations, which linearly ramps up the learning rate from 0 to the initial lcarning rate. Similar to GS-LRM, we apply the common practice to save GPU VRAMs using xFormers [41], deferred backpropagation [101], gradient checkpointing [12], and BF16 mixed-preed -exxeddprecision training [36].
a learning rate warm-up
4.1 Novel Dynamic View Synthesis
We evaluate DGS-LRM on DyCheck [25] and DAVIS [8]. DGS-LRM requires the input to be a temporally continuous video with non-stationary camera, and cannot process teleporting cameras with discrete poses. Due to the lack of real-world benchmarks with multiple non-stationary and synchronized cameras, our quantitative evaluation is limited DyCheck. We use the iPhone subset of DyCheck, which includes two synchronized novel-view cameras for reconstruction metrics evaluation. The iPhone subset contains long monocular videos, 200-400 frames each. addition, DyCheck also labels the covisibility between training and novel-view cameras and evaluates the masked version of reconstruction metrics. For fair comparison, all methods use the same structure- crmncuur-frommmotionn depth for initialization or scene normalization. DAVIS provides a large and diverse set of in-the-wijg monocular videos to test the method's generalization. We use MegaSaM [48] to create the camera poses for DAVIS, We only evaluate qualitative results, as DAVIS does not have novel-view cameras.
7

---

Input (Time A)
(Time B) Ours (Time A)
Ours (Time B)
(Time (Time B)
Figure 5: DAVIS dataset. DGS-LRM outperforms D3DGS in synthesizing thin details and can
predict consistent dynamic motion trajectories (visualized in color).
In Table 1 and Figure 4, we show that our DGS-LRM outperforms the baseline predictive method, L4GM [67]. while performing comparably to the state-of-the-ee optimization-but baasd reconstruction methods, which all take hours to complete. L4GM is designed for 4D object reconstruction without reconstructing explicit deformation. As L4GM only reconstructs dynamic foreground objects, label the foreground mask following [83]. then reconstruct and evaluate the masked region only. In Figure 4, both D3DGS [97] and L4GM often resolve wrong scene scale remi-transparent geometries, PGDVS [106] often creates broken geometry due to warping, while our method can handle them well. Neither D3DGS nor PGDVS correctly resolve the repetitive and fine-grained deformations in the paper windmill scene. In addition, all optimization baseline methods are optimized with the entire sequence, while our DGS-LRM only uses 24-frame continuous clip and 4 reference frames to reconstruct the scene. The 4 reference frames are selected [-96, -48, 48, 96) frames apart from the main input sequence. and increase the number of frames in the opposite direction when meeting the start end of the sequence. In addition, the covisibility labeling of DyCheck is dedicated methods optimizing with the full video sequence, which is disadvantageous to DGS-LRM with a small input window.
In Figure 5, show that DGS-LRM can perform well even on in-the-wild videos. Compared with D3DGS [97]. we can correctly reconstruct the thin geometries of the bike wheels and challenging scenes with water deformation. In flow visualizations, the complex deformations in hand motion and wheel turning while maintaining consistent flow in the train's rigid body movements, For visual clarity, we randomly downsample the per-pixel dense flows into sparse flows and then mask out the background flows using the object mask provided by DAVIS.
illustrate that DGS-LRM effectively tracks
4.2 3D Tracking
An accurate deformable 3D reconstruction should have 3D deformation field aligned to the phys- ically grounded 3D tracking trajectories. In Table and Figure 6, we evaluate the quality of the reconstructed scene flow on the PointOdyssey benchmark [107]. PointOdyssey includes 13 videos (ranging from 1,000 frames to 4,000 frames) of synthetic scenes with humanoid and animal meshes articulated with transferred real-world motions. We compare DGS-LRM with the -fate-of-the-an Spa- tial'racckrr [90] and two baselines proposed in it. DGS-LRM with flow chaining achieves comparable performance with Spatial/Tracker and performs better than other baselines. In Figure 6, we found that SpatialTracker struggles on texture-less surfaces, where the tracking points drift over time in the regions with similar textures. In contrast, as DGS-LRM prediction involves a total reconstruction of the object geometry with accurate depth prediction, it can better track points in completely texture-less areas. For instance, the point registration on the ear root of the rabbit and the knee of the humanoid. As discussed in Sec 3.4, the chaining process involves failures when some points are completely invisible in a video segment. Therefore, we provide two additional variants: the native tracking performance without chaining (marked as Native) and an oracle case where we omit the flows with significant discontinuity (threshold by L1 distance 0.1 meters) during the chaining process, which is marked FC FV. While not directly comparable SpatialTracker, these two variants show the actual quality of the predicted scene flows without the impairment caused by flow chaining.
8

---

Input Vid
Initial
SputialTracker
Figure 6: Qualitative comparisons to SpatialTracker on the PointOdyssey dataset. DGS- LRM shows better performance and consistency in texture-less areas. SpatialTracker predicts certain tracks inconsistent with the object's moving direction. Such several tracking points drift and collide in the humanoid's knee.
Table 3: Ablation Study. We show that each component contributes to the final performance. Note that the study is conducted with low-resolution models trained 256 resolution. (TT: temporal tokenization. DV: dual-view sampling. SF: scene flow loss. RF: reference frames.)
6.38
Method
DyCheck
Kubric MV (Test)
0.25
mPSNR (t)
mLPIPS
mPSNR
mLPIPS (4)
3020
10300
15000
22000
22000
50002
32000
40003
TT
0OM
OOM
Figure 7: Dual view supervision (blue curve) provides better training conver- gence in Perceptual Loss (LPIPS) com- pared its counterpart (orange curve).
w/o DV
14.72
0.412
25.77
0,171
SF
14.29
0.423
25.06
0.189
w/o RF
13.91
0.438
24.69
0,186
Full
14.67
0.412
26.05
0,161
4.3
Ablation Study
In Table 3, show that each proposed components contribute to the final performance. The ablation is conducted at the 256 resolution training stage, and we compare the rendering quality on the DyCheck benchmark. The temporal tokenization enables training at scale by significantly reducing the memory consumption. In Figure we show that the dual-view sampling improves the training convergence Kubric and leads to better performance our additionally rendered Kubric holdout test set (with new objects and HDRIs unseen during training). The scene flow objective and reference frames both significantly boost the performance by large margin. In practice, found the scene flow loss significantly improves the rigidity of the deformation, and the reference frames help solve the scene scale and depth by triangulating with distant views.
In the right table, we evaluate static GS-LRM on DyCheck dataset, quantitatively showing the significance of model- ing the deformation. Additionally, despite our DGS-LRM only emits 3D Gaussians at down-sampled keyframes, we show that the performance gap between the keyframes and non-keyframes negligible.
Method
LPIPS
LRM (static)
13.02
0.444
DGS-L.RM (keyframes)
15.04
0.419
DGS-LRM (non-kcyframes)
14.84
0.421
DGS-LRM (all)
14.89
0.420
Conclusion and Limitations
We introduce DGS-LRM, the first feed-forward network capable of predicting deformable 3D Gaussians from a posed monocular video in real-time. The predicted deformable Gaussians enable novel-view rendering, geometry reconstruction, and 3D scene flow estimation in world space. We train DGS-LRM on a large-scale multi-view rendered synthetic dataset and show that it generalizes well to real-world videos of varying complexity. Unlike prior monocular deformable 3D reconstruction methods, which require lengthy optimization to reconstruct and fuse priors from multiple individual networks. demonstrate the potential of predicting deformable 3D Gaussians end-to-end and learning dynamic scene priors within a single network.
Limitations. DGS-LRM has a few limitations that can be explored in future works. As the model is trained with temporally continuous video, cannot handle discrete image frames. Our predicted scene flow cannot handle extremely large motion in the scene, which may stem from the motion distribution of the physically simulated synthetic dataset. Although our flow chaining enables continuous 3D tracking, it still causes a noticeable jump in appearances for novel view synthesis. The input video baseline and distribution significantly influence the quality of novel view rendering quality. As the view deviates from the input trajectory, artifacts gradually intensify in the rendered images as well.

---

References
[1] Herbert Bay, Tinne Tuytelaars, and Lue Van Gool. Surf: Speeded up robust features. In ECCV, pages 404-417, 2006. 3
[2] M J Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In ICCV. pages 231-236, 1993.
[3] Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and Matthias Nießner. Deepdeform: Learning non-rigid rgb-d reconstruction with semi-supervised data. In CVPR, pages 7002-7012, 2020. 2
[4] Tim Brooks. Bill Peebles, Connor Holmes, Will DePue, Yufei Guo. Li Jing, David Schnurr. Joe Taylor, Troy Luhman. Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators, 2024.
[5] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert, High accuracy optical flow estimation based on a theory for warping. In ECCV, pages 35-36, 2004. 3
[6] Thomas Brox, Christoph Bregler, and Jitendra Malik. Large displacement optical flow. In CVPR, pages 41-48, 2009. 3
[7] Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with a layered mesh representation. ACM Transactions on Graphics, 39(4):86-1, 2020. 3
[8] Sergi Caelles, Alberto Montes, Kevis-Kokitsi Maninis, Yuhua Chen, Luc Van Gool, Federico Perazzi, and Jordi Pont-Tuset, The 2018 davis challenge on video object segmentation.
2018. 7
[9] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR, pages 130-141, 2023. 2023. 3
[10] David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann, pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In CVPR, pages 19457-19467, 2024. 2
[11] Anpei Chen, Zexiang Xu, Fugiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pages 14124-14133, 2021.
[12] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR, abs/1604.06// 0744..117.2.4 2016.
[13] Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In ECCV, pages 770-386,22 2024.
[14] Malt Deilke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, pages 13142- 13113, 2023. 2
[15] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte. Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. In NeurlPS, 2024. 2
[16] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. In CVPR, pages 224-236. 2018. 3
[17] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In ICCV, pages 10061-10072, 2023.
[18] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt. Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, pages 2758-2766. 2015. 3
[19] Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 20Xiv: 19999, 2020.
2
10

---

[20] Mingsong Dou, Sameh Khamis. Yury Degiyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle. Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, al. Fusion4d: Real-time performance capture of challenging scenes. ACM Transactions on Gruphics, 35(4):1-13, 2016.
[21] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Fimmre-Emmanul Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281, 2024. 6
[22] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and vidco processing. In ICCV, pages 1330414314, 2021. 3
[23] Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH, pages 1-11, 2024. 3
[24] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, pages 12479- 12488, 2023. 3
[25] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Dynamic novel-viow synthesis: A reality check. In NeurIPS, 2022. 2, 3, 6,
[26] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilum Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann. Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laadjii, Hsuch-Ti Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzerahrai, Cengiz Oztireli, Etienne Por, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wa, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: scalable dataset generator. CVPR, 2022. 6
[27] Xiuye Gu, Yijie Wang, Chongruo Wu, Yong Jae Lee, and Panqu Wang. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In CVPR, pages 3254-3263, 2019. 3
[28] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In ECCV, pages 59-75, 2022. 3
[29] Jixuan Hc, Chieh Hubert Lin, Lu Qi, and Ming-Hsuan Yang. Restage4d: Reanimating deformable 3d reconstruction from single video. In NeurlPS, 2025. 3
[30] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: Large reconstruction model for single image 3d. In JCML, 2024. 2, 3,
4, 5
[31] Junhwa Hur and Stefan Roth. Self-supervised monocular scene flow estimation. In CVPR, pages 7396-7405, 2020. 3
[32] Eddy Ilg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox Flownet 2.0: Evolution of optical flow estimation with deep networks. CVPR, pages 2462-2470, 2017. 3
[33] Matthias Innmann, Michael Zollhöfer, Matthias Nicßner, Christian Thcobalt, and Mare Stamminger. Volumedeform: Real-time volumetric non-rigid reconstruction. In ECCV, pages 362-379, 2016. 2
[34] Mariano Jaimez, Mohamed Souiai, Javier Gonvalez-Jimenez, and Daniel Cremers. A primal-dual framework for real-time dense rgb-d scene flow. In ICRA, pages 104. 2015. 3
[35] Hanwen Jiang, Zexiang Xu, Desai Xie, Ziwen Chen, Haian Jin, Fujun Luan, Zhixin Shu, Kai Zhang, Sai Bi, Xin Sun, Jiuxiang Gu, Qixing Huang, Georgios Pavlakos, and Hao Tan. Megasynth: Scaling up 3d scene reconstruction with synthesized data. 2024. 2
[36] Dhiraj D. Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang. Hcctor Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradeep Dubey. A study of BFLOAT16 for deep learning training. CoRR, abs/1905. 12322, 2019. 7
[37] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In ECCV, pages 18 35, 2024. 2, 3,
11

---

[38] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühher, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4))139-1, 2023. 2,
[39] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Yen Lo, et al. Segment anything. ICCV, pages 4015 4026. 2023.
[40] Johannes Kopf. Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In CVPR, pages 1611-1621, 2021.
[41] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore. Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizow. xformers: A modular and hackable transformer modelling library. 2022. 7
httpa://github. httptttbbb.ooo/accccoeeeeeeeeeee httpacccbbboeeeeeeeeeeeeeeeeeeee
[42] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In ECCV, pages 71-91, 2024. 2,
[43] Jiahao Li, Hao Tan. Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong. Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.062/ 2023. 3
[44) Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In CVPR, pages 5521-5531, 2022. 3
[45] Zhengqi Li. Simon Niklaus. Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, pages 6498 6508, 2021. 3
[46] Zhengqi Li, Qianqian Wang. Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In CVPR, pages 4273-4284, 2023. 2,3
[47] Zhan Li. Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In CVPR, pages 8508-8520, 2024. 3
[48] Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang. Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. arXiv preprint arXiv:2412.044463 2024. 2, 2, 6,
[49] Hanxue Liang, Jiawei Ren, Ashkan Mirzaei, Antonio Torralba, Ziwei Liu, Igor Gilitschenski, Sanja Fidler, Cengiz Oztireli, Huan Ling, Zan Gojeic, et al. Feed-forward bullet-time reconstruction of dynamic scenes from monocular videos. arXiv preprint arXiv:2412.03526, 2024. 2,3,4
[50] Xingyu Liu, Charles R Qi, and Leonidas Guibas. Flownet3d: Learning scene flow in 3d point clouds. In CVPR, pages 529-537,200 2019. 3
[51] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Tseng, Ayush Saraf, Changil Kim, Yung-Yo Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In CVPR, pages 13-23, 2023. 3
[52] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv: 1906.07751, 2019.
[53] David G Lowe. Distinetive image features from scale-invariant keypoints. IJC,, 60:91-110, 2004.
[54] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM Transactions on Graphics, 39(4) 2020. 3
[55] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 6591)999- 2021. 3,3
[56] Michal Neoral, Jonáš Serych, and Jif Matas. Mft: Long-term tracking of every pixel. In WACV, pages 6837-6847, 2024. 3
[57] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time, In CVPR, pages 343 353,3,2,00 2015. 2
12

---

[58] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo BaralinBruuall.. Nerfies: Deformable neural radiance fields. In ICCV, pages 5865-5874, 2021.
[59] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Scitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106. 002/1000..22232. /3228, 2021.
[60] Federico Perazzi, Jordi Pont- Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR, pages 724-732, 2016. 2
[61] Luigi Piccinelli, Yung-Hsp Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Lue Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR, pages
2024.
[62]
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apporv Vyas, Bowen Shi, Chih- Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang. Geet Sethi, Guan Pang. Haoyu Ma, Ishan Misra, Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li. Luxin Zhang, Mannat Singh, Mary Williamson, Mat Le. Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya. Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Ven-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng. Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani. Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: A cast of media foundation models, 2025. 2,
[63] Gilles Puy, Alexandre Boulch, and Renaud Marlet. Flot: Scene flow on point clouds guided by optimal transport. In ECCV, pages 527-544, 2020.
[64] Julian Quiroga, Thomas Brox, Frédéric Devernay, and James Crowley. Dense semi-rigid scene flow estimation from rgbu images. In ECCV, pages 567-582, 2014.
[65] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. 2
[66] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pages 12179-12188, 2021.
[67] Jiawei Ren, Cheng Xie, Ashkan Mirzaei, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, Huan Ling, et al. L4gm: Large 4d gaussian reconstruction model. NeurlPS, pages 56828-5666 56858, 2024. 2, 3, 7,8
[68] Michael Rubinstein, Ce Liu, and William Freeman. Towards longer long-range motion trajectories. In BMVC, 2012. 3
[69] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternttive sift surf. In ICCV, pages 22611, 2011. 3
[70] Tim Salimans and Durk P Kingma. Weight normalization: A simple reparamettorization accelcrate training deep neural networks. In NeurlPS, 2016.
[71] Peter Sand and Seth Teller. Particle video: Long-range motion estimation using point trajectories. IJCV, 80:72-91, 2008. 3
[72] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 29(5):2732 2742, 2023.
[73] Timo Stich, Christian Linz, Georgia Albuquerque, and Marcus Magnor. View and time interpolation in image space. In Computer Graphics Forum, pages 1781-1787, 2008.
[74] Deqing Sun, Erik B Sudderth, and Hanspeter Pfister. Layered rgbd scene flow estimation. In CVPR. pages 556.556,20 2015.
[75] Zachary Teed and Jia Deng. Raft: Recurrent all all-pairs field transforms for optical flow. In ECCV, pages 402 419, 2020. 3.3
13

---

[76] Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. In CVPR, pages 8375-8384, 2021. 3,7
[77] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017. 2,
[78]
Chaoyang Wang, Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprint arXiv:2105,0000000 05994, 2021. 3
[79] Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In CVPR, pages 13524-13534, 2022. 3
[80] Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Weaping Wang. Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311. /20244, 2023. 3
[81] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan. Howard Zhou, Jonathan T Barron, Ricardo Martin- Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image- based rendering. In CVPR, pages 4690 46000-4669.010 2021.
[82] Qianqian Wang, Yen-Yu Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In ICCV, pages 19795-19806, 2023. 3
[83] Qianqian Wang, Vickic Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint 77Xiv22407.13744 2024. 2,
[84] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexci A, Efros, and Angjon Kanazawa. Continuous
3d perception model with persistent state, 2025.
[85] Shuzhe Wang, Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, pages 20697-20709, 2024. 2,
[86] Zirui Wang, Shuda Li, Henry Howard-Jenkins, Victor Prisacariu, and Min Chen. Flownet3d+4: Geometric losses for deep scene flow estimation. In CVPR, pages 91-98, 2020. 3
[87] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404. 12385, 2024. 3
[88] Guanjun Wu. Taoran Yi, Jiemin Fang. Lingxi Xie, Xiaopeng Zhang, Wei Wei. Wenyu Liu, Qi Tian, Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. CVPR, pages 2024. 2,3
[89] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free- viewpoint video. CVPR, pages 9421 2211-9331.20 2021.
[90] Yuxi Xiao, Qianqian Wang. Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any pixels 3d space. In CVPR, pages 20406-20417, 2024. 2, 3, 7, 8,
[91] Desai Xie. Sai Bi, Zhixin Shu. Kai Zhang, Zexiang Xu, Yi Zhou, Sören Pirk. Arie Kaufman, Xin Sun, and Hao Tan. Lrm-zero: Training large reconstruction models with synthesized data, arXiv preprint 09337,, 2024. 2
[92] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In CVPR, pages 5438-5448, 2022. 3
[93] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.0922
2023.
[94] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. In ECCV, pages 1-20, 2024. 2.
[95] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, pages 10371-10381 2024. 2
[96] Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint Xiv::2100000-422 2023. 3
14

---

[97] Ziyi Yang. Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR, pages 20331-20341, 2024.
2, 3,3,7, 8
[98] Junqi You, Chich Hubert Lin, Weijie Lyu, Zhengbo Zhang, and Hisan Yang. Instainpaint: Instant 3d-scene inpainting with masked large reconstruction model. In NeurlPS, 2025. 3
[99] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from few images. CVPR, pages 4578 4587, 2021.
[100] Junyi Zhang, Charles Herrmann, Junhwa Hur. Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry the presence of motion. arXiv preprint urXiv:2410.0382. 2024. 2,3
[101] Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely. Arf: Artistic radiance fields. In ECCV, page 717-733, Berlin, Heidelberg, 2022. 7
[102] Zhang. Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In ECCV, pages 1-19, 2024. 2, 3,5
[103] Richard Zhang, Phillip Isola, Alexei Efros, Shechtman, and Oliver Wang. unreasonable effectiveness of deep features as a perceptual metric. CVPR, 2018.
[104] Zhoutong Zhang, Forrester Cole, Richard Tucker, William Freeman, moving objects in video. ACM Transactions on Graphics, 40(4):1-12, 2021. 3
Dekel. Consistent depth
[105] Zhoutong Zhang, Forrester Cole, Zhengqi Li. Michael Rubinstein, Noah Snavely, and William Freeman. Structure and motion from casual videos. In ECCV, pages 20-37, 2022.
[106] Xiaoming Zhao, R Alex Colburn, Fangchang Ma, Miguel Ángel Bautista, Joshua M. Susskind, and Alex Schwing. Pscudo-generrliized dynamic view synthesis from a video. In /CLR, 2024. 7, 8
[107] Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstoin, and Lconidas J. Guibas. Pointodysscy: A large-scale synthetic dataset for long-term point tracking. In ICCV, 2023. 7,8
[108] Michael Zollhöfer. Matthias Nieliner. Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, et al. Real-time non-rigid reconstruction using an rgh-d camera. ACM Transactions on Graphics, 33(4):1-12, 2014. 2
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose a feed feed-forward framework for deformable 3DGS reconstruction. We benchmark our model the commonly used benchmarks to show its effectiveness and generalization. We also evaluate the scene flow quality on a benchmark to show the quality and continuity of our deformation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include such discussion in the last section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
This paper does not propose new theories or mathematical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We reported all the architectural design, training data creation, and hyper- parameters with our best effort. Other unspecified hyperparameters follow the originally released codes papers of the cited works.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code release requires additional review. We will put best effort into releasing the codes and the pretrained model.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
wwwasssioollll amPllicy)) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /ttpppccpppblliiepppp hoplllicggiiesss
seabbmmmsssonPae ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we put our best effort into including all the implementation and experiment setting details. We followed the previous proposed evaluation protocol as closely as possible.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our framework involves large-scale and resource- innturceinttesive training. It is infeasible report a meaningful error margin with multiple individually trained models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, report the GPUs for training the model and execution time. However, we do not keep track the past experiments, thus challenging to report meaningful total resource consumption throughout the project.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp://neuuippll http///uurrr..pppuulieeeeeeeeeee cctpublic//eieecccccecccccc Coublic/Ethicsaiie wtw.chhhccideeeeeeee
Answer: [Yes]
Justification: We put our best effort into complying with NeurIPS Code of Ethics and all privacy concerns.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work improves the inference speed of deformable 3D reconstruction, which aims to recreate real-world environment. Our method does not allow altering content or spreading fake information. We comply with privacy considerations, our training data does not include human or unlicensed data.
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our dataset is purely synthetic without unlicensed content. Our model is reconstruction model that preserves the real-world information without any alteration.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us, should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We utilize pure synthetic data during training. The Google Scanned Objects (GSO) assets are licensed under CC-BY 4.0. The PolyHaven HDRI is licensed under CCO.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We only provide alterations to an existing simulation engine. We do not include additional data propose new datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not use human-related data for model training.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We did not use LLM in developing the research. The only potential use of LLM is grammatical error correction after writing is completed.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: htpps://neute wwccc//////o/eeeeeeeeee ccCCConferenness/222/LL. for what should or should not be described.
22


Reviews:
[{"review_id": "FA9A12tpGI", "reviewer": "Reviewer_fXV6", "summary": "This paper presents the Deformable Gaussian Splats Large Reconstruction Model, the first forward model to predict dynamic Gaussians for handling dynamic scenes. Specifically, the proposed model takes a video sequence and corresponding camera poses as input and directly outputs deformable dynamic Gaussians, enabling novel view synthesis of dynamic videos. By adopting a feed-forward approach instead of optimization-based methods, the model significantly reduces inference time while maintaining competitive quality. The main contributions are:\n\n1、A deformable 3D Gaussian representation combined with a keyframe-based rendering strategy for efficient Gaussian generation and rendering.\n\n2、A large-scale synthetic dataset for training and evaluation.\n\n3、A novel Transformer-based architecture that reduces memory consumption.", "questions": "1.\tWhy do deformations such as rotation or opacity in Deformable Gaussian Splats have little effect on quality improvement?\n\n2.\tWould incorporating Frame Attention inspired by VGGT further enhance performance?\n\n3.\tIn the keyframe-based rendering strategy, is there a noticeable quality gap between keyframes and non-keyframes?\n\n4.\tThe demo does not include long videos or scenes with large viewpoint changes. Will the model’s performance degrade under these conditions?\n\n5.\tPlease provide ablation studies on the number of parameters, the consumption of GPU memory and quality.\n\nIf the above questions are addressed effectively, or if there is a clear open-source plan, I will consider raising my score accordingly.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe design of the deformable 3D Gaussian representation combined with the keyframe-based rendering strategy is highly novel.\n\nThe feed-forward approach addresses the limitations of optimization-based methods, significantly improving speed while maintaining good quality.\n\nA rich dataset has been collected, which—if open-sourced—could greatly benefit future research in this area.\n\nWeaknesses:\n\nThe training and inference processes are unacceptable for most researchers, making the model difficult for others to reproduce or deploy.\n\nThe method struggles with large object motions within videos, leading to my worries about significant domain gaps; more experiments are needed to clarify this limitation. Please see the questions.", "was_revised": true, "final_justification": "This work makes three key contributions: (1) a novel deformable 3D Gaussian representation with keyframe-based rendering, (2) an efficient feed-forward approach that outperforms optimization-based methods in speed while maintaining quality, and (3) a valuable dataset that could benefit the community if released. The combination of innovation and practical impact strengthens this work.", "rebuttal": "We thank the reviewer for their constructive feedback and recognition of our work. We appreciate their noting that our representation, combined with the key-frame rendering strategy, is \"highly novel,\" and that our feed-forward method succeeds in \"significantly improving speed while maintaining good quality.\"\n\n&nbsp;\n\n**Reproducibility** We thank the reviewer raising the interest of our work and the impact of the datasets, which we deeply agree as well. We are preparing the release of our code, model and dataset. We are pending formal legal review to confirm the promise but we are working hard towards achieving this goal.\n\n&nbsp;\n\n**Fixed rotation and opacity** For *rotation*, the per-pixel deformable 3D Gaussians are densely distributed in the foreground area. Each Gaussian typically has a small covariance. Therefore, we empirically did not observe any impact from fixed rotation. For *opacity*, our pilot study indicates that allowing changes in opacity enables the model to \"fake\" dynamic appearances without accurately predicting the deformation field, which negatively impacts tracking accuracy.\n\n&nbsp;\n\n**Frame attention** Our preliminary experiments do not show significant improvements with Frame Attention under DGS-LRM's setting. We assume this is because our method's inputs include camera poses. But we also observe that Frame Attention is very effective in pose-free setting. \n\n&nbsp;\n\n**Keyframes and non-keyframes** Thank you for the insightful suggestion! We conducted a quantitative comparison of view synthesis quality for keyframes and non-keyframes respectively on the Dycheck dataset. We observe a moderate degradation on non-keyframes compared to keyframes, which demonstrates the generalization ability of our method. We will include these results in a revised version of the paper for more comprehensive experiments. \n\n**Table 1** Quantitative comparisons of view synthesis quality for keyframes and non-keyframes on the Dycheck Dataset\n|  Methods | PSNR | LPIPS |\n| ------------ | -------------- | -------------- |\n| Keyframes | 15.04 | 0.419 |\n| Non-keyframes | 14.84 | 0.421 |\n| DGS-LRM | 14.89 | 0.420 |\n\n&nbsp;\n\n**Large viewpoint changes and long video** For *viewpoint changes*, as mentioned in the limitations, our view synthesis quality deteriorates as the viewpoints deviate from the input trajectory. This is also true for other view synthesis methods. To alleviate this issue, we leverage dual view supervision from synthetic data and temporally distance reference frames as extra inputs. Both the qualitative results in Figure 4 and the quantitative results in Table 1 demonstrate that DGS-LRM achieves better quality compared to the SOTA feed-forward method and some optimization-based methods on the Dycheck dataset, even with large viewpoint changes.\n\nFor *long videos*, while we design flow chaining method to connect the scene flow from 2 sets of frames, it still causes a noticeable jump in appearance. Further works with a larger context window is required to address this challenge, such as an architecture similar to long LRM [1]. We will provide more detailed explanations in the limitations section of our revised paper. \n\n\n&nbsp;\n\n**Ablation studies** We empirically find that further increasing the model size does not significantly reduce the training loss. Due to constraints on training time and cost, it is challenging for us to train multiple versions with varying model sizes. Our current model has 323 million parameters. Training at a batch size of 4 per GPU takes 80GB of VRAM. We fully agree that verifying whether DGS-LRM follows the scaling law is an important research problem, and we leave this for future work.\n\n> [1] Ziwen, Chen, et al. \"Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats.\" arXiv preprint arXiv:2410.12781 (2024)."}, {"review_id": "CH5W0nvNLZ", "reviewer": "Reviewer_zCHb", "summary": "The paper presents DGS-LRM, a novel feed-forward method for real-time dynamic scene reconstruction from posed monocular videos using deformable 3D Gaussian representations.\nIt augment GS-LRM to handle dynamic reconstruction by predicting on keyframes pixel-aligned gsplat, where each gsplat contains additional translation vectors that warps the gsplat to any other frames. To decrease enormous memory demand, the image patches in GS-LRM is replaced with spatial-temporal volumetric patches. \nThe model is learned with strong supervision and ground truth 3D scene flow.\nThe authors created a new dataset with synchronized camera and scene flow from Kubric.", "questions": "- Why is the DyCheck reconstruction so much blurrier than PGDVS? \n- Minor: the use of \\sum in equation 2 should be replaced by a union?", "limitations": "- Handling unseen regions ungracefully. The representation itself is based on strict warping from input views, causing the disoccluded parts to be just black. I don't entirely agree with the authors that \"the invisible area is expected to remain unreconstructed\". One benefit of ML is to use data priors to fill the holes. \n- Scalability. It's hard to imagine the system could scale up to higher resolution and more dynamic scenes. The data creation dictates that the motion mode only covers rigid motion. The all-to-one warping means huge amount of redundant gaussians will be rendering.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n- The paper tackles a very timely topic and a very challenging task. \n- It provided a new dataset and it looks like the trained model generalizes reasonably well to the test data. \n- Strong performance in 3D tracking.\n\n# Weakness\n- Overall, I find the methodology section quite hard parse. \n- The proposed model appears to be the very definition of a \"brute-force\" attempt. Specifically,\n  - all scene flow is directly supervised. \n  - Eq 2 suggests that the prediction at the target view fuses all input views pixel-aligned GS to the target view through warping. It means the target view will be rendered with a significantly amount of redundant GS. How does the model scale with high resolution? How many GS are we talking about? Will there be memory issues? \n  - Dynamic sparsity is addressed in terms of dataloading, but not in the rendering / representation.\n- In the result, the contending model PGDVS seems to work really well, in fact better than the proposed model. Some explanation was mentioned in the README.pdf about how the evaluation is disadvantageous for the proposed model. Yet, the wording is confusing to me.\n- With PGDVS being a strong contender in the comparison, I expect to see more information about it in the related work section. At least, there should be a baseline paragraph, introducing the comparing methods.", "was_revised": true, "final_justification": "Given the context, even though the paper has many issues, I think it still paved way for the advance of this very active research field. It should serve as a solid baseline for feedforward dynamic scene reconstruction.", "rebuttal": "We thank the reviewer for their constructive feedback and recognition of our work. We appreciate their noting that our method \"tackles a very timely topic and a very challenging task\", \"generalizes reasonably well to the test data\" and achieves \"strong performance in 3D tracking\". \n\n&nbsp;\n\n**Writing of the method section**  Thank you for the constructive feedback! We noted that reviewer qSmi mentioned that \"our paper is mostly well-written and easy to follow\" and provided several suggestions to improve the clarity of the paper. We will include changes in our rebuttal to their review in a revised version of the paper, including more detailed explanations of key concepts, such as dual view supervision and reference views, as well as a better visualization for the flow chaining method.\n\n&nbsp;\n\n**Model design** Thank you for the insightful feedback! First, we would like to highlight that our method is one of the first attempts to build a feed-forward approach for monocular dynamic scene reconstruction. It is also the first method to jointly predict both 3D Gaussians and deformation fields, supporting novel view synthesis and tracking simultaneously.\n- *Scene flow*: Our deformation prediction similarly follow the common learning-based 2D and 3D flow prediction methods that also heavily benefit from direct flow supervision, with representative works such as CoTracker [1], SpatialTracker [2], RAFT [3], TAPIR [4], as well as classical works like FlowNet [5] and FlowNet 2.0 [6]. In our pilot study, we put significant effort into testing whether the model could learn deformation fields from rendering loss supervision. Finally, we concluded that ground-truth supervision is necessary to ensure physical meaningful tracking in the monocular video setting and we incorporate them as default in this work. We believe future work can relax this constraints and push this direction further.  \n- *Large number of 3D Gaussian points*:While it is true that a per-pixel Gaussian is an over-complete representation, it offers unique advantages such as fast convergence and high-quality texture details. Consequently, numerous recent feed-forward reconstruction methods adopt it as the scene representation, including PixelSplat [7], GS-LRM [8], GRM [9], Long-LRM [10], FLARE [11], etc. DGS-LRM follows this trend. For a 1-second video with 4 key frames, DGS-LRM generates $512 \\times 512 \\times 4 = 10^{6}$ Gaussian points. We have empirically found that this does not cause memory issues under our current settings. To scale up to higher resolutions, we may adopt Long-LRM's architecture and strategy, using opacity values to filter out redundant Gaussians, which we leave for future work.\n- *Dynamic sparsity*: We have empirically found that a per-pixel deformation field fits well with a per-pixel Gaussian representation. We attempted to further explore dynamic sparsity by predicting the motion field from a K-plane representation in our pilot study, but found that it converged very slowly.\n\n&nbsp;\n\n**Comparisons with PGDVS** We agree PGDVS is a strong baseline but we share a different objective. Our DGS-LRM is a general feed-forward method that emphasize on learning priors from data. It can run three orders of magnitude faster at inference while being closely on par with quality delivered by PGDVS.  We observe that DGS-LRM can perform slightly better in some cases involving complex motion, such as the pinwheel example in Figure 4. This improvement is due to our method predicting more accurate deformation fields. We also note that PGDVS reconstructs texture details more effectively, as reflected in both the qualitative and quantitative results, which serves as a strong reference for learning based method to catch up. In the revised version of the paper, we will avoid using the phrase \"comparable quality\" and instead provide a more specific description of the advantages of each method.\n\nWe will add the following descript of PGDVS to the related works:\n\nPGDVS achieves high-quality novel view synthesis on the DyCheck benchmark while significantly reducing reconstruction time compared to many prior optimization-based methods. It leverages off-the-shelf depth and optical flow estimators for initialization. Uniquely, it renders novel views at different timestamps via image-space warping and aggregates results using dynamic object masks, which produce sharp appearances. However, the method still requires hours to build its representation, and in some challenging cases with complex motion, warping and masking errors can cause temporal flickering artifacts. In our work, we focus on an end-to-end feedforward representation that can deliver similar outputs while being orders of magnitude faster in inference. This provides the potential to run at scale. \n\n&nbsp;\n\n**Handling invisible region** Since DGS-LRM model is supervised with ground-truth scene flow, it has the ability to recover parts of dynamic objects that are partially observation in a subset of frames. For regions that has never been seen,  we require generative prior to hallucinate its appearance, which is beyond the scope of this paper. \n\n&nbsp;\n\n**Scalability** Thanks for the insightful feedback! We would like to re-emphasize DGS-LRM is one of the first attempts tackling this challenging problem and it shows competitive generalization ability as well as tracking and view synthesis accuracy on real-world videos under our current setting. \n- *Scaling to high-resolution*  While our current model achieves competitive view synthesis quality without causing memory issues, further scaling up resolution is possible with better training strategies and novel network architectures. Long-LRM manages to decode 32 images with resolution $960 \\times 540$. Similar strategy can be adopted by DGS-LRM and we leave this for future works. \n- *Non-rigid motion* Although our current simulation pipeline only contains rigid motion, the generated per-pixel deformation fields are actually diverse and comprehensive. Qualitative experiments also demonstrate our model's generalization ability on non-rigid motion, such as in Teddy and Apple examples. It is also possible to utilize our blender pipeline to simulate non-rigid deformable objects, which we leave for future works. \n\n&nbsp;\n\n## Response to questions:\n**Q1. Bluriness**: As shown in the Supplementary, PGDVS has visible flickering deformation artifacts. Our learning based framework may obtain better deformation but fall short in the textural sharpness at this moment. It is worth noting our inference times a few orders of magnitude faster due to the nature of feedforward prediction, which is not reflected in the video comparisons.  \n**Q2. Union instead of sum**: Thanks for detailed suggestion, we will correct it in a revised version. \n\n&nbsp;\n\n\n> [1] Karaev, Nikita, et al. \"Cotracker: It is better to track together.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2024.\n\n> [2] Xiao, Yuxi, et al. \"Spatialtracker: Tracking any 2d pixels in 3d space.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n> [3] Teed, Zachary, and Jia Deng. \"Raft: Recurrent all-pairs field transforms for optical flow.\" European conference on computer vision. Cham: Springer International Publishing, 2020.\n\n> [4] Doersch, Carl, et al. \"Tapir: Tracking any point with per-frame initialization and temporal refinement.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n> [5] Dosovitskiy, Alexey, et al. \"Flownet: Learning optical flow with convolutional networks.\" Proceedings of the IEEE international conference on computer vision. 2015.\n\n> [6] Ilg, Eddy, et al. \"Flownet 2.0: Evolution of optical flow estimation with deep networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n> [7] Charatan, David, et al. \"pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n> [8] Zhang, Kai, et al. \"Gs-lrm: Large reconstruction model for 3d gaussian splatting.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n> [9] Xu, Yinghao, et al. \"Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n> [10] Ziwen, Chen, et al. \"Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats.\" arXiv preprint arXiv:2410.12781 (2024).\n\n> [11] Zhang, Shangzhan, et al. \"Flare: Feed-forward geometry, appearance and camera estimation from uncalibrated sparse views.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "HhSfrnfw2J", "reviewer": "Reviewer_xyfA", "summary": "The paper addresses the task of 4D scene reconstruction from monocular videos.\nTo solve this task, the authors propose a feed-forward transformer-based approach. Compared to existing feed-forward reconstruction transformer, the main novelty lies in the addition of dynamics. \n\nNext to predicting Gaussian primitive parameters for each input pixel, the authors propose to predict motion offsets for each Gaussian to each frame in the sequence. Another novelty is the usage of a temporal tokenizer from MovieGen, which reduces the computational requirements by a factor of 4 due to 4x temporal compression. One downside of this choice, as the authors state, is that the model can only operate on temporally continuous inputs. To handle longer videos the authors additionally introduce a set of reference frames to help large baseline scenarios or strong temporal scene changes.\n\nAnother major part of the paper is data preparation and training.\nFirst of all the authors claim the importance of using a synthetic dataset, in order to provide enough scene and motion diversity. To this end the authors create a new synthetic dataset and state that including real-world data in training did not improve performance on real-world evaluation settings.\n\nQuantitatively the authors evaluate NVS on 7 videos from DyCheck and 3D point tracking on PointOdyssey.\nThe quantitative results are very promising, and the proposed method outperforms most baselines significantly. Mainly an optimization based method (PGDVS) outperforms them on NVS, but takes order of magnitudes more time.\n\nOverall, the paper adds another important addition to realm of novel feed-forward reconstruction transformers by incorporating dynamics. Handling dynamics is very relevant for many application scenarios, as the world that we observe is usually not static.", "questions": "- While the presented method are novel, and the dataset generation and training show great engineering efforts, the evaluation and chosen baselines are not a strong side of the paper. As mentioned in the \"Weaknesses\" section, I believe that additional single-view feed-forward 3DGS baselines, which can handle scenes would be a great addition. Similarly evaluating NVS on another dataset would also be beneficial, since 7 videos is not all that much.\n- Another option to make the evaluation more bullet-proof would be to use the proposed method as initialization for an optimization-based approach.\n- Would it be possible to add 3D point tracking numbers for the baselines to Table 2, which are comparable to the (FC + FV) setting and the (Native) setting?\n- How over-complete are the reconstructed scenes? Since every a gaussain is spawned for every pixel all keyframes, I assume that objects with little movement have many layers of Gaussians.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strenghts**\n- The novelty of the proposed representation for feed-forward transformer, which explicitly allows for disentangled motion prediction, next to 3D scene reconstruction. Explicitly modeling motion (e.g. as opposed to L4GM) makes this approach more usable for down-stream applications, and also allows for additional training supervision, which is ablated to help.\n- The use of temporal tokenization and a set of reference views are both very useful additions, and are also well ablated. Both especially help for videos with more frames, or more (camera) movement. Designing systems that can handle many frames in a video is of special importance.\n- The proposed method shows impressive performance across both NVS and Point tracking tasks on real-world data.\n- If it will be, released the dataset could be very useful for the community. The training strategy is also well ablated.\n\n\n**Weaknesses**\n- The novel-view evaluation is only done on 7 videos from the DyCheck dataset. is not that much. The authors claim that multi-view video datasets with a moving camera are sparse, but what about multi-view video datasets with a fixed camera rig? For example there are multi-view video datasets for\n  - general deformations: see \"Immersive Light Field Video\nwith a Layered Mesh Representation\", \"Dataset and Pipeline for Multi-View Light-Field Video\" or \"Neural 3D Video Synthesis from Multi-view Video\"\n  - human bodies: \"Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis\"\n  - human heads: \"NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads\"\n\n- There is only one feed-forward baselines in the paper (L4GM), which is trained on an extremely different domain. Additional comparisons to single-image 3D reconstruction, e.g. pixelSplat or follow-up single-view 3DGS predictors, could be used frame-by-frame.  Or comparison to scene-scale methods (LVSM, long-lrm, or any other lrm-style model that e.g. evaluates on RealEstate10k) could also be used. If these methods break on dynamic scenes, even a comparison on static scenes such as RealEstate10k would yield some interesting insights.\n- While it is great to see that the model generalizes well to real world data, the need for synthetic data could be seen as a negative. Especially when it comes to the limitations of the training data, such as only rigid deformations and small camera movements. \n\nTypos:\n- line 183 has the \".\" before \"convergence\" instead of afterwards.\n- line 186 eads \"GDS-LRM\" instead of \"DGS-LRM\"", "was_revised": true, "final_justification": "The paper proposed a novel method for an extremely relevant and upcoming task. \nA novel and simple dynamic scene representation, temporal tokenization, successful synthetic-to-real transfer and impressive results on real world data, are significantly raising the paper above the acceptance threshold.", "rebuttal": "We thank the reviewer for their insightful feedback and recognition of our work. We appreciate their noting that \"the proposed method across both NVS and Point tracking tasks on real-world data\", our temporal tokenization and reference frames \"are both very useful additions, and are also well ablated\", and our explicit motion modeling \"makes this approach more usable for down-stream applications\". \n\n&nbsp;\n\n**Multi-view video datasets** DGS-LRM is designed as a feed-forward deformable Gaussian reconstruction method for monocular videos of dynamic scenes. As noted in the limitations section, our model, trained on continuous video with temporal tokenization, has difficulty generalizing to discrete image frames that simulate unnatural camera movements. As highlighted in DyCheck [1], teleporting among cameras using multi-view dynamic video datasets reaches unnatural speed far from real-world distribution. We leave developing a framework that can handle both continuous and discrete camera poses as future work. We also hope to highlight that DyCheck is the most widely used dataset for recent dynamic novel view synthesis from a monocular video. \n\n&nbsp;\n\n**Comparisons with static LRM methods** Thank you for the constructive suggestion! Most state-of-the-art static feed-forward Gaussian reconstruction methods, such as GS-LRM [2], are not open-sourced. Therefore, we re-implemented GS-LRM to train a static baseline on the RealEstate10K dataset. This baseline achieved a **PSNR** of **28.77** on the RealEstate10K testing set, slightly higher than the **PSNR** of **28.10** reported in the original paper. The novel view synthesis quality on the Dycheck dataset, summarized in the table below, is significantly worse compared to DGS-LRM. We agree that this experiment underscores the value of explicitly modeling a deformation field. We will include this experiment in a revised version of the paper.\n|  Methods | PSNR | LPIPS |\n| ------------ | -------------- | -------------- |\n| static GS-LRM | 13.02 | 0.444 |\n| DGS-LRM | 14.89 | 0.420 |\n\n&nbsp;\n\n**Synthetic data for training** We find synthetic data crucial for the following 2 main reasons. \n- Dual view supervision enhances novel view synthesis quality, especially with larger baselines.\n- Direct supervision of the deformation field is necessary for accurate 3D tracking. Without it, rendering loss alone is insufficient for precise scene flow reconstruction.\n\nOur method is among the first to address this challenging problem. Future work could involve more effective use of real data by distilling knowledge from foundational models, such as monocular depth and flow prediction.\n\n&nbsp;\n\n**Over-complete scene representation** Thank you for the insightful feedback! We acknowledge that the per-pixel Gaussian is an over-complete scene representation for the foreground. In our current setting, we have not observed a significant impact on training and rendering speed. However, we anticipate that this issue may become more noticeable with longer context windows. Recent work [3] addresses this problem by utilizing opacity values to filter Gaussian points. We leave exploring this approach as future work. \n\n&nbsp;\n\n**Typos** Thank you for pointing out! We will correct them in a revised version. \n\n**Table 2** We apologize for any confusion. Table 2 already includes the tracking numbers for the baselines. \n\n&nbsp;\n\n> [1] Hang, Gao, et al. \"Monocular Dynamic View Synthesis: A Reality Check\", NeurIPS 2022.\n\n> [2] Zhang Kai, et at. \"GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting\", ECCV 2024. \n\n> [3] Ziwen, Chen, et al. \"Long-lrm: Long-sequence large reconstruction model for wide-coverage gaussian splats.\" arXiv preprint arXiv:2410.12781 (2024)."}, {"review_id": "KNhaeg9nCA", "reviewer": "Reviewer_qSmi", "summary": "The paper proposes DGS-LRM a large generalizable reconstruction model that given a posed monocular video predicts a 4D reconstruction in form of deformable 3D Gaussians in a single feed-forward pass. After applying both spatial and temporal tokenization on the input Video and concatenation of Plücker ray and timestep encodings, a Standard transformer Architecture followed by two-layer MLP predicts the parameters of the deformable 3D Gaussians. These are parameterized using pixel-wise depth and remaining Gaussian attributes (RGB, rotation, scale, opacity) predictions for a single Gaussian along each ray of (temporally downsampled) input and additional reference frames. Additionally, the network predicts pixel-aligned 3D deformation (translation) vectors for all timestamps that can be used to warp the Gaussians predicted for one frame to the timestamps of all other frames. The 3D representation at a certain timestamp is therefore the union of all Gaussians warped to this particular time.\nFor supervision the authors employ standard reconstruction losses of rendered images (MSE, LPIPS) as well as L1-losses to directly supervise pixel-wise depth and deformation predictions with ground truth depth and Scene flow.\nSince these supervision Signals are usually not available for real-world datasets, the authors create a synthetic dataset generated with the Kubric engine. Despite Training only on this synthetic data, the resulting model generalizes well to real-world Videos. Since the model is only trained for short Clips, the paper further proposes to chain the Scene flow for longer-horizon 3D Point tracking.\nIn an Evaluation on real-world datasets DyCheck and DAVIS, the proposed method outperforms a feed-forward baselines L4GM and falls slightly behind the state-of-the-art 4D reconstruction Approach while enabling real-time inference. The authors further Show that they achieve similar Performance as the state-of-the-art in feed-forward 3D Point tracking. The paper further contains an Ablation studies w.r.t. design choices and loss functions.", "questions": "Please see the weaknesses for detailed Questions.\nMy main concern is the lack of clarity regarding the technical contributions as well as additional ablation studies and possible comparison with BTimer.\nA convincing rebuttal w.r.t. these points is necessary for me to increase my evaluation score.\n\nAdditional Questions:\n- Is temporal downsampling a Problem for the flow chaining?\n- Why do Rotation & opacity deformations not help? Do you have any Intuition?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The method is simple, intuitive, and elegant:\n  - The explicitly predicted deformation vectors allow for straightforward 3D Point tracking.\n  - The model is trained on synthetic data only:\n    - This allows for the use of additional supervision Signals like Ground truth depth and Scene flow.\n    - Still, the trained model generalizes well to real-world Videos, which is a strong result.\n- The Evaluation Shows strong quantitative results compared to feed-forward baselines:\n  - It achieves competitive Performance in 3D Point tracking compared to the state of the art.\n  - It outperforms L4GM significantly in 4D reconstruction.\n- The Ablation studies validate the effectiveness of the design choices (dual-view sampling, Scene flow loss, reference frames).\n- The paper is mostly well-written and easy to follow.\n- The supplementary material contains qualitative Video results for 4D reconstruction and 3D Point tracking.\n\nWeaknesses:\n- Main lack of clarity: The paper does not explain its technical contributions well.\n  - The dual-view sampling is not easy to understand. I understood it as simply supervising with Always 2 target views during Training but in line 258, the authors Mention that they use 8 Output views per Scene during Training.\n  - The view selection (lines 177 ff.) is also not completely clear from the text. A visualization similar to Fig.3 could be beneficial.\n  - The role of the reference frames is unclear. They are introduced in the context of reducing the GPU VRAM utilization, but it remains unclear how they contribute to that, as more views / frames should mean even larger Memory utilization in the self-attention between all patches.\n  - The flow chaining procedure (section 3.4) is not easy to follow. A visualization could be helpful.\n- Further lack of clarity:\n  - If I understood lines 145 ff. correctly, the last frame within a temporal downsampling chunk is used for pixel-aligned Gaussians. Is that a Problem for flow chaining as the last (temporally downsampled) frame of the 1st Video and the first frame of the 2nd Video are not the same?\n  - In equation 2, should these be unions instead of sums?\n  - Why do Rotation & opacity deformations not help? Do you have any Intuition?\n- The Claim of comparable Performance with optimization-based Methods is questionable. The 4D reconstruction Performance is significantly worse than of PGDVS (Pseudo-Generalized Dynamic View Synthesis from a Video. ICLR 2024).\n- The Claim that DGS-LRM is the first feed-forward method predicting deformable 3D Gaussian splats from monocular posed Videos is questionable:\n  - Why is L4GM (Large 4D Gaussian Reconstruction Model. NeurIPS 2024) not considered for this Claim?\n  - BTimer (Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos) appeared on arxiv in December 2024. Therefore, I would not consider it to be concurrent work anymore.\n    - Are quantitative comparisons possible?\n- The Ablation study is missing the effect of:\n  - depth supervision and\n  - Scene normalization (lines 200ff.) adopted from MegaSaM.\n    - This one is especially important since the main baseline L4GM struggles to resolve the correct Scene scale, as pointed out by the authors themselves in lines 284f.\n- Fig. 5 seems to Show input view reconstructions or at least novel views extremely close to the input views. Qualitative results for larger camera motions would be more interesting.\n- It is not trivial to see which Point tracks are better in Fig. 6. Should it not be possible to Show Ground truth tracks for comparison?", "was_revised": true, "final_justification": "Overall, I have been positive regarding this paper already prior to the rebuttal.\nThe main strengths are the novelty of the method, generalization to real data while trained on synthetic data only, and the strong experimental results w.r.t. both novel view synthesis and point tracking for a generalizable feed-forward and therefore fast method.\nMy main concern prior to rebuttal was lack of clarity that the authors largely resolved during the rebuttal. If the authors include the additional explanations into the revised paper version, I do not have any major concerns any w.r.t. acceptance.\nI also appreciate the additional results provided and pilot ablation studies conducted for the rebuttal. \n\nAs a result, I decided to raise my score to 5: Accept.", "rebuttal": "We thank the reviewer for their insightful feedback and recognition of our work. We appreciate their noting that \"the method is simple, intuitive, and elegant\", \"the evaluation show strong quantitative results\" and \"the ablation studies validate the effectiveness of design choices\". \n\n&nbsp;\n\n**Issues of clarity** We deeply appreciate your constructive and detailed feedback. We will fix these issues in a revised version of the paper.\n- *The dual view sampling*: As mentioned from line 182 to 185, we sample $Q=8$ output images at $\\frac{Q}{2} = 4$ timesteps for ground-truth supervision.  Specifically, at each timestep, we sample 2 images from 2 camera trajectories. This dual view supervision is essentially a multi-view (two-view) supervision that enhances both geometry and novel view synthesis quality. In the revised version of the paper, we will strictly differentiate between the terms \"timestep\" and \"frame.\"\n- *View selection*: For videos of static scenes, frames sampled at different timesteps provide sufficient multi-view supervision for novel view synthesis. However, for videos of dynamic scenes, simply using frames sampled at different timesteps can cause ambiguity, as both cameras and objects are moving. Therefore, we employ dual view supervision to sample 2 camera poses at the same timestep for rendering supervision.\n- *Role of reference views*: The role of reference views is to increase the input baseline, which is important for both geometry and view synthesis quality. We may achieve sufficiently large baselines if the camera moves enough across a long temporal span. However, naively increasing the input video length would add too many input frames, leading to prohibitively large GPU memory consumption. Therefore, we sample $K=4$ extra temporally distant reference frames to help increase the baseline. A similar strategy was used in BTimer. We will further acknowledge this prior work in the revised version of the paper.\n- *Flow chaining procedure* We agree that a visualization would be helpful, but we did not include it due to the page limit. We will add a better visualization in the revised version of the supplementary material. Our predicted per-Gaussian deformation field can deform Gaussians to every input timestep, not just the temporally downsampled key timesteps, and therefore will not cause issues in flow chaining.\n- *Union instead of sum*: Thanks for the detailed comment! We will correct this in a revised version of the paper.\n- *Opacity and rotation deformation*: For *rotation*, the per-pixel deformable 3D Gaussians are densely distributed in the foreground area. Each Gaussian typically has a small covariance. Therefore, we empirically did not observe any impact from fixed rotation. For *opacity*, our pilot study indicates that allowing changes in opacity enables the model to \"fake\" dynamic appearances without accurately predicting the deformation field, which negatively impacts tracking accuracy. We will add this analysis to the revised version of our paper. \n\n&nbsp;\n\n**Comparisons with PGDVS** While DGS-LRM performs better in some examples with complex motion, such as the pinwheel in Figure 4, we agree that PGDVS generally recovers more detailed textures, as indicated by the quantitative data. In the revised version of the paper, we will tone down the use of the word \"comparable\" and be more specific about the properties of each method. We would also like to emphasize that DGS-LRM is three orders of magnitude faster at inference compared to PGDVS, which requires several hours for inference.\n\n&nbsp;\n\n**First feed-forward method** We emphasize that DGS-LRM is the first feed-forward method that predicts *deformable* 3D Gaussians, supporting both novel view synthesis and spatial tracking simultaneously. Neither BTimer nor L4GM predicts deformation fields. Regarding BTimer, we fully agree that it would be an informative comparison, but their code and trained model are not currently available. Based on NeurIPS policies, we will remove the word \"concurrent\" when describing BTimer.\n\n&nbsp;\n\n**Ablation studies**  \n- For *depth supervision*, our pilot study shows that our dual view supervision already provides sufficient geometry constrain, and adding depth supervision does not significantly impact the reconstruction quality. We simply keep it as we have ground-truth depth data. \n- For *scene normalization*, both methods normalize the scene using the structure-from-motion depth for a fair comparison. Such a depth is exclusively used in scene normalization to align the scene scale to the benchmark unit. We will include this implementation detail in the revised paper.\n\n&nbsp;\n\n**Qualitative figures** While the baselines in Figure 5 are relatively small, we did present view synthesis results with larger baselines in Figure 4 and the supplementary material, where DGS-LRM demonstrates competitive reconstruction quality. For Figure 6, we agree that adding ground truth flow would be useful, and we will include it in the revised version of the paper. Thank you again for the constructive feedback!"}]

Author rebuttal:

Meta-review: The paper proposes DGS-LRM, a novel feed-forward approach for real-time dynamic scene reconstruction from a monocular posed video of any dynamic scene.
It received two accept and two accept and two borderline accept.
Most of the concerns have been solved during the rebuttal.
The AC recommends acceptance for this paper.
Decision: Accept (poster)