Title: DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large reconstruction model

LRM

dynamic
Abstract: We introduce the Deformable Gaussian Splats Large Reconstruction Model (DGS-LRM), the first feed-forward method predicting deformable 3D Gaussian splats from a monocular posed video of any dynamic scene. Feed-forward scene reconstruction has gained significant attention for its ability to rapidly create digital replicas of real-world environments. However, most existing models are limited to static scenes and fail to reconstruct the motion of moving objects. Developing a feed-forward model for dynamic scene reconstruction poses significant challenges, including the scarcity of training data and the need for appropriate 3D representations and training paradigms. To address these challenges, we introduce several key technical contributions: an enhanced large-scale synthetic dataset with ground-truth multi-view videos and dense 3D scene flow supervision; a per-pixel deformable 3D Gaussian representation that is easy to learn, supports high-quality dynamic view synthesis, and enables long-range 3D tracking; and a large transformer network that achieves real-time, generalizable dynamic scene reconstruction. Extensive qualitative and quantitative experiments demonstrate that DGS-LRM achieves dynamic scene reconstruction quality comparable to optimization-based methods, while significantly outperforming the state-of-the-art predictive dynamic reconstruction method on real-world examples. Its predicted physically grounded 3D deformation is accurate and can be readily adapted for long-range 3D tracking tasks, achieving performance on par with state-of-the-art monocular video 3D tracking methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DGS-LRM a large generalizable reconstruction model that given a posed monocular video predicts a 4D reconstruction in form of deformable 3D Gaussians in a single feed-forward pass. After applying both spatial and temporal tokenization on the input Video and concatenation of Plücker ray and timestep encodings, a Standard transformer Architecture followed by two-layer MLP predicts the parameters of the deformable 3D Gaussians. These are parameterized using pixel-wise depth and remaining Gaussian attributes (RGB, rotation, scale, opacity) predictions for a single Gaussian along each ray of (temporally downsampled) input and additional reference frames. Additionally, the network predicts pixel-aligned 3D deformation (translation) vectors for all timestamps that can be used to warp the Gaussians predicted for one frame to the timestamps of all other frames. The 3D representation at a certain timestamp is therefore the union of all Gaussians warped to this particular time.\nFor supervision the authors employ standard reconstruction losses of rendered images (MSE, LPIPS) as well as L1-losses to directly supervise pixel-wise depth and deformation predictions with ground truth depth and Scene flow.\nSince these supervision Signals are usually not available for real-world datasets, the authors create a synthetic dataset generated with the Kubric engine. Despite Training only on this synthetic data, the resulting model generalizes well to real-world Videos. Since the model is only trained for short Clips, the paper further proposes to chain the Scene flow for longer-horizon 3D Point tracking.\nIn an Evaluation on real-world datasets DyCheck and DAVIS, the proposed method outperforms a feed-forward baselines L4GM and falls slightly behind the state-of-the-art 4D reconstruction Approach while enabling real-time inference. The authors further Show that they achieve similar Performance as the state-of-the-art in feed-forward 3D Point tracking. The paper further contains an Ablation studies w.r.t. design choices and loss functions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The method is simple, intuitive, and elegant:\n  - The explicitly predicted deformation vectors allow for straightforward 3D Point tracking.\n  - The model is trained on synthetic data only:\n    - This allows for the use of additional supervision Signals like Ground truth depth and Scene flow.\n    - Still, the trained model generalizes well to real-world Videos, which is a strong result.\n- The Evaluation Shows strong quantitative results compared to feed-forward baselines:\n  - It achieves competitive Performance in 3D Point tracking compared to the state of the art.\n  - It outperforms L4GM significantly in 4D reconstruction.\n- The Ablation studies validate the effectiveness of the design choices (dual-view sampling, Scene flow loss, reference frames).\n- The paper is mostly well-written and easy to follow.\n- The supplementary material contains qualitative Video results for 4D reconstruction and 3D Point tracking.\n\nWeaknesses:\n- Main lack of clarity: The paper does not explain its technical contributions well.\n  - The dual-view sampling is not easy to understand. I understood it as simply supervising with Always 2 target views during Training but in line 258, the authors Mention that they use 8 Output views per Scene during Training.\n  - The view selection (lines 177 ff.) is also not completely clear from the text. A visualization similar to Fig.3 could be beneficial.\n  - The role of the reference frames is unclear. They are introduced in the context of reducing the GPU VRAM utilization, but it remains unclear how they contribute to that, as more views / frames should mean even larger Memory utilization in the self-attention between all patches.\n  - The flow chaining procedure (section 3.4) is not easy to follow. A visualization could be helpful.\n- Further lack of clarity:\n  - If I understood lines 145 ff. correctly, the last frame within a temporal downsampling chunk is used for pixel-aligned Gaussians. Is that a Problem for flow chaining as the last (temporally downsampled) frame of the 1st Video and the first frame of the 2nd Video are not the same?\n  - In equation 2, should these be unions instead of sums?\n  - Why do Rotation & opacity deformations not help? Do you have any Intuition?\n- The Claim of comparable Performance with optimization-based Methods is questionable. The 4D reconstruction Performance is significantly worse than of PGDVS (Pseudo-Generalized Dynamic View Synthesis from a Video. ICLR 2024).\n- The Claim that DGS-LRM is the first feed-forward method predicting deformable 3D Gaussian splats from monocular posed Videos is questionable:\n  - Why is L4GM (Large 4D Gaussian Reconstruction Model. NeurIPS 2024) not considered for this Claim?\n  - BTimer (Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos) appeared on arxiv in December 2024. Therefore, I would not consider it to be concurrent work anymore.\n    - Are quantitative comparisons possible?\n- The Ablation study is missing the effect of:\n  - depth supervision and\n  - Scene normalization (lines 200ff.) adopted from MegaSaM.\n    - This one is especially important since the main baseline L4GM struggles to resolve the correct Scene scale, as pointed out by the authors themselves in lines 284f.\n- Fig. 5 seems to Show input view reconstructions or at least novel views extremely close to the input views. Qualitative results for larger camera motions would be more interesting.\n- It is not trivial to see which Point tracks are better in Fig. 6. Should it not be possible to Show Ground truth tracks for comparison?\n\nQUESTIONS:\nPlease see the weaknesses for detailed Questions.\nMy main concern is the lack of clarity regarding the technical contributions as well as additional ablation studies and possible comparison with BTimer.\nA convincing rebuttal w.r.t. these points is necessary for me to increase my evaluation score.\n\nAdditional Questions:\n- Is temporal downsampling a Problem for the flow chaining?\n- Why do Rotation & opacity deformations not help? Do you have any Intuition?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the task of 4D scene reconstruction from monocular videos.\nTo solve this task, the authors propose a feed-forward transformer-based approach. Compared to existing feed-forward reconstruction transformer, the main novelty lies in the addition of dynamics. \n\nNext to predicting Gaussian primitive parameters for each input pixel, the authors propose to predict motion offsets for each Gaussian to each frame in the sequence. Another novelty is the usage of a temporal tokenizer from MovieGen, which reduces the computational requirements by a factor of 4 due to 4x temporal compression. One downside of this choice, as the authors state, is that the model can only operate on temporally continuous inputs. To handle longer videos the authors additionally introduce a set of reference frames to help large baseline scenarios or strong temporal scene changes.\n\nAnother major part of the paper is data preparation and training.\nFirst of all the authors claim the importance of using a synthetic dataset, in order to provide enough scene and motion diversity. To this end the authors create a new synthetic dataset and state that including real-world data in training did not improve performance on real-world evaluation settings.\n\nQuantitatively the authors evaluate NVS on 7 videos from DyCheck and 3D point tracking on PointOdyssey.\nThe quantitative results are very promising, and the proposed method outperforms most baselines significantly. Mainly an optimization based method (PGDVS) outperforms them on NVS, but takes order of magnitudes more time.\n\nOverall, the paper adds another important addition to realm of novel feed-forward reconstruction transformers by incorporating dynamics. Handling dynamics is very relevant for many application scenarios, as the world that we observe is usually not static.\n\nSTRENGTHS AND WEAKNESSES:\n**Strenghts**\n- The novelty of the proposed representation for feed-forward transformer, which explicitly allows for disentangled motion prediction, next to 3D scene reconstruction. Explicitly modeling motion (e.g. as opposed to L4GM) makes this approach more usable for down-stream applications, and also allows for additional training supervision, which is ablated to help.\n- The use of temporal tokenization and a set of reference views are both very useful additions, and are also well ablated. Both especially help for videos with more frames, or more (camera) movement. Designing systems that can handle many frames in a video is of special importance.\n- The proposed method shows impressive performance across both NVS and Point tracking tasks on real-world data.\n- If it will be, released the dataset could be very useful for the community. The training strategy is also well ablated.\n\n\n**Weaknesses**\n- The novel-view evaluation is only done on 7 videos from the DyCheck dataset. is not that much. The authors claim that multi-view video datasets with a moving camera are sparse, but what about multi-view video datasets with a fixed camera rig? For example there are multi-view video datasets for\n  - general deformations: see \"Immersive Light Field Video\nwith a Layered Mesh Representation\", \"Dataset and Pipeline for Multi-View Light-Field Video\" or \"Neural 3D Video Synthesis from Multi-view Video\"\n  - human bodies: \"Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis\"\n  - human heads: \"NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads\"\n\n- There is only one feed-forward baselines in the paper (L4GM), which is trained on an extremely different domain. Additional comparisons to single-image 3D reconstruction, e.g. pixelSplat or follow-up single-view 3DGS predictors, could be used frame-by-frame.  Or comparison to scene-scale methods (LVSM, long-lrm, or any other lrm-style model that e.g. evaluates on RealEstate10k) could also be used. If these methods break on dynamic scenes, even a comparison on static scenes such as RealEstate10k would yield some interesting insights.\n- While it is great to see that the model generalizes well to real world data, the need for synthetic data could be seen as a negative. Especially when it comes to the limitations of the training data, such as only rigid deformations and small camera movements. \n\nTypos:\n- line 183 has the \".\" before \"convergence\" instead of afterwards.\n- line 186 eads \"GDS-LRM\" instead of \"DGS-LRM\"\n\nQUESTIONS:\n- While the presented method are novel, and the dataset generation and training show great engineering efforts, the evaluation and chosen baselines are not a strong side of the paper. As mentioned in the \"Weaknesses\" section, I believe that additional single-view feed-forward 3DGS baselines, which can handle scenes would be a great addition. Similarly evaluating NVS on another dataset would also be beneficial, since 7 videos is not all that much.\n- Another option to make the evaluation more bullet-proof would be to use the proposed method as initialization for an optimization-based approach.\n- Would it be possible to add 3D point tracking numbers for the baselines to Table 2, which are comparable to the (FC + FV) setting and the (Native) setting?\n- How over-complete are the reconstructed scenes? Since every a gaussain is spawned for every pixel all keyframes, I assume that objects with little movement have many layers of Gaussians.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents DGS-LRM, a novel feed-forward method for real-time dynamic scene reconstruction from posed monocular videos using deformable 3D Gaussian representations.\nIt augment GS-LRM to handle dynamic reconstruction by predicting on keyframes pixel-aligned gsplat, where each gsplat contains additional translation vectors that warps the gsplat to any other frames. To decrease enormous memory demand, the image patches in GS-LRM is replaced with spatial-temporal volumetric patches. \nThe model is learned with strong supervision and ground truth 3D scene flow.\nThe authors created a new dataset with synchronized camera and scene flow from Kubric.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n- The paper tackles a very timely topic and a very challenging task. \n- It provided a new dataset and it looks like the trained model generalizes reasonably well to the test data. \n- Strong performance in 3D tracking.\n\n# Weakness\n- Overall, I find the methodology section quite hard parse. \n- The proposed model appears to be the very definition of a \"brute-force\" attempt. Specifically,\n  - all scene flow is directly supervised. \n  - Eq 2 suggests that the prediction at the target view fuses all input views pixel-aligned GS to the target view through warping. It means the target view will be rendered with a significantly amount of redundant GS. How does the model scale with high resolution? How many GS are we talking about? Will there be memory issues? \n  - Dynamic sparsity is addressed in terms of dataloading, but not in the rendering / representation.\n- In the result, the contending model PGDVS seems to work really well, in fact better than the proposed model. Some explanation was mentioned in the README.pdf about how the evaluation is disadvantageous for the proposed model. Yet, the wording is confusing to me.\n- With PGDVS being a strong contender in the comparison, I expect to see more information about it in the related work section. At least, there should be a baseline paragraph, introducing the comparing methods.\n\nQUESTIONS:\n- Why is the DyCheck reconstruction so much blurrier than PGDVS? \n- Minor: the use of \\sum in equation 2 should be replaced by a union?\n\nLIMITATIONS:\n- Handling unseen regions ungracefully. The representation itself is based on strict warping from input views, causing the disoccluded parts to be just black. I don't entirely agree with the authors that \"the invisible area is expected to remain unreconstructed\". One benefit of ML is to use data priors to fill the holes. \n- Scalability. It's hard to imagine the system could scale up to higher resolution and more dynamic scenes. The data creation dictates that the motion mode only covers rigid motion. The all-to-one warping means huge amount of redundant gaussians will be rendering.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the Deformable Gaussian Splats Large Reconstruction Model, the first forward model to predict dynamic Gaussians for handling dynamic scenes. Specifically, the proposed model takes a video sequence and corresponding camera poses as input and directly outputs deformable dynamic Gaussians, enabling novel view synthesis of dynamic videos. By adopting a feed-forward approach instead of optimization-based methods, the model significantly reduces inference time while maintaining competitive quality. The main contributions are:\n\n1、A deformable 3D Gaussian representation combined with a keyframe-based rendering strategy for efficient Gaussian generation and rendering.\n\n2、A large-scale synthetic dataset for training and evaluation.\n\n3、A novel Transformer-based architecture that reduces memory consumption.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe design of the deformable 3D Gaussian representation combined with the keyframe-based rendering strategy is highly novel.\n\nThe feed-forward approach addresses the limitations of optimization-based methods, significantly improving speed while maintaining good quality.\n\nA rich dataset has been collected, which—if open-sourced—could greatly benefit future research in this area.\n\nWeaknesses:\n\nThe training and inference processes are unacceptable for most researchers, making the model difficult for others to reproduce or deploy.\n\nThe method struggles with large object motions within videos, leading to my worries about significant domain gaps; more experiments are needed to clarify this limitation. Please see the questions.\n\nQUESTIONS:\n1.\tWhy do deformations such as rotation or opacity in Deformable Gaussian Splats have little effect on quality improvement?\n\n2.\tWould incorporating Frame Attention inspired by VGGT further enhance performance?\n\n3.\tIn the keyframe-based rendering strategy, is there a noticeable quality gap between keyframes and non-keyframes?\n\n4.\tThe demo does not include long videos or scenes with large viewpoint changes. Will the model’s performance degrade under these conditions?\n\n5.\tPlease provide ablation studies on the number of parameters, the consumption of GPU memory and quality.\n\nIf the above questions are addressed effectively, or if there is a clear open-source plan, I will consider raising my score accordingly.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}