Title: GeoVideo: Introducing Geometric Regularization into Video Generation Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Geometric Regularization

3D/4D Generation
Abstract: Recent advances in video generation have enabled the synthesis of high-quality and visually realistic clips using diffusion transformer models. However, most existing approaches operate purely in the 2D pixel space and lack explicit mechanisms for modeling 3D structures, often resulting in temporally inconsistent geometries, implausible motions, and structural artifacts. In this work, we introduce geometric regularization losses into video generation by augmenting latent diffusion models with per-frame depth prediction. We adopted depth as the geometric representation because of the great progress in depth prediction and its compatibility with image-based latent encoders. Specifically, to enforce structural consistency over time, we propose a multi-view geometric loss that aligns the predicted depth maps across frames within a shared 3D coordinate system. Our method bridges the gap between appearance generation and 3D structure modeling, leading to improved spatio-temporal coherence, shape consistency, and physical plausibility. Experiments across multiple datasets show that our approach produces significantly more stable and geometrically consistent results than existing baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthis paper proposes a method to predict depth video, camera parameters, and motion probability along with the rgb videos. by using the additional information, an additional loss term that measures the 3d consistency is defined and applied, thus improving the 3d consistency of the generated video.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths:\n1. the idea is quite simple and straightforward, and the paper is easy to read.\n\nweaknesses:\n1. the experiments are less convincing. for example, the authors collected a dataset for training and evaluation for dynamic video generation. as the data has a critical influence on the experimental results, i am curious about the fairness of the experimental results as shown in Table 3.\n2. forcing such an loss may introduce some negative influences on the generation performance, but there is no such analyses.\n\nas i have major concerns on the experiments, i would suggest borderline reject for now.\n\nQUESTIONS:\n1. [critical] please provide more details on the 1000 test videos in Table 3, which i think is very important but is missing.\n2. [critical] please include more qualitative results to show the effectiveness of proposed method, with various inputs. the cases shown in the paper (as in Figure 3,4,5) are over simple for a practical video generation applications.\n3. [less critical] why does not perform a complete vbench evaluation benchmark? i am curious about the performance of other metrics.\n4. [less critical] i notice that in Table 1 and 3, the performance of \"ours w/o L_geo\" is already better than the baseline. is there any case where \"ours w/o L_geo\" performs better than \"ours\"? is there any case where the baseline performs better than \"ours w/o L_geo\"?\n\nLIMITATIONS:\n[critical] the Limitation section is quite brief. please include the negative influence of apply such a artificial loss term.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces geometric regularization into latent video diffusion models to improve spatiotemporal consistency. \nBuilding on a standard DiT backbone, the authors add parallel depth and camera-pose prediction branches so that each generated RGB frame is accompanied by a depth map and estimated camera pose. \nDuring training, they first jointly learn RGB and depth generation, then in a second stage enforce a geometric consistency loss: depth predictions from different frames are back-projected into a shared 3D point cloud and re-projected to each view, penalizing depth misalignment across frames. \nThis two-stage fine-tuning yields videos whose frames not only look realistic but also adhere to a coherent underlying 3D structure. Extensive experiments on both static (DL3DV) and dynamic video datasets show marked improvements in multi-view depth consistency, reprojection error, downstream SfM reconstruction success, and perceptual/video-quality metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.By generating depth and poses alongside RGB and enforcing cross-frame depth alignment, GeoVideo addresses temporal shape flicker and structural drift common in 2D-only methods.\n\n2. The manuscript reports improvements on multi-view consistency scores, re-projection error, 3D reconstruction success, and standard perceptual/video metrics, demonstrating both geometric and visual gains.\n\n3. Separating joint RGB-depth learning from geometric regularization stabilizes training and prevents early geometric constraints from harming visual quality.\n\n\nWeaknesses:\n1. The idea of simultaneously generating RGB frames and geometric cues like depth is not entirely new—earlier work such as [1] has already explored joint synthesis of appearance and geometry (e.g., point maps).\n\n2. The evaluation omits several state-of-the-art text-to-3D and video generation methods (e.g., AC3D, Viewcrafter), so it is unclear how GeoVideo compares against the full spectrum of recent 3D-aware approaches.\n\n3. Relying solely on VBench metrics assesses perceptual and temporal coherence but does not directly evaluate 3D world fidelity; incorporating WorldScore [2]  would provide a more complete picture of GeoVideo’s reconstruction capabilities.\n\n\n[1] Zhang Q, Zhai S, Martin M A B, et al. World-consistent video diffusion with explicit 3d modeling[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 21685-21695.\n[2] Duan H, Yu H X, Chen S, et al. Worldscore: A unified evaluation benchmark for world generation[J]. arXiv preprint arXiv:2504.00983, 2025.\n\nQUESTIONS:\nplease refer to weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GeoVideo, a geometry-regularized latent video generation framework that integrates explicit geometric consistency constraints into diffusion-based video synthesis. The key idea is to predict depth and camera pose jointly with RGB frames and enforce multi-frame consistency via a novel cross-frame reprojection loss computed from 3D point clouds. The proposed method is lightweight and modular, making it easy to plug into existing latent diffusion architectures such as CogVideoX and DiT.\n\nSTRENGTHS AND WEAKNESSES:\n--- Strengths\n\n- Well-motivated integration of geometry: Instead of pursuing full 3D scene modeling (e.g., NeRF-like pipelines), the paper proposes a lightweight, practical mechanism to improve structural consistency in video generation using explicit geometric priors.\n\n- Plug-and-play design: The method requires minimal modification to existing latent diffusion models and can be seamlessly integrated into popular T2V frameworks.\n\n- The cross-frame reprojection loss with motion-aware masking is effective in promoting geometric coherence without requiring ground-truth depth or camera pose.\n\n- The paper reports better results on both traditional perceptual metrics (FID, CLIPScore, NIQE) and structural metrics (MVCS, Reprojection Error). The video visualization in the supplemental materials also demonstrate the effectiveness of the proposed method.\n\n--- Weaknesses\n\n- The method relies on predicted depth and camera pose from the same model, which may introduce circular supervision bias. Errors in pose estimation can propagate into the reprojection loss.\n\n- Limited comparison to other recent geometry-aware diffusion models, such as WVD or UniGeo, especially in ablations or direct quantitative analysis.\n\n- No analysis on failure cases or limits of the geometric consistency under highly dynamic or low-texture regions.\n\n--- Reasons of Rating\n\n- Overall, GeoVideo offers a practical, effective, and relatively novel solution to enhancing structural consistency in video diffusion models. While it does not radically redefine the field, its design simplicity, compatibility, and demonstrated gains make it a meaningful contribution. Strengthening comparative analysis and discussing failure cases would significantly increase its impact. Hence, I'm positive for this work.\n\nQUESTIONS:\n1. How robust is the geometric supervision when the predicted camera poses are inaccurate? Have you evaluated sensitivity to pose errors?\n\n2. Can the model handle complex dynamic scenes (e.g., fast motion, occlusions)? How does the motion probability map behave in such cases?\n\n3. Have you considered using external pose/depth priors (e.g., from COLMAP or pretrained estimators) to improve supervision or initialization?\n\n4. How does the proposed method compare quantitatively against WVD or UniGeo, which also integrate geometry into diffusion-based pipelines?\n\nLIMITATIONS:\n- The geometric regularization is limited to self-predicted geometry, which may not be fully reliable under complex motion, low lighting, or ambiguous structures.\n\n- The method assumes rigid camera motion and does not account for object-level 3D dynamics, which limits applicability to highly dynamic scenes.\n\n- The geometric consistency loss is currently applied only as an auxiliary signal and not deeply coupled into the generation process (e.g., via 3D-aware attention mechanisms or geometry-conditioned denoising).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to predict a high-quality 3D reconstruction by introducing geometric regularization into video diffusion models. Specifically, the authors augment VDM with depth prediction, which serves as an implicit form of geometric supervision. In addition, the paper proposes an explicit geometric regularization term that enhances the geometric consistency of the underlying shapes. The proposed approach is validated through experiments on both text-to-video (T2V) and image-to-video (I2V) tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The proposed geometric regularization provides explicit supervision for video diffusion models, significantly improving both 3D consistency and temporal coherence in generated videos.\n\n2. The paper also introduces an effective model initialization strategy and a multi-stage training framework, which contribute to stable optimization and improved performance.\n\n**Weaknesses**\n\n[MAJOR]:\n\n1. Diffusion models typically require multiple denoising steps to produce clean outputs. During training with geometric regularization, does the model compute the regularization from one-step denoising output or many-step denoising output? What is the rationale behind this design choice?\n\n2. While the method primarily emphasizes 3D consistency in generated videos, a more comprehensive comparison with state-of-the-art approaches like ViewCrafter[1] and DimensionX[2]—specifically in terms of 3D consistency—would further validate its advantages.\n\n[Minor]\n\n3. Is there any quantitative evaluation of the accuracy or performance consistency across different prediction heads? Such an analysis would better demonstrate whether the model effectively leverages geometric regularization.\n\n4. There are several writing errors. For instance, in line 209, “estimate camera poses using MegaSaM” appears to be incorrect.\n\n[1] Yu W, Xing J, Yuan L, et al. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis[J]. arXiv preprint arXiv:2409.02048, 2024.\n[2] Sun W, Chen S, Liu F, et al. Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion[J]. arXiv preprint arXiv:2411.04928, 2024.\n\nQUESTIONS:\nThe main issues lie in unclear training details (denoising steps) and insufficient experimental comparisons (3D consistency); for details, see the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}