Title: Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach
Venue: NeurIPS 2025
Accepted: yes
Keywords: video generation
Abstract: We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling it to track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video to eliminate undesired artifacts, e.g., non-physical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent in current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos, where 3D information is essential for perceiving shape and motion of interacting solids. Our method can be seamlessly integrated into existing video diffusion models to improve their visual plausibility.


Peer reviews:
comments: SUMMARY:
This paper proposes injecting 3D point trajectories into video diffusion models to improve physical understanding. The authors introduce PointVid, a dataset with pixel-aligned 3D tracks, which is used to train models with point-based regularization to encourage smooth motion and shape consistency. The method improves video realism and can be applied to most existing diffusion models.

STRENGTHS AND WEAKNESSES:
Strengths：

1. The paper addresses an important limitation of current video diffusion models: lack of 3D physical understanding, through a simple yet effective solution.

2. The proposed framework is model-agnostic and can be integrated into most existing video diffusion architecture with minimal changes.

Weaknesses:

1. The rigid loss introduced for point clouds appears critical to preserving physical structure. However, it's unclear whether similar effects could be achieved by applying it directly in the RGB space. An ablation comparing point regularization to RGB-space alternatives would strengthen the paper.

2. The framework relies on the first frame for reference and sparse point tracking, which limits its scalability to long videos, multi-object scenes, or cases with occlusion and fast motion. The method is mainly demonstrated on short, single-object scenarios.

QUESTIONS:
1. Can the rigid loss be applied directly to object pixels in the RGB space instead of 3D point clouds? Would such an alternative achieve similar structure-preserving effects? An ablation would help clarify this.

2. Your framework relies on the first frame for segmentation and point tracking. How does the method generalize to longer or more complex videos with multiple moving objects or occlusions?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the task of image-to-video generation, intending to inject explicit 3D understanding into diffusion-based video models. Starting from a pre-trained video generator, the authors propose to use a new modality: a three-channel map encoding of foreground objects' 3D points as pixel coordinates (u,v) and depth d. These point trajectories are obtained with an existing 3D point-tracking model, creating a dedicated training dataset PointVid. The diffusion model is then fine-tuned to predict both RGB frames and the (u,v,d) maps simultaneously. During the next training stage, the predicted point maps serve as an additional regulariser, guiding the network toward better 3D understanding and yielding stronger temporal and object consistency in the generated videos.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Clarity of presentation. The paper is well-written, with clear explanations and informative visualisations that make the method easy to follow.\n\n2. Novel 3D regularisation. Introducing an implicit 3D point-map modality and using it as a regulariser is a creative way to enhance geometric consistency in video diffusion models.\n\n3. Compelling qualitative results. The method produces visually convincing videos that exhibit improved temporal and object consistency compared with baseline models.\n\nWeaknesses:\n\n1. Custom evaluation set. The main quantitative study uses a hand-curated subset of images, making it difficult to compare with existing or future work. Public benchmark suites such as VBench already support image-to-video evaluation and should be used for head-to-head comparisons.\n\n2. Insufficient dataset analysis. The paper does not examine failure modes of the SpaTracker point-tracking stage or analyse tracking quality in the generated videos. A brief error study would strengthen the empirical section.\n\n3. Missing cost breakdown. The additional regularisation loss incurs extra computation, but the paper does not report its runtime or memory overhead relative to the standard diffusion loss. \n\n4. Limited evaluation scope. Results are presented only in the image-to-video scenario, even though the technique appears directly applicable to text-to-video generation. Demonstrating this extension would broaden the paper’s impact.\n\nQUESTIONS:\n1. Can you quantify the extra compute cost your regularizer adds compared to the standard diffusion loss? How does that overhead grow as you increase point density?\n2. Have the authors conducted a systematic analysis of failure modes? In the supplementary video, the method exhibits improved temporal coherence but reduced motion amplitude. To what extent might the regularizer inadvertently suppress desirable motion dynamics, and how might this be mitigated?\n3. Why choose a custom image subset over the public VBench split for evaluation? Could that decision skew your results or limit generality?\n4. The technique seems readily applicable to text-to-video generation. Have you tested this scenario, and if so, can you share qualitative or quantitative results? \n5. How does the number of tracked points affect output quality? Can you achieve similar results with a much sparser point cloud, and what trade-offs arise?\n\nLIMITATIONS:\nPartially. \nThe manuscript lacks a failure-mode analysis, does not report the runtime or memory overhead introduced by the regulariser, and relies on a private evaluation set, preventing comparison with other improvement methods. Including these elements would substantially strengthen the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a video generation framework that incorporates 3D geometry and dynamic information into 2D videos. It introduces PointVid, a 3D-aware dataset created by augmenting videos with 3D point trajectories aligned in pixel space. By fine-tuning a latent diffusion model on this dataset, the method enables tracking of objects in 3D space and regularizes their shape and motion to reduce artifacts. This leads to improved visual quality and better handling of complex, contact-rich scenarios. The approach is compatible with existing video diffusion models to enhance their realism.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n- The paper is clearly written and easy to follow. The authors effectively communicate their motivation and methodology in a straightforward manner. Additionally, the figures are clean and illustrative, contributing positively to the paper’s readability.\n\n**Weaknesses**\n\n- As acknowledged in the Limitations section, the proposed method inherits constraints from the base diffusion models. To better support the claimed generality of the plug-and-play design, it would be beneficial to include additional experiments with a broader set of base models. Using only I2VGen-XL and CogVideoX 1.5 is insufficient to convincingly demonstrate general effectiveness.\n\n- The use of a VAE to encode latent features of the point modality raises concerns, as the VAE was not specifically trained on point-based data. It is unclear whether point regularization alone can adequately address this mismatch.\n\n- I am concerned that the second and third terms in the reconstruction loss (Equation 1 of the main manuscript) may introduce a trade-off—such as improved temporal consistency at the cost of reduced temporal dynamics. It would be helpful if the authors could address this potential issue.\n\n- The paper would benefit from a clearer description of the generated 3D trajectories. Are there any qualitative visualizations or quantitative metrics to evaluate the generated 3D points?\n\nQUESTIONS:\nPlease check above weaknesses.\n\nLIMITATIONS:\nThe authors addressed the limitations in main manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes a new video generation framework that integrates 3D geometries and motions. The key to the proposed method is: 2) The use of 3D point tracking as an additional modality along with RGB, 2) Regularization on the shapes and motions of objects represented by 3D points.

STRENGTHS AND WEAKNESSES:
Strength: 
* The proposed framework with additional 3D point modality is well motivated, with simple and effective design.
* The proposed framework can be seamlessly integrated into existing video diffusion models for better physical plausibility.
* The paper is well-written and easy-to-follow.

QUESTIONS:
Can the proposed method handle the cases where new objects appear in the mid of videos?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03