Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach
Authors: Yunuo Chen
Junli Cao
Vidit Goel
Sergei Korolev
Chenfanfu Jiang
Jian Ren
Sergey Tulyakov
Anil Kag
Venue: NeurIPS 2025 poster ()
Keywords: video generation
Primary area: deep_learning

Abstract: We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling it to track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video to eliminate undesired artifacts, e.g., non-physical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent in current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos, where 3D information is essential for perceiving shape and motion of interacting solids. Our method can be seamlessly integrated into existing video diffusion models to improve their visual plausibility.

Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach
Yunuo Chen 22 Junli Cao 1.2, Vidit Goel?, Sergei Korolev?, Chenfanfu Jiang', Jian Ren', Sergey Tulyakov Anil Kag" University of California, Los Angeles, SSnp Inc.
Abstract
We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space. The resulting 3D-aware video dataset, PointVid, is then used fine-tune latent diffusion model, enabling track 2D objects with 3D Cartesian coordinates. Building on this, we regularize the shape and motion of objects in the video eliminate undesired artifacts, e.g., non-physical deformation. Consequently, we enhance the quality of generated RGB videos and alleviate common issues like object morphing, which are prevalent current video models due to a lack of shape awareness. With our 3D augmentation and regularization, our model is capable of handling contact-rich scenarios such as task-oriented videos, where 3D information is essential for perceiving shape and motion of interacting solids. Our method can be seamlessly integrated into existing video diffusion models to improve their visual plausibility.
Introduction
The recent development of video diffusion models has attracted significant research interests and seen tremendous progress [5, 35, 15]. With vast amounts of training data and sophisticated architectures, video models have greatly improved in expressiveness and aesthetics [6, 37]. Today, state-of-the-an video models can realistically represent content from the real world with high creativity, enabling various applications in the entertainment industry and scientific research [27]. However, existing video diffusion models primarily focus on improving the appearance of content and the smoothness of motion. The models are trained in a way to understand the movements of 2-dimensional pixels, where the object movements in 3D space is approximated through the changing of RGB values.
As result, the underlying 3-dimensional shapes and motions, which represent how objects truly exist in the physical world, are not learned well in the existing video generation models. Even worse, we observe that when the video diffusion models neglect the underlying 3D information, they struggle to understand how objects occupy space, change shape and location, and interact with each other, which is essential for generating videos with complex motion. For instance, imaging situation when perceiving a scene with a human performing specific tasks, e.g., hands tying shoelaces, viewpoint projection and occlusion make it impossible capture the true 3D shape of the hand and the shoelaces using 2D pixels, complicating the modeling of how they make contact. Without understanding the underlying geometry, objects the video may change shape arbitrarily or even appear and disappear suddenly, leading to the common issue of object morphing (as example results from prior arts in Figure 1). This problem is exacerbated for complex objects like humans; without 3D awareness, limbs and body parts may move unnaturally or change shape abruptly, resulting in dislocated arms or twisted bodies.
Work done during internship at Snap Inc.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prompt
Cog 1.5
DynamiCratter
book plame
stiring food
as gotton mome.
baoi
resorna speaker from
car
6B The person bokdau sice of d'dppiin with deer Bands.
Figure 1: Comparison on Task-Oriente Videos. We present videos generated by different baselines (Cog VideoX 1.5 [43], SVD [5], DynamiCrafter [42], LTX Video [14] and I2VGen-XL. [46]) and compare them with our method. We use the same input conditions for all methods (except that SVD is conditioned only on the image). It can be observed that existing baselines often exhibit severe distortions of hands or objects during human hand-object interactions. In contrast, our method preserves the shapes of both the hand and object during such interactions.
To address the above issue, aim improve video diffusion models towards the better understanding of physical world, thus generating more reasonable shape and motion. Previous works have attempted control shape and motion directly in 2D space [34, 25]. Nonetheless, without a comprehensive understanding of 3D geometry, these methods are limited to directing pixel movement confined within the 2D plane, making them unable to handle out-of-plane motions or fully resolve 3D dynamics [25]. One straightforward approach could be to model the complete 3D geometry of objects using representations such as NeRF [28] or 3D Gaussian-S Sppiiatn--paaigngggg [20], essentially leading to 4D generation [3, 48]. While these methods provide accurate 3D understanding, they computationally intensive and generally limited to generating simple scenes with only few specific types of objects.
To bridge the gap in leveraging 3D information for improved video generation, we propose method augment and regularize video diffusion models using 3D point clouds. Specifically, our model enriches 2D videos with out-of out-of-plane data data-absent in traditional RGB video models withhou requiring full-scale 3D reconstruction. Instead of generating complete mesh or point cloud for the video diffusion model, we track the movement of 2D pixels in 3D space, creating pseudo- representation objects the video. To support this optimization, we develop a 3D-aware video dataset, named PointVid, which includes segmented objects in pixel space along with their corresponding 3D dynamic states. We fine-tune video diffusion models on the PointVid dataset to extend their generative capabilities to 3D shape and motion, enhancing the model's ability to understand object behavior in the physical world. By leveraging prior knowledge of 3D space, we use this additional information to guide pixel movements toward more visually plausible outcomes.
Incorporating out-of-plane 3D information as prior, our approach does not rely on specific architec- tures of the diffusion denoiser and thus can be integrated into various video models. To demonstrate this, we implement our method on two base models with distinct architectures: U-Net-based I2VGen- XL [46] and DiT-based Cog VideoX 1.5 [43]. Guided by 3D trajectories, our model achieves superior quality terms of smooth transitions of shape and motion compared to existing works, generating visually plausible videos. We summarize our contributions as follows:

---

1. Injecting 3D Knowledge into Video Diffusion. We propose a novel 3D point augmentation strategy to bridge the 3D spatial domain of the physical world with the pixel space of 2D videos. By explicitly incorporating 3D information into the video diffusion process, our method significantly enhances video generation quality, especially in challenging scenarios such as solid object interactions.
2. Regularization with 3D Prior. We introduce a novel regularization strategy to guide the point cloud diffusion process, ensuring that the generated point cloud is well-structured. This structured point cloud, in turn, enhances the RGB video generation by providing robust 3D priors for better spatial coherence and alignment.
3. 3D-Aware Video Dataset. We curate a 3D-aware video dataset, PointVid, by tracking 3D points in the first frame of given video. We segment the first frame to obtain foreground objects and mainly track these points in the original video. The resulting dataset contains 3D coordinates corresponding to the first frame, pixel aligned with the video dataset.
2 Related Work
Video Diffusion Models. Following the success of diffusion-based text-to-image (T2I) models [I, 30, 11, 24, 22, 26, 38, 23, 12, 19], many video generation models have been proposed in the literature [5, 27, 15]. These models allow video generation following input conditioning such as text and image. Video diffusion models can be categorized into three categories. First, auto-regressive models like VideoPoet [21] follow the causal language model paradigm of next token prediction given previous tokens. These models encode the input conditioning into latent tokens using tokenizers such as MAGVIT [44].
Second, pixel-space diffusion models [27, 15, 35) directly model the video synthesis on the pixel space to avoid artifacts arising in the compressed VAE latent spaces. While these models provide more photo-realism and better motion dynamics, they typically only offer low-resolution video generation due to higher computational cost in generating high-resolution videos. These base models are augmented with super-resolution components to increase the video resolution resulting in deep cascaded diffusion models. Imagen-video [15], a UNettbased architecture, generates videos at low-resolution and utilizes sequence super-resolution models to increase the video resolution. Make-a-vides [35] uses a T2I as prior to encode text input and trains a deep cascade of spatial and temporal layers to generate high resolution videos. Video [27] replaces the popular UNet architecture design with a FIT network for improved generation efficiency.
Third, latent diffusion models transform the raw pixel space into compressed temtool latent representation. enables training a higher resolution and higher capacity base model instead of pixel-space diffusion models. These include Stable Video Diffusion [5], SORA [6]. VideoCrafter2 [8], ModelScopeT2V [39], Mochi [37], LTX Video [14], Cog Video [43, 16) etc. While latent models are
typically single-stage pipelines, they can extend to deep cascade pipelines to offer even higher-quality video generations.
videoogenerationns MovieGen [31] proposed a transformer-baat based cascade latent diffusion model.
While our approach is applicable to various types of video diffusion models, we select two models with distinct underlying architectures 12VGen-XX -L [46] and Cog VodeoXX [43]- as our base models, and demonstrate how incorporating 3D geometry and dynamics improves video generation.
Dynamics Aware Video Generation. There have been various works [34, 25, 47] that instill physically plausible and temporally consistent dynamics in video generation. PhysGen [25) added simple control such as force or torque to object in within the image, enabling simple physically plausible dynamics to the resulting video. MotionCraft (34] synthesized videos by incorporating optical flow in the noise diffusion process. This allows image animation with a temporally consistent video content. Physdreamer [47] tried distilling dynamic priors of static 3D objects from the video generation models. It creates physical material field around the 3D object, thereby easily synthesizing 3D dynamics under arbritary forces. Although these works are able to incorporate simple dynamics to a static image, they assume that the underlying diffusion model understands and captures the object dynamics. This assumption is not necessarily true, as pointed by VideoPhy [4], current video generation models suffer from many inconsistencies, both physically implausible as well as temporally inconsistent. There have been other works along the line of incorporating camera intrinsics during video generation. 3D-Aware diffusing noise along a motion and content codes while sampling different camera poses. The model
[2], first trains a model by jointing
3

---

Missiignmnt
Reciedras tae Pakily Fandly
LDM
SU-Alggmmett
prrtt
Terppel Searthoee
Ap-2
Phool-alogned. 3D Points
Condition Cruss A ttonson
RGB and Points Cimos A thrntion
Figure 2: Method Overview. During training, sample video-point pairs, concatenate them along the channel dimensions, and use the augmented data to train latent diffusion model. We introduce cross-attention between video and point data in corresponding channels enhance alignment between the two modalities. The model predicts both RGB video and 3D points, leveraging the 3D information further regularize video generation by applying a misalignment penalty during the diffusion process. During inference, the model generates both video and points from random noise, conditioned on text-image prompt.
is trained using a generative adversarial network based discriminator which discriminates the resulting video, by difference between frames and time duration.
In contrast to previous works, we incorporate 3D knowledge by jointly diffusing point clouds alongside the video modality. This enhances the diffusion model's understanding of 3D world during training and improves the visual plausibility and spatio-temporal consistency of the generated videos.
3 Method
As mentioned earlier, current video models primarily learn moving objects sequences of 2D pixels, without considering their true 3D states. Their understanding of the physical world limited to a projected plane, making it challenging resolve changes in shape and motion that are not visible in the projected view. To address this, we propose incorporating a partial 3D representation into the generation pipeline. The inclusion of 3D information as a second modality will not only enhance the capability of the video model but also guide the generation of RGB videos through joint optimization.
In this section, we outline the details of our 3D-aware video generation pipeline, which consists of three essential components: First, we introduce our novel 3D-aware video dataset, Point in subsection 3.1, providing videos with spatial information orthogonal to color dimensions. Next, in subsection 3.2, we present our joint training approach, enhancing the video model's ability to incorporate 3D geometry and dynamics. Finally, deseribe 3D regularization strategy in subsection 3.3, designed to resolve non-physical artifacts the generated results using the newly incorporated modality.
3.1 PointVid Dataset
We build our 3D-aware PointVid dataset upon standard 3-channel videos of shape [T, H,W,W,WC C]. where C. = 3 encodes the RGB channels, height H and width W digitize 2D images into pixels, and T represents the temporal dimension that sequences the pixel data over time.
The key challenge lies in selecting a 3D representation that is suitable for learning by a video model. Explicit 3D representations, such meshes (surface volumetric), point clouds, voxels, can effectively model geometry but often have variable sizes due irregular object shapes, complicating the diffusion training process. Moreover, aligning 3D points with 2D pixels in video remains non-trivial task. In contrast, implicit representations like NeRF [28] typically require substantial time to convert into explicit form for direct usage, making them impractical for training purposes.
Recent advancements 3D point tracking [41] have made this task more feasible. Instead of representing moving objects as sequences of meshes point clouds with arbitrary resolution, we focus exclusively on tracking the movements of pixels. Here, 2D pixels are unprojected into 3D space and tracked throughout the video. This allows the 3D locations of the pixels to be treated three

---

Vitro 9. [7'x lI W
Refoensz Franse
Samanti Stancettaionn
Randen Query
3D Poins Trackins
Poire oP IT x Wx
Figure 3: VointVid Dataset Generation. Given an input video, the first frame as a reference frame and perform semantic segmentation to obtain masks for foreground objects. Next, we randomly sample pixels with distribution favoring pixels inside foreground objects. We perform 3D point tracking on these queried pixels, and map these points to the input video frames. The resulting data point contains 3D coordinates of tracked foreground pixels while remaining pixels are zeroed out.
additional attributes, similar to color intensities. As a result, we can format the 3 spatial coordinates as extra channels, yielding point tracking tensor P of shape [T" H, W, CJ, matching the dimensions of the RGB video. In practice, we store the 3D tracking information using 2D pixel coordinates (u, v) and depth d, requiring only simple camera unprojection to reconstruct the 3D world coordinates (z, y, 2). By using (u, v, d). the first two channels correspond to the pixel's s position projected onto the 2D plane, which aligns precisely with the moving point in the RGB video, thus simplifying the task of learning the alignment between video and point tracking.
We visualize the workflow of generating point tracking in Figure 3. For each RGB video V. we extract the first frame as a reference image and perform semantic segmentation using Grounded-S AM-2 [33, 32]. yielding masks for the foreground objects. Next, we randomly sample pixels on the reference image with a main focus on those inside the foreground objects. These queried pixels are then tracked with trajectories of 3D Cartesian coordinates (x. y. z) throughout the video using SpaTracker [41]. Finally, we post-process the tracked results to align the shape with the video: we fill out the pixels in the foreground mask to store the pixel's ' spatial positions, formatted as 3 channels, and discard pixels in the background (setting all background pixels to zero). Since we apply sparse queries for tracking efficiency, some foreground pixels may not have corresponding trajectory; searching to interpolate the positions for such pixels.
KDDTree
One remaining issue encounter in the tracked results is that they sometimes contain tiggufrreuency noise, possibly due to the imprecision of the tracking. This causes problems when the video model attempts to learn from its distribution, as the backbone diffusion model relies on sampling with random noise. To address this, we apply a Kalman Filter for 3D point tracking to remove temporal frequeryy noise and recover its primary motion.
3.2 Point Augmentation in Diffusion Denoiser
With our 3D-aware video dataset, we now aim to extend the capabilities of the video model to generate not only the RGB video V but also the corresponding 3D representation P. Aligning the video with the point cloud enables the model to inherently acquire 3D knowledge. The augmented vidco-point model learns not only how the object appears in 2D color image but also its 3D geometry.
Our augmentation pipeline is illustrated in Figure It is worth noting that training the video- point model entirely from scratch is unnecessary, as training video diffusion models already requires significant computational resources and large-scale datasets. Instead, it is more practical to fine-tune pretrained video model to incorporate the point-tracking modality. Additionally, our injection of 3D points is agnostic to the network architecture of the pretrained video model. In this work, we employ both UNet- and DiT-based denoisers sprrrsssssccccllll..c.as 12VGen-XI [46] and Cog VideoX-1.5 conditioned on text-image prompts. While our training pipeline incorporates an additional point modality, inference remains conditioned only on text-image pairs, consistent with the base model.
both
To collectively learn from and generate V and P. the straightforward way would be to feed them together to the diffusion model. Since we are matching the shapes of video and point tensors, we can concatenate them along the channel dimension, resulting in an point-augment shape of [T. H, W. 2C). This concatenated video can be seamlessly integrated into the denoiser by simply adjusting the input/output dimensions to accommodate the additional channels. This increase in dimensionality does not overshadow the weights of the original model; instead, only new projection layers are introduced to handle the added modality. To leverage pretrained RGB weights, we can load the pretrained model for the corresponding RGB channels and initialize the remaining weights as
video VP with a
5

---

zeros. For the conditioning frame, we continue using the first RGB frame and repeat the condition image tensor to prompt the generation of both video and point tracking, as our goal is for the model not to rely on the initial point positions during the inference stage. This strategy is effective because the same reference frame is used for generating the point-tracking data earlier.
In addition to the pixel-wise alignment of video and points during dataset construction, we introduce cross-attention layer at the input of the latent diffusion model. This layer links the first and second halves of the channel dimensions--coreeppouui cere--coorrppnnnnggge to color information and spatial location in the latent space- to further enhance their alignment. We apply two passes of cross-attention, with each modality alternately serving as the query, while the other provides the key and value in each pass.
Incorporating the aforementioned changes, the training pipeline proceeds as usual: we encode VP into the latent space 2, add noise to obtain ž₁ using randomly sampled noise and a diffusion step t. The denoiser model then predicts the added noise and compares it with the ground-truth noise, where we apply DDIM [36] for diffusion sampling. With this modified pipeline, we fine-tune the diffusion model on Point Vid dataset to obtain our video-point generation model. At inference, given an image and a text prompt, the model takes in double-sized random noise and predicts both the RGB video and the corresponding 3D trajectories.
3.3 Point Regularization
With the model capable of generating pixel-aligned video and points, we leverage this additional modality to enhance RGB video generation. To achieve this, we introduce regularization terms on the point tracking component of the diffusion output, aligning it with the ground truth, which in turn implicitly improves the RGB video component (see Figure 2).
Firstly, we need to recover directly usable point trackings for evaluation in 3D space. Since injecting random noise into the point cloud would likely render it unusable, we employ diffusion sampling to recover noise-free stage before applying regularization. each training step, given a latent input and its corresponding noisy stage 2,, we apply DDIM denoising to iteratively recover 2g. Following the approach of [45], we adopt gradient checkpointing freeze gradient flow except during the final step, thereby preventing memory overflow. The denoised latent 20 is subsequently decoded into explicit 3D point tracking P. representing the sampled 3D trajectory of the video.
With the explicit modeling of trajectories 3D representation, we introduce regularization terms to improve the quality of the generated results, focusing on motion fidelity and temporal smoothness. For simplicity, let us assume P is reshaped to T x N x 3, and use pr to denote the point cloud of shape N x 3 at time
Ground Truch Point
Without Regulatization
With Regularizatinn
common issue that arises during point track- ing generation is the presence of excessive noise (see Figure middle). This noise can originate from various sources, such as the variational autoencoder (VAE) used to encode P into the latent space. High-frequency noise in both spa- tial and temporal dimensions tends to obscure the main shape and motion, thereby impairing the video model's ability to track 3D points effectively. To mitigate noise and increase fidelity of reconstruction, propose the reconstruction loss Lrecon follows:
Point Regularization. The recon- structed point cloud in the diffusion output often contains noise and deformations (middle). This issue is mitigated using our point regularization (right). The synthetic point cloud above (e.g., box and shoes falling the ground) is generated by Kubric [13] and trained with our pipeline.
Figure 4:
T 11 P-1
T
T
Crecon Co
Σ
+ C1 Σ 11p0 - -p--p--11-- Σ 1pp- - 2pr-1 + p1- (1)
Here the first term penalizes the difference between generated point cloud and the noise-free free ground truth P. ensuring generation fidelity, while the second term penalizes the difference between con- secutive time step to encourage temporal smoothness; the third term further enforeces acceleration smoothness, discouraging abrupt changes in velocity. We selected the weights 6; so that each term are about the same scale initially.

---

Apart from the microscopic high- freeuenyy noise, we observe macroscopic shape deformation many of the generated point cloud (see Figure bottom middle). Such deformation of arbitrary degree is undesirable as most solid objects, including human body, inderently requies conservation of volumn and local rigidity. Violating this can lead to unnatural shape deformation and morphing, which is considerd undesired in our generation task. Hence we propose the rigidity loss Figgid to enforce local shape preservation:
Ligid ΣΣ JEN(()
(dist(P7,, P7) (?))
Here, we apply kNN search on the reference frame to construct local neighbor pairs, then penalize changes in distance between these pairs.
By combining reconstruction and rigidity loss, enhance the generation quality of the 3D point cloud, as shown in Figure 4 (right). This improvement in the point space implicitly promotes shape preservation and smooth motion transitions in the RGB video, as our video-point model closely aligns the two modalities. To prevent the geometric regularizations from dominating the semantic aspects, we jointly optimize these regularization terms with the standard diffusion denoiser losses Lai [36] during training, resulting in an overall loss of the form (where X are corresponding weights):
L Aam + Arecon Frecom + Arigid frigid
(3)
Experiments
In this section, demonstrate the effectiveness of proposed method through both qualitative and quantitative evaluations. Our fine-tuning approach can be seamlessly integrated into different base video diffusion models. In our experiment, we adopt two baseline image-to- video models with distinct architectures: a UNet-based model, 12VGen-XL, [46], and DiT-based model, Cog VideoX 1.5 [43]. We refer readers our supplementary material for more details the training setup.
4.1 Qualitative Results
Although our model is designed to generate sequences of 2D images, our point augmentation and regularization improve its ability to capture inherently 3D motions. To evaluate this, test models on unseen images and visualize the generated RGB videos. For generation quality, we primarily assess visual plausibility, with particular focus shape consistency and preservation. We refer readers to our supplementary document and video for additional results.
Prompt
12 XL
Ous (LNe)
E-pVidooX-1.5
del is black and white
ahortagggh suicer
sivisig mash.
Oun ee
(Ds
ii male gonssr
Coo V L
Duca
4e using pousr
that farge venden
noble
Figure 5: Qualitative Comparison. We compare our UNet and DiT models against their respective baselines. The results show that both base models exhibit unrealistic artifacts, such as morphing, while our models ensure smooth transitions in object shape and motion.
7

---

Table 1: Quantitative Evaluation. We evaluate various aspects of method against other models using the VBench [17] and VideoPhy [4] benchmarks. The evaluated metrics are as follows: (VBench) SC: subject consistency, BC: background consistency, MS: motion smoothness, AQ: aesthetic quality, IQ: imaging quality; ( (VideoPhy) PC: physical commonsense. Our UNet and DiT models are compared against their respective baselines, with higher scores highlighted in bold, while the highest achiever among all models is underscored.
Method
S
BC f
MS 1
AQ f
IQ 1
PC
SVD
0,91807
0.93563
0,97735
0.43276
0.55174
0.34707
LTX Video
0.89001
0.90423
0.99152
0.44670
0.58667
0.33400
12VGen-XL
0.83704
0.89356
0.96144
0.43140
0.60338
0.32876
Ours (UNet)
0.96003
0.95120
0.98458
0.42246
0.61003
0.37805
CogVideoX 1.5
0.91754
0.92923
0.98798
0.44534
0.60067
0.36549
Ours (DiT)
0.91768
0.92987
0.99201
0.45298
0.60531
0.41384
Comparison with base models. We compare our models side by side with their corresponding base models: 12VGen-XL. [46] for the UNet model and Cog VideoX 1.5 [43] for the DiT model. Using the same conditioning text and image, we visualize the generated RGB videos in Figure In the videos generated by the base models, noticeable artifacts such as object morphing and unnatural deformations can be observed- for example, the car in (a), the gymnast in (b), and the power saw in (c). In contrast, models effectively eliminate unnatural shape and appearance distortions, resulting in visually plausible outputs. While our UNet and DiT models differ architecture and pretrained baselines, we consistently observe a reduction in artifacts compared to their respective base models.
Task-oriented videos. A prevalent category of real-world videos features humans performing specific tasks (see Figure 1), such as playing with toy, cooking food, or handling tools. To highlight the challenges of such scenarios, we evaluate three additional video models: SVD (stable-video- diffusion-img22idxxxx [5]. DynamiCrafter [42], and LTX Video [14), alongside the two baseline models. We observe that these models often struggle capture localized details, particularly how human hands interact with objects. This challenge stems from the inherently complex 3D nature of these tasks, which cannot be fully captured using only 2D data. As result, hands and objects frequently undergo severe distortion or blurring, With our 3D regularization, our model effectively preserves the shapes of both hands and objects, ensuring smooth transitions throughout the video and demonstrating consistent quality improvements over the compared models.
4.2 Quantitative Results
To quantitatively evaluate the performance of our model, we adopt test batch of randomly selected images. The test images are sampled from a single batch of video clips (unseen by our model) from the video dataset [9], with the initial frame randomly selected within each clip, totaling 372 images. To ensure consistency, we use the same test batch across all quantitative evaluation metrics (including ablation studies in subsection 4.3 and user studies in subsection A.5). In this section, compare various aspects of video quality between our two models and their respective baseline models, along with two additional models for reference. The results summarized in Table 1.
First, we assess the general aspects of video generation quality using VBench [17], evaluating both temporal quality (subject/backgroumn consistency and motion smoothness) and frame-wise quality (aesthetic and imaging quality) the videos. The results show that our fine-tuned models outperform their baselines in most categories, demonstrating the effectiveness of our approach in improving video generation qualitics, particularly in content consistency.
Beyond these general aspects, we are particularly interested in how the injection of 3D awareness enhances alignment with real-world physical principles. whether 2D video is physically accurate is largely subjective, as ground-trute -ruth 3D physical states are not available. Therefore, we utilize VideoPhy [4], which employs human annotations to train a discriminator that provides physical commonsense score. VideoPhy defines physical commonsense from various perspectives, such as solid mechanics, where solid objects are expected to retain their shape and size throughout the video, aligning with our objective. As shown in Table 1, our models
important to note that determining
8

---

Prompt
No Point
(i)) No Regularization
(ii) Ours (Full)
(iv) No Cross-Attention
No Diffusion Loss
(aj & person using a blue puintbruuk to paint while stors ou table.
(b) strring spo0s.
hot chocolate with
Figure 6: Ablation Studies on Different Components. We compare the results from (iii) our full generation pipeline with (i) training with video only, (ii) training with point augmentation only, (iv) training without channel cross- -tts-att,n,ion, and (v) training without diffusion loss.
achieve higher score for alignment with physical principles, outperforming their respective baselines and other tested models.
To further demonstrate the efficacy of our 3D point augmentation and regularization modules, we evaluate the mean squared еттог (MSE) of the predicted 3D point trajectories during diffusion (using the input point tracking tensor as ground truth), as summarized in Table 2. We observe that, compared an untrained diffusion model, network design effectively predicts 3D point
Table 2:
Evaluation of Points. We report the MSE of predicted 3D points for the untrained model, the model without regularization, and the model with regularization, respectively.
Item
Untrained No Reg
With Reg
MSE
0.2125
0.0031
0.0020
trajectories after the augmentation stage and further reduces the error through the regularization stage.
4.3 Ablation Study
We conduct ablation studies on our UNet-based model to evaluate the ef-
Table 3: Quantitative Ablation Studies. We use
VBench [17] to evaluate the subject and background con- sistency (SC/BC) of our full model, as well as models trained without point augmentation (PA), without point regulariza- tion (PR), cross-attention (CS), and diffusion loss (DL).
fectiveness of our training pipeline de- sign. We visualize the inference re- sults in Figure 6 and use VBench [17] to evaluate the models' subject and background consistency in Table 3.
Metric
Full
No PA
No PR
No CS
No DL
Our two-stage design of point aug- mentation and regularization progres- sively enhances the video model. Compared training the model with RGB videos only (Figure 6 (i)), our point augmentation alone (Figure 6 (ii)) injects 3D awareness into the model and improves its ability to perceive 3I) shapes, evidenced by more consistent object shapes in the videos. Our point regularization (Figure 6 (iii)) further improves quality by optimizing point generation and implicitly guiding RGB generation towards higher fidelity. Our channel-wise cross-attention mechanism ensures cross-dimensional and cross-modality alignment, without which video quality may degrade (Figure 6 (iv)). Furthermore, in the regularization stage, adopt a joint training strategy that combines regularization loss with diffusion loss. We observe that dropping diffusion loss (Figure 6 (v)) causes 3D information to dominate semanties, resulting in completely degraded results.
SC
0.9681
0,9434
0.9544
0,9332
0.7166
BC
0.9639
0.9492
0.9522
0.9329
0.9254
5 Conclusion and Discussion
To summarize, we propose a generation framework that incorporates 3D-awareness into video diffusion models. By tracking objects in 3D and aligning them in pixel space, we elevate traditional video datasets to new dimension, revealing out-of-plane information previously inaccessible to video models. Through joint training, the video model learns to perceive 3D shape and motion, acquiring physical commonsense that is inherent in 3D. We then apply regularization to refine the

---

generation process and further enhance the results, eliminating artifacts such as object morphing. Our method can serve as an augmentation for various video models, and we showcase its effectiveness by fine-tuning two widely-used video generation models (I2VGen-XI, Gl. and Cog VideoX 1.5). The results demonstrate a strong capability in reducing non-physical artifacts in video generation, particularly when applied to contact-rich, task-oriented videos.
Motion Magnitude and Smoothness Trade-off. In the design of our reconstruction loss (Equa- tion 1). we align the generated 3D points with the ground truth positions while enhancing temporal smoothness by minimizing velocity and acceleration. While we acknowledge that this loss introduces a trade-off between motion magnitude and smoothness, the global dynamics are primarily governed by the position term (to which we assign larger weights). As a result, we observe faithful reconstruction of ground-truth dynamics during training (see Table 2). Moreover, discouraging abrupt changes in velocity is often beneficial, as they typically induce morphing artifacts in generated video (see Figure 5). We refer the reader to subsection A.2 for more details on the training setup.
Limitations. We acknowledge few limitations of our framework. The degree of 3D-awareness, rooted in our video- joint training, is constrained by the resolution of our 3D points. While our method does not require highly accurate point tracking for 3D prior, objects that are not sufficiently represented may lead to suboptimal 3D perception by the video model. Recent advancements in 3D point tracking [29, 40] could be deployed to enhance the 3D perception in our model. Additionally, the generation quality of our model is limited by the base diffusion backbone and could potentially be improved by employing larger-scale models. We leave these aspects as directions for future exploration.
Concurrent Works. Concurrently, several studies have emerged to enhance shape and motion consistency video generation. [7] uses optical flow as a representation of object motion and does not explicitly model object geometry. [18] proposes using point tracking to improve coherence; however, it only utilizes 21) pixel correspondence without 3D spatial information. On the contrary, our method explicitly models the 3-dimensional geometry and motion to provide spatial guidance to the diffusion model.
References
[1] Stability AL. Stable diffusion 2.1. http::/wwneiiiggec..oohhiiihlaad
2022. [2] Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Hao Tang, Gordon Wetzstein, Leonidas Guibas, Luc Van Gool, and Radu Timofte. 3d-aware video generation, 2023.
[3] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7996 8006, 2024.
[4] Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsense for video generation. arXiv preprint 206.022406. 2024.
[5] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Vidco generation models as world simulators. 2024.
[7] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-mmtton representations for enhanced motion generation in video models. arXiv preprint arXiv:2502.02492 2025.
[8] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang. Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024.
10

---

[9] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao. Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming-Ilsuan Yang. al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 13320-13323 13331, 2024.
[10] LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving 1mm. https: //github. com http///nttrnii//mmmeepooy// 2023.
[11] DeepFloyd. Deepfloyd.
2023.
[12] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, al. Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers, arXiv preprint arXiv:2405.0544
2024.
[13] Klaus Greff. Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Hermman, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hi Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Scla, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator, 2022.
[14] Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Fitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint 2.:1::5500.00000 2024.
[15] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022.
[16] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv: 2.5s.15865 15868, 2022.
[17] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vhench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 21807-21818, 2024.
[18] Hyconho Jeong, Chun-Hao P Huang, Jong Chul Ye, Niloy Mitra, and Duygu Ceylan. Track4gen: Teaching video diffusion models to track points improves video generation. In Proceedingy of the Computer Vision and Pattern Recognition Conference, pages 7276-7287, 2025.
[19] Anil Kag, Huseyin Coskun, Jierun Chen, Junli Cao, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov, and Jian Ren. Ascan: Asymmetric convolution-attention networks for efficient recognition and generation. arXiv preprint arXiv:2411.04940 0444777 2024.
[20] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Trans. Gruph., 139-),
[21] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng, Josh Dillon, Agrim Gupta. Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang. Hartwig Adam, Ming-Hsual Yang. Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold, and Lu Jiang. Videopoet: A large language model for zcro-shot video generation, 2024.
[22] Black Forest Labs. Flux.
hhap://sss...oooooooonnocccacaoa flrst--orsss-labbss 2024.
[23] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys. Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffuusin model mobile devices within two seconds. Advances in Neural Information Processing Systems, 36, 2024.
[24] Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Limmiao Xu. Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to-image alignment with decp-fusion large language models. arXiv preprint arXiv:2409. 10695, 2024.
[25] Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics- grounded image-to video generation. In European Conference Computer Vision 2024
11

---

[26] Xian Liu, Jian Ren, Aliaksandr Siarohin, Ivan Skorokhodov, Yanyu Li, Dahua Lin, Xihui Liu, Ziwei Liu, and Sergey Tulyakov. Hyperhuman: Hyper-realistic human generation with latent structural diffusion. arXiv preprint arXiv:2310.08579 2023.
[27] Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, and Sergey Tulyakov. Snap video: Scaled spatiotemporal transformers for to--to-video synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 7038-7048, June 2024.
[28] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[29] Tuan Due Ngo, Peiye Zhuang, Chuang Gan, Evangelos Kalogerakis, Sergey Tulyakov, Hsin- Ying Lec, and Chaoyang Wang. Delta: Dense efficient long-range 3d tracking for any video. arXiv preprint arXiv:2410.24211, 2024.
[30] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955 2023.
[31] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410. 13720, 2024.
[32] Nikhila Ravi, Valentin Gabeur, Yuan- Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloc Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.0077 00.144 2024.
[33] Tianhe Ren, Shilong Liu, Ailing Zeng. Jing Lin, Kunchang Li, Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng. Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
[34] Luca Savant Aira, Antonio Montanaro, Emanuele Aiello, Diego Valsesia, and Enrico Magli. Motioncraft: Physics-based zero-shot video generation. In Advances in Neural Information Processing Systems, 2024.
[35] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, In The Eleventh International Conference Learning Representations, 2023.
[36] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint arXiv:2010.02550 2020.
[37] Genmo Team. Mochi new sota in open-source video generation. http::///ggttuub..oogemaeoee/e genmoaii models, 2024.
[38] Kolors Team. Kolors: Effective Training of Diffusion Model for Photorealistic Text-to-Image Synthesis. arXiv preprint, 2024.
[39] Jiuniu Wang, Hangjic Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang, Modelscope text-to-vidoo technical report. arXiv preprint arXiv:2308.066: 2023.
[40] Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackerv2: 3d point tracking made easy. arXiv preprint mrXiv:2507.12462 2025.
[41] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20406-20417, 2024.
[42] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with vidco diffusion priors. In European Conference on Computer Vision pages 399-417. Springer, 2025.
[43] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06077 2024.
12

---

[44] Lijun Yu, Yong Cheng. Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann. Ming-Hsuun Yang. Yuan Hao. Irfan Essa, and Lu Jiang. MAGVIT: Masked generative video transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2023.
[45] Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng. Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, Dong Ni. Instructvideo: instructing video diffusion models with human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6463-6474, 2024.
[46] Shiwei Zhang, Jiayu Wang. Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: x1: High-quality image-te to video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04143 2023.
[47] Tianyuan Zhang, Hong- Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T. Freeman. PhysDreamer: Physics based interaction with 3d objects via video generation. arxiv, 2024.
[48] Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate124: Animating one image 4d dynamic arXiv preprint arXiv:2311. 146603 2023.
13

---

A Technical Appendices and Supplementary Material
Supplementary the main paper, provide more details dataset creation in subsection A. more details on implementation in subsection A.2, additional qualitative results in subsection A.3, and human preference evaluation in subsection A.5.
A.1 VointVid Dataset Generation
In this section, we present additional details on generating our 3D-aware video dataset, Point A visualization of the generation pipeline is provided in the main paper. Our pipeline requires no additional annotations beyond the video-text pairs in common video datasets.
Preparation Starting with any video-text pair, we trim the videos to a fixed length 7 and shape H x W. extracting the first frame of the clipped video as the reference frame. We then apply semantic segmentation to the reference frame using Grounded- SA---AMM22 [33, 32] to automatically generate masks for foreground objects. To streamline the process, we utilize a language model, LMDeploy [10]. to infer the main moving objects in the scene based on the video's 's caption. For example, the caption man swinging a baseball but studio results [man, baseball bat). This object-level label subsequently fed to Grounded-SAMM for prompt-based segmentation.
Point Tracking Having obtained reference frame and its segmentation, apply SpaTracker [41] to track pixel movements accordingly. We use the same reference frame to query points during tracking so that the 3D positions align with the first frame's pixel locations initially. We set downsample to 1 and gridsize to 80 in [41] to enable sparse tracking. We observe that sparse tracking this resolution provides sufficient number of pixels while remaining relatively efficient in terms of time. Increasing the resolution or using dense tracking could potentially enhance the results, but we leave this for future exploration. The model outputs point tracking data of shape T N where N is the number of sampled points.
Algorithm 1 Post-processing for Point Tracking
Require: Tracked points P RT*N83 Foreground mask M {0, 1) H*W Resolution (H, W) Ensure: Processed tensor E RTxMxWxx
I: Initialize: P -
Initialize all values to
2: P < KalmanFilterrD(P)
Reduce temporoal noise Initial wu of 2D pixels
3: Po :
4: P1 < PoΩM
b Filter foreground pixels Use initial ww for searching
5: Tr KITTree(P1)
6: for each foreground pixel M(u, uj = do
7: if' found P Pi] (u,v) then
> Tracked pixel
8: Pl: 0,; - P[.
9: else
> Untracked pixel
10:
ind + Que (2, v). k= 3) Pl:, v.:] < Interp ((: ind, :))
12: end if
end for
14:
return P
Post-processing To mitigate temporal noise caused by inaccurate tracking, we first apply a Kalman Filter the 3D point cloud estimate a global velocity and climinate high-frequency noise. The result is then reshaped to T' x H x W 3, where the spatial coordinates of pixels are treated as three channels, similar RGB. To address the difference in shape between the N tracked points and the video resolution, we apply the following rules: if a tracked pixel is not within the foreground mask, we discard its value (effectively setting all background pixels to zero). Otherwise, we assign its value to the corresponding pixel entry in the tensor. Since we use sparse tracking, only subset of foreground pixels have values. We then apply KDTree searching on the sparse pixels based on their uy coordinates in the first frame. For all untracked pixels within the foreground mask, we approximate their trajectory by interpolating their three nearest tracked pixels. The pseudocode for processocesiin is summarized in Algorithm 1.
14

---

A.2 Implementation Details
In this section, we provide more details on our experiments.
Dataset We collect approximately 100K videos from [9] and our internal dataset (each accounting for half) to form our PointVid dataset. The videos are processed to a resolution of 640 x 360 and cropped to 2-second length (the exact number of frames may vary due to varying input frame rates, typically is around 50 frames or more), and then paired with 3D point tracking using [41].
Model Training We implement pipeline two baseline models: UNet-based model I2VGen- XL [46] and DiT-based model Cog VideoX 1.5-5B [43]. For our UNet model, we use a resolution of 256 with 16 frames; for our DiT model, we use resolution of x360 768 with 48 frames. During finetuning, we train on 8 NVIDIA A100 80G GPUs with a batch size of 4 for UNet model and batch size of 1 for DiT model.
Joint Optimization To effectively inject 3D-awareness into video model without degrading the RGB modality (see subsection 4.3 for details), adopt joint optimization strategy combining diffusion and regularization losses. In our experiments, we observe that when the regularization loss significantly outweighs the diffusion loss (by orders of magnitude), tends to suppress motion. Conversely, when the diffusion loss dominates training, non-physical artifacts reappear. In practice, balance the two losses maintain comparable magnitudes, which consistently reduces reconstruc- tion misalignment and noticeably mitigates non-physical artifacts. To further preserve the motion magnitute, we apply the regularization loss less frequently (once every & iterations, use k 5). This adjustment preserves larger motion dynamics at the cost of requiring more training steps.
Computation Cost For our UNet model trained A100 GPU with batch size of 4, report the additional computational cost as follows: incorporating 3D components and regularization losses increases training time (~99 per iteration, compared ~4.75s when fine-tuning RGB video only) while maintaining comparable GPU memory usage (adding less than 2GB on top of +45GB)
A.3 Additional Qualitative Results
Additional Ablation Results We provide additional qualitative results from our ablation studies in Figure 7. Our point augmentation and regularization approach progressively enhance the visual quality of the generated videos, while the channel cross-attention mechanism and diffusion loss are also critical for ensuring high-quality results.
Additional Qualitative Comparisons We provide additional qualitative evaluations of model compared to other models: 12VGen-XLe XL [46], SVD [5], DynamiCrafter [42]. These include both task-oriented videos (see Figure 8) and videos from non-specific categories (see Figure 9), such as humans, animals, and general objects. Our results demonstrate improved shape and motion consistency of objects, effectively minimizing common non-phy non-physical artifacts like object morphing.
A.4 Additional Quantitative Results
While we use fixed test set in all quantitative evaluations to ensure consistency, also report the VBench [17] results of our UNet model using the official batch split provided by VBench. The results summarized in Table 4. We observe similar outcomes in the main comparison table (Table 1).
Table 4: Quantitative Results on VBench Test Batch. Our model outperforms the base model in most evaluation metrics.
Method
SC +
BC +
MS f
AQ 1
IQ +
Baseline
0.89703
0.94009
0.96993
0.52809
0.62961
Ours 0.96418
0.96722
0.98319
0.51754
0.65428
15

---

Prompt
(3) Point
(i) No Regulacization
(i Ours (Full)
(iv) Cross-Atiention
(v) Diffisson Loss
younng hand
looking
(B) A person minng tool
romove speaker from a
car.
(c) The person is holding a place of dumpling with their handd.
Figure Additional Ablation Results.
A.5
User Study
Apart from the quantitative evaluation using VBench [17] and VideoPhy [4], we additionally conduct a user study on our UNet model in comparison to its base model 12VGen-XL, [46] to assess human preference the models, focusing on identifying non-physical artifacts and evaluating overall visual plausibility. The following questions are designed for evaluation:
1. Which video appears to have more physically realistic object movements and interactions? (Consider aspects like gravity, collisions, and the natural flow of objects.)
2. (Negative) Which vidco contains morphing, stretching, or sudden changes in shape?
noticeable non-physical artifacts, such as object
3. In which video do the objects maintain a more consistent size, shape, and appearance throughout the entire sequence?
4. Which video better adheres to natural physics laws, such as consistent lighting, shadow behavior, and material properties (e.g., rigidity fluidity)?
5. Overall, which video feels more coherent and believable based on the physical interactions and behavior of objects?
Given the evaluation questions, we 10 labelers to compare our generated videos side by side with results from the base model and select the one that best fits each question. We use the same testing batch used quantitative evaluation. As shown Table 5, our method demonstrates significant improvement in terms of physical plausibility by incorporating 3D awareness into the video.
Table 5: User Study Results. Our model demonstrates significant improvement in physical plausibility, as assessed by human labelers. Here, Q2 asks the user identify negative artifacts in the videos, while the other four questions positively assess physical plausibility.
Method Q1
Q2 J
Q3 1
Q4 f
Q5 1
Baseline 0.138
0.862
0.135
0.132
0.137
Ours
0.862
0.137
0.865
0.868
0.863
16

---

Our
EvGen
SVD
DysamiCialll
(a) persou aring black marker culur the huir of stuffed animal.
SVD
Dynami
(b) chocolate with spoon.
wooden cutting board.
Duts
SVD
proparing Alack
Uuts
EVG
person piece of fabric.
JEN JEV
MIAN
bolding
Chi
DeG Xl
SVID
Dynanitring
A working with kix hands worksshyp
Figure 8: Additioanl Comparison on Task-oriented Videos.
17

---

(a A young mon standing front of motorcycle in gurage.
(i A mon sitting on noaf and wikking to the camera..
Velen
MnD
Dymmic
potting red exciosure.
Gue XL
d gutnca piga cage
GBCD GAac MANER
AvifoIGE CO
GEIDO GELOOGECO GEIDO
aeloo aeCoo
GEIO DE
UEICO GE
(c) An athiete racer his car arond turn on the mack
SVD
DymamiCaallur
close pony Muue hair:
12VGm-X ML
(g) A piace being furnace.
Figure 9: Additional Comparison on General Categories.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The main claims are supported by various experiments (see "experiments"
section).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
See "conclusion and discussion" section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides detailed descriptions of experiments (see "cxperiments" section and "implementation details" section of the appendix).
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The authors will release the dataset and generation pipeline upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
sosoodeeesmmissssooooococ for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //ni httppssscccppbbicguuuucuoe
/CodeSubmiss ssisnsoiicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
See "experiments" section and "implementation details" section of the ap- pendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper provides detailed information on the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See "implementation details" section of the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ /tturs://eeuips. cttp:://ourrss...pppll
Answer: [Yes]
Justification:
The paper conforms with every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper does not introduce new societal impact compared to its baseline models.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
23

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets used the paper are properly credited and respected.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper provides detailed information regarding new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]

---

Justification:
See "user study" section of the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: See "user study" section of the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
The paper does not involve LLM in its core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
25


Reviews:
[{"review_id": "KsoIMHljPv", "reviewer": "Reviewer_gf94", "summary": "This paper proposes a new video generation framework that integrates 3D geometries and motions. The key to the proposed method is: 2) The use of 3D point tracking as an additional modality along with RGB, 2) Regularization on the shapes and motions of objects represented by 3D points.", "questions": "Can the proposed method handle the cases where new objects appear in the mid of videos?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n* The proposed framework with additional 3D point modality is well motivated, with simple and effective design.\n* The proposed framework can be seamlessly integrated into existing video diffusion models for better physical plausibility.\n* The paper is well-written and easy-to-follow.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We appreciate your recognition of the effectiveness of our method. Please find our response to your question below:\n\n**Q1: Can the proposed method handle the cases where new objects appear in the mid of videos?**\n\nYes, our method can handle these cases effectively. For objects not present in the first frame, our model treats them as background and relies on the RGB modality to predict them. For example, in the ‘Aston Martin’ case (2’06’’ in the supplementary video), a second race car appears midway through the video, despite not being visible in the first frame. Since the RGB modality is jointly trained with the point modality, our method can accurately handle such scenarios. A similar situation occurs when objects are only partially visible initially but are still correctly modeled in the RGB space—for instance, the human hand in the ‘tool’ example at 3’42’’. We would be happy to include additional visual results in the revision if needed."}, {"review_id": "GrXy2zwKbt", "reviewer": "Reviewer_Df51", "summary": "The paper proposes a video generation framework that incorporates 3D geometry and dynamic information into 2D videos. It introduces PointVid, a 3D-aware dataset created by augmenting videos with 3D point trajectories aligned in pixel space. By fine-tuning a latent diffusion model on this dataset, the method enables tracking of objects in 3D space and regularizes their shape and motion to reduce artifacts. This leads to improved visual quality and better handling of complex, contact-rich scenarios. The approach is compatible with existing video diffusion models to enhance their realism.", "questions": "Please check above weaknesses.", "limitations": "The authors addressed the limitations in main manuscript.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n- The paper is clearly written and easy to follow. The authors effectively communicate their motivation and methodology in a straightforward manner. Additionally, the figures are clean and illustrative, contributing positively to the paper’s readability.\n\n**Weaknesses**\n\n- As acknowledged in the Limitations section, the proposed method inherits constraints from the base diffusion models. To better support the claimed generality of the plug-and-play design, it would be beneficial to include additional experiments with a broader set of base models. Using only I2VGen-XL and CogVideoX 1.5 is insufficient to convincingly demonstrate general effectiveness.\n\n- The use of a VAE to encode latent features of the point modality raises concerns, as the VAE was not specifically trained on point-based data. It is unclear whether point regularization alone can adequately address this mismatch.\n\n- I am concerned that the second and third terms in the reconstruction loss (Equation 1 of the main manuscript) may introduce a trade-off—such as improved temporal consistency at the cost of reduced temporal dynamics. It would be helpful if the authors could address this potential issue.\n\n- The paper would benefit from a clearer description of the generated 3D trajectories. Are there any qualitative visualizations or quantitative metrics to evaluate the generated 3D points?", "was_revised": true, "final_justification": "Most of the concerns I raised have been satisfactorily addressed during the rebuttal. After reviewing the feedback from other reviewers, I have decided to change my score from “borderline reject” to “accept.”", "rebuttal": "Thank you for your constructive feedback. Below, we address your concerns:\n\n**Q1: Using only I2VGen-XL and CogVideoX 1.5 is insufficient to convincingly demonstrate general effectiveness.** \n\nWe chose I2VGen-XL and CogVideoX 1.5 as representative models due to their distinct underlying architectures, which are based on UNet and Diffusion Transformer (DiT) frameworks, respectively. These two architectures are foundational, as most current video diffusion models share a similar backbone structure with one of them. We acknowledge that including experiments with additional baseline models would further strengthen our paper. However, due to constraints on computational resources, we leave this for future exploration.\n\n**Q2: VAE was not specifically trained on point-based data.** \n\nUsing a VAE trained on RGB video to encode other modalities has been shown to be effective in prior work. For example, [1] uses a pretrained RGB VAE to encode depth maps without fine-tuning. Similarly, [2] leverages the RGB VAE to encode both depth and normal maps as input.\n\nDuring our training, we transform the point modality to the same range as color intensities ([-1, 1]) and reshape it to match the input format of the video. We observed that even without fine-tuning, the VAE is capable of reconstructing the main shape and motion of point clouds, while the point regularization further reduces noise (as shown in Figure 4 of our main manuscript). Additionally, we report that the mean squared error (MSE) of the encoded-decoded point tensor is less than 0.01, comparable to that of the reconstructed video tensor. Further statistics on point augmentation and regularization are provided in Q4.\n\n**Q3: The second and third terms in the reconstruction loss may introduce a trade-off.** \n\nIn our design of the reconstruction loss in Equation 1, the first term (position) serves as the primary component for controlling macroscopic motion dynamics. The second and third terms (velocity and acceleration) act as complementary losses (with small weights) to suppress high-frequency, microscopic noise. While we acknowledge that incorporating motion smoothness losses introduces a trade-off, the global dynamics are primarily governed by the position term, and we observe faithful reconstruction of ground-truth dynamics during training. Moreover, discouraging abrupt changes in velocity is often beneficial, as such changes can lead to artifacts in the generated videos. For example, in the ‘race car’ case (5’00’’ in the supplementary video), highly dynamic baselines exhibit non-physical morphing, whereas our result demonstrates physically plausible motion while preserving a high degree of dynamism.\n\n**Q4: Evaluation of the generated 3D points** \n\nWe report the MSE of predicted 3D point trajectories in diffusion (using the input point tracking tensor as ground truth) as a quantitative metric, summarized below:\n\n| Item | Untrained | Trained without Regularization | With Regularization |\n|----------|----------|----------|----------|\n| MSE | 0.21252 | 0.00310 | 0.00205 |\n\nWe observe that our network design effectively predicts 3D point trajectories after the augmentation stage and further reduces the error through the regularization stage.\n\n[1] Shao J, Yang Y, Zhou H, et al. Learning temporally consistent video depth from video diffusion priors[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 22841-22852. \n\n[2] Liu X, Ren J, Siarohin A, et al. Hyperhuman: Hyper-realistic human generation with latent structural diffusion[J]. arXiv preprint arXiv:2310.08579, 2023."}, {"review_id": "umlJJynajV", "reviewer": "Reviewer_cz8h", "summary": "The paper tackles the task of image-to-video generation, intending to inject explicit 3D understanding into diffusion-based video models. Starting from a pre-trained video generator, the authors propose to use a new modality: a three-channel map encoding of foreground objects' 3D points as pixel coordinates (u,v) and depth d. These point trajectories are obtained with an existing 3D point-tracking model, creating a dedicated training dataset PointVid. The diffusion model is then fine-tuned to predict both RGB frames and the (u,v,d) maps simultaneously. During the next training stage, the predicted point maps serve as an additional regulariser, guiding the network toward better 3D understanding and yielding stronger temporal and object consistency in the generated videos.", "questions": "1. Can you quantify the extra compute cost your regularizer adds compared to the standard diffusion loss? How does that overhead grow as you increase point density?\n2. Have the authors conducted a systematic analysis of failure modes? In the supplementary video, the method exhibits improved temporal coherence but reduced motion amplitude. To what extent might the regularizer inadvertently suppress desirable motion dynamics, and how might this be mitigated?\n3. Why choose a custom image subset over the public VBench split for evaluation? Could that decision skew your results or limit generality?\n4. The technique seems readily applicable to text-to-video generation. Have you tested this scenario, and if so, can you share qualitative or quantitative results? \n5. How does the number of tracked points affect output quality? Can you achieve similar results with a much sparser point cloud, and what trade-offs arise?", "limitations": "Partially. \nThe manuscript lacks a failure-mode analysis, does not report the runtime or memory overhead introduced by the regulariser, and relies on a private evaluation set, preventing comparison with other improvement methods. Including these elements would substantially strengthen the paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Clarity of presentation. The paper is well-written, with clear explanations and informative visualisations that make the method easy to follow.\n\n2. Novel 3D regularisation. Introducing an implicit 3D point-map modality and using it as a regulariser is a creative way to enhance geometric consistency in video diffusion models.\n\n3. Compelling qualitative results. The method produces visually convincing videos that exhibit improved temporal and object consistency compared with baseline models.\n\nWeaknesses:\n\n1. Custom evaluation set. The main quantitative study uses a hand-curated subset of images, making it difficult to compare with existing or future work. Public benchmark suites such as VBench already support image-to-video evaluation and should be used for head-to-head comparisons.\n\n2. Insufficient dataset analysis. The paper does not examine failure modes of the SpaTracker point-tracking stage or analyse tracking quality in the generated videos. A brief error study would strengthen the empirical section.\n\n3. Missing cost breakdown. The additional regularisation loss incurs extra computation, but the paper does not report its runtime or memory overhead relative to the standard diffusion loss. \n\n4. Limited evaluation scope. Results are presented only in the image-to-video scenario, even though the technique appears directly applicable to text-to-video generation. Demonstrating this extension would broaden the paper’s impact.", "was_revised": true, "final_justification": "I remain positive about the paper and am keeping my original (accept) score, as the core idea is solid and the rebuttal clarified several technical points. However, two of my earlier concerns are still only partially addressed in the manuscript itself:\n\n1. Evaluation split. The paper relies on a curated test subset. While the authors supplied numbers of other dataset in rebuttal, those results (and a short justification for using both splits) need to appear in the main paper so future work can compare directly.\n\n2. Motion-regularisation trade-off. The supplementary video still shows reduced motion amplitude when the regulariser dominates. A brief analysis of this limitation, plus guidance on loss scheduling (e.g. how often the regulariser is applied, typical weight ratios), would help readers understand the quality-versus-motion balance and reproduce the method.\n\nThese points do not change my overall assessment, but incorporating them in the camera-ready would align the claims with the evidence and strengthen the final version.", "rebuttal": "Thank you for your constructive feedback. We appreciate your recognition of the novelty and effectiveness of our method. Below, we address your questions and concerns:\n\n**Q1 Can you quantify the extra compute cost your regularizer adds compared to the standard diffusion loss? How does that overhead grow as you increase point density?** \n\nWe report the additional computational cost of our UNet model, trained on an A100 GPU with a batch size of 4 as follows: our 3D incorporation and regularization losses increase training times (∼9s per iteration, compared to ∼4.75s when fine-tuning on RGB video only) while maintaining comparable GPU memory usage (less than 2GB on top of ∼45GB). This additional computational cost is acceptable, as we are fine-tuning a pretrained video model rather than training one from scratch. Further statistics can be provided if needed.\nIn our current design, the increased computational cost is primarily driven by the resolution of the training video rather than the density of the tracked points (see Q5 for more details). \n\n**Q2 To what extent might the regularizer inadvertently suppress desirable motion dynamics, and how might this be mitigated?** \n\nIn our ablation studies (Figure 6 of the manuscript), we demonstrate that training with RGB alone lacks 3D awareness and results in failure cases, while using only the regularization loss leads to severe degradation of the RGB modality. In our approach, we adopt a joint optimization of diffusion and regularization losses, appropriately balancing them to be of comparable magnitude. This strategy yields a consistent decrease in regularization losses and a noticeable reduction in non-physical artifacts.\n\nWe observe that when the regularization loss significantly outweighs the diffusion loss (by orders of magnitude), it tends to suppress motion. To address this, we apply the regularization loss less frequently, for example, once every five iterations. This adjustment helps preserve larger motion dynamics while requiring more training steps.\n\n**Q3 Why choose a custom image subset over the public VBench split for evaluation?** \n\nWe use a custom test set for VBench to ensure consistency across all evaluation metrics, including generic video quality (using VBench), physical common sense (using VideoPhy), ablation studies on model design, and user studies (reported in the supplementary material). The test images are sampled from a single batch of video clips (unseen by our model) from video dataset [1], with the initial frame randomly selected within each clip.\n\nBelow, we report the VBench results of our UNet-based model using their provided image set for your reference:\n\n| Item | SC | BC | MS | AQ | IQ |\n|----------|----------|----------|----------|----------|----------|\n| Baseline | 0.89703 | 0.94009 | 0.96993 |  0.52809 |  0.62961 |\n| Ours | 0.96418 | 0.96722 | 0.98319 | 0.51754 | 0.65428 |\n\n**Q4 The technique seems readily applicable to text-to-video generation. Have you tested this scenario?** \n\nWe chose image-to-video tasks for a more controlled comparison of physical correctness, as the same objects are present in the scene. While our method is adaptable to text-to-video models, we did not evaluate them, as this is not our main focus. We look forward to exploring this direction in the future. As an alternative, text-to-video applications could be demonstrated by incorporating an additional text-to-image step prior to our model, and we are happy to include such examples if requested.\n\n**Q5 How does the number of tracked points affect output quality? Can you achieve similar results with a much sparser point cloud, and what trade-offs arise?**\n\nFirst, we would like to clarify that although we initially obtain point trajectories through sparse 3D tracking, during post-processing, we fill in all untracked foreground pixels (i.e., pixels inside the foreground mask) by interpolating from neighboring tracked points. Furthermore, they are stored as a tensor of the same dimension of the video and used for subsequent computations.  As a result, the point modality used in diffusion effectively becomes ‘dense’ tracking from the perspective of the video model. Thus, during training, the number of tracked points depends on how much of the reference frame is occupied by foreground objects, rather than on the tracking resolution used in data generation. On the other hand, during inference, we do not provide the point cloud, so there is no explicit control over point resolution.\n\nCurrently, our dataset is generated using a fixed sparsity setting in SpaTracker [2]. We acknowledge that the tracking sparsity used during data generation affects accuracy of 3D information, and likely affects our training, we did not further explore this as this requires costly regeneration of large-scale dataset; we leave further investigation of this aspect for future work.\n\n**Q6 The paper does not examine failure modes of the SpaTracker point-tracking stage or analyse tracking quality in the generated videos.** \n\nSince we do not prompt the model with point tracking during the inference stage, we are unable to directly assess whether generation failures are caused by inaccurate tracking. To evaluate the quality of predicted point tracking during training, we report straightforward MSE loss measurements between the generated points and the ground truth:\n\n| Item | Untrained | Trained without Regularization | With Regularization|\n|----------|----------|----------|----------|\n| MSE | 0.21252 | 0.00310 | 0.00205 |\n\n[1] Chen T S, Siarohin A, Menapace W, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 13320-13331.\n\n[2] Xiao Y, Wang Q, Zhang S, et al. Spatialtracker: Tracking any 2d pixels in 3d space[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20406-20417."}, {"review_id": "qP8nOwdotu", "reviewer": "Reviewer_PwUi", "summary": "This paper proposes injecting 3D point trajectories into video diffusion models to improve physical understanding. The authors introduce PointVid, a dataset with pixel-aligned 3D tracks, which is used to train models with point-based regularization to encourage smooth motion and shape consistency. The method improves video realism and can be applied to most existing diffusion models.", "questions": "1. Can the rigid loss be applied directly to object pixels in the RGB space instead of 3D point clouds? Would such an alternative achieve similar structure-preserving effects? An ablation would help clarify this.\n\n2. Your framework relies on the first frame for segmentation and point tracking. How does the method generalize to longer or more complex videos with multiple moving objects or occlusions?", "limitations": "Yes", "rating": 2, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1. The paper addresses an important limitation of current video diffusion models: lack of 3D physical understanding, through a simple yet effective solution.\n\n2. The proposed framework is model-agnostic and can be integrated into most existing video diffusion architecture with minimal changes.\n\nWeaknesses:\n\n1. The rigid loss introduced for point clouds appears critical to preserving physical structure. However, it's unclear whether similar effects could be achieved by applying it directly in the RGB space. An ablation comparing point regularization to RGB-space alternatives would strengthen the paper.\n\n2. The framework relies on the first frame for reference and sparse point tracking, which limits its scalability to long videos, multi-object scenes, or cases with occlusion and fast motion. The method is mainly demonstrated on short, single-object scenarios.", "was_revised": true, "final_justification": "I thank the authors for their response and efforts. However, I still find the work insufficiently supported in terms of robustness and generalizability. The provided video results are only around 2 seconds in length, which is not adequate to demonstrate consistent performance in complex, multi-object, or long-term scenarios. Without experiments on longer sequences and corresponding evaluations, it is difficult to validate the method’s effectiveness beyond short clips. Therefore, I do not recommend acceptance.", "rebuttal": "Thank you for your constructive feedback. We appreciate your recognition of the contribution of our work. Below, we address your questions and concerns:\n\n**Q1 Can the rigid loss be applied directly to object pixels in the RGB space instead of 3D point clouds?** \n\nWe propose the use of point trackings due to the lack of 3D spatial information in RGB space. Directly applying rigidity losses in RGB space is suboptimal. For instance, in a video of a race car moving from a distance toward the camera, the object’s size in the RGB frame increases from a small region of pixels to a large portion of the image. Applying rigidity loss directly to these pixel regions fails to preserve the object’s shape and may even cause undesirable shrinkage. To address this, we introduce 3D point tracking as a prior to overcome this limitation. \n\nOur ablation study in Figure 6(i) of the manuscript partially illustrates this issue, showing that training solely on RGB pixel differences does not achieve the desired 3D consistency. We agree that applying similar physical constraints directly in RGB space is a promising direction, provided that alternative methods can be developed to extract 3D information from RGB videos efficiently. We look forward to exploring such approaches in future work.\n\n**Q2 How does the method generalize to longer or more complex videos with multiple moving objects or occlusions?** \n\nThe length of the generated video is constrained by the clip lengths in the training data. It can be generalized to predicting longer videos by scaling up the dataset. In our current training pipeline, we track foreground objects in the first frame and use point tracking to determine their 3D positions across time, accounting for possible interactions and occlusions between them. This added point modality enables our model to resolve multiple moving objects (see ‘Aston Martin’ example at 2’06’’ of the supplementary video) and handle contact-rich, multi-object scenarios with occlusion (see our task-oriented video examples). Even when an object is only partially visible or absent in the first frame, our model can still effectively handle it using the RGB modality. (For example, in the ‘tennis player’ case at 4’37’’, the player’s arm is not fully visible in the first frame, yet our model correctly infers its motion.)\n\nTo extend our approach to longer videos, more complex scenes and object interactions, we plan to scale up the training data with richer scene compositions and more accurate, higher-resolution 3D tracking."}]

Author rebuttal:

Meta-review: This paper presents a method for video generation based on incorporating 3D point tracks as a prior. The idea is: given a pre-trained video generator, add 3 new channels to encode the foreground objects' 3D point coordinates (UVD), and then learn to predict the RGB together with the UVD map, and then also use the point maps as a space for applying regularizations. The overall effect is to reduce the well-known "morphing" artifacts of current video generators The paper also introduces a new pseudo-labelled dataset (corresponding to the zeroth step of the pipeline, where point tracks need to be pre-computed for some training videos).  This paper received mixed reviews: 5, 5, 5, 2. The reviewers largely find the approach well motivated, and the paper well-written. One review is confidently negative, pinpointing that the method is only demonstrated on two-second clips, which is perhaps too short to support claims about alleviating issues to do with shape consistency. The positive reviews contain their caveats as well, pointing out for example that the trade-offs of the motion regularizer (and details of how it should be applied in practice) are missing from the current draft, making it potentially difficult to reproduce the work. The AC also notes that while the paper heavily relies on the recent advancements in point tracking, and is one of many papers that are finding point trackers useful for video generation, the authors only cite the point tracker that they happen to use (SpaTracker); some advancements in point tracking, and video generative methods that rely on point trackers (e.g., Track4Gen, VideoJAM), should be acknowledged and ideally compared against, to better situate the work (and perhaps justify 3D point tracking instead of 2D as those works used). Overall, the AC sides with the majority positive reviews and recommends that the paper be accepted. The authors are encouraged to carefully revise the work to incorporate feedback gathered during the review process, in addition to the work already done during the rebuttal.
Decision: Accept (poster)