Title: VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Video Diffusion Models
Abstract: Recent advancements in text-to-video (T2V) diffusion models have enabled high-fidelity and realistic video synthesis. However, current T2V models often struggle to generate physically plausible content due to their limited inherent ability to accurately understand physics. We found that while the representations within T2V models possess some capacity for physics understanding, they lag significantly behind those from recent video self-supervised learning methods. To this end, we propose a novel framework called {VideoREPA}, which distills physics understanding capability from video understanding foundation models into T2V models by aligning token-level relations. This closes the physics understanding gap and enables more physics-plausible generation. Specifically, we introduce the {Token Relation Distillation (TRD) loss}, leveraging spatio-temporal alignment to provide soft guidance suitable for finetuning powerful pre-trained T2V models—a critical departure from prior representation alignment (REPA) methods. To our knowledge, VideoREPA is the first REPA method designed for finetuning T2V models and specifically for injecting physical knowledge. Empirical evaluations show that VideoREPA substantially enhances the physics commonsense of baseline method, CogVideoX, achieving significant improvement on relevant benchmarks and demonstrating a strong capacity for generating videos consistent with intuitive physics. Code and more video results are available at https://videorepa.github.io/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on ensuring that text-to-video (T2V) models generate physically plausible content. Authors identify that the main limitation of current models is that the representations within such models have reduced capacity for physics understanding in contrast to self-supervised methods that focus on representational learning. Thus, a method to distill physics understanding from such foundational models into T2V models. This method enables fine-tuning of T2V to inject physical knowledge. The authors perform experiments comparing their results with CogVideoX.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n* The core idea proposed by the authors is interesting and sound. Authors propose to leverage the representation capabilities of foundational video models, e.g., DinoV2 or VideoMAE2 to align the representations used by text-to-video latent diffusion models and thus capture more realistic and consistent dynamics (e.g., consistent object shapes and physical plausibility of how objects move and interact with other objects). At the same time the authors devise a method to achieve this alignment by enforcing relation similarity between token embeddings. Thus the alignment is not done directly over the representations but over their relations. This is a softer condition useful for fine-tuning as opposed to REPA. \n* Qualitative results look good and the authors selected a set of videos that clearly show the difference between the baselines and the proposed method results. Quantitative evaluation is aligned with these results. \n* In general the presentation is good (but writing could be improved). \n\n# Weaknesses\n\nIn order of importance:\n\n**Vague Link Between SSL Encoders and Physics**\nThe method claims to “enhance physics” by distilling knowledge from self-supervised video encoders. However, SSL encoders (e.g., DINOv2, or VideoMAE2) are typically trained to capture general motion or appearance representations — not necessarily physical plausibility or constraints like dynamics, contact, or force interactions.\n\nWhat evidence is there that SSL video encoders actually encode meaningful physics properties (e.g., Newtonian constraints, conservation laws, contact dynamics) rather than just correlation patterns or appearance flow? Also, how do the authors ensure that the distilled relations truly correspond to physically plausible relations rather than dataset-specific visual correlations?\n\n**Ambiguity in ‘Pairwise Token Similarity’ = Physics**\nThe logic assumes that aligning pairwise token similarities (via TRD loss) transfers physics. But pairwise similarity can capture texture, background motion, or trivial frame correlations — it doesn’t inherently encode contact constraints, collision avoidance, or kinematic feasibility. Why should pairwise token similarities enforce physically realistic behavior? Could this encourage spurious correlations instead of actual dynamic consistency?\n\n**Writing and Clarity**\n\nThe writing could be improved to enhance readability and ensure a smooth flow of information and transitions between ideas. For example, Lines 40–41: *“Yet, WISA struggles to generalize to open-domain data, showing improvements primarily when trained on videos with explicit physics.”* This sentence is confusing and makes the paper more difficult to follow. Similar unclear phrasing can be found throughout the manuscript. Clarifying these statements would significantly improve the overall quality and accessibility of the paper.\n\nQUESTIONS:\nQuestions were included in the previous section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nVideoREPA introduces an approach to enhance the physical plausibility of videos generated by T2V diffusion models. The method leverages an improved version of REPA by distilling physics knowledge from video foundation models into video diffusion models, focusing on finetuning rather than training from scratch.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The paper proposes a novel way to adapt REPA for finetuning, shifting its original scope from training acceleration to knowledge transfer.\n- Experimental results and video supplements demonstrate that the physical plausibility of generated videos improves using the proposed method.\n- The method effectively addresses the potential issue of feature dimension mismatch between VFMs and VDMs through interpolation.\n\n### Weakness\n- It remains unclear why the **physical knowledge** of VFMs should be distilled into VDMs as proposed. The manuscript gives the impression that enhanced physical plausibility is merely a training byproduct, not an explicit goal. The justification for why the TRD loss contributes to physics understanding needs to be clarified.\n- The experiments are limited to CogVideoX-2B and 5B, which share the same structure and are now relatively outdated. Demonstrating the method's integration with more recent and diverse open-source video generation models would strengthen its credibility.\n- A broader quantitative comparison of video fidelity, consistency, and text-video alignment (e.g. using VBench) is lacking. Without this, it is unclear whether the improvement in physical plausibility compromises other aspects of video quality.\n\nQUESTIONS:\n- In Figure 1 (right), the evaluation of physics understanding extracts intermediate features from VDMs to predict video futures (according to Appendix B). This approach raises concerns because VDMs are trained for denoising, not for extracting rich features like MAEs. My questions (or concerns) are:\n1. Is it fair to compare **noisy activations** from VDMs with the structure features of MAEs?\n2. Can you claim that large VDMs understand physics worse than smaller VFMs based on this unfair comparison (L47-50)? \n3. Doesn't the enhanced alignment is a trivial results by directly aligning the feature with MAE, rather than the results of truly enhanced physics understanding?\n- Why does the spatial component of the TRD loss not exclude the case $i=j$, while the temporal component does exclude $d=e$? Please clarify / justify this design choice.\n- The quantitative results show VideoREPA-5B outperforming HunyuanVideo, yet visually the latter appears better in quality. Why was HunyuanVideo not used as the baseline model?\n\nLIMITATIONS:\nThe authors addressed the limitations of their work.\n\nCommon comment for the authors: If I couldn't fully understand your methods or am underestimating your contributions, feel free to explain and let's discuss. I will gladly raise the scores if I have misunderstood key points or if my concerns are addressed well.\n\nI think the most critical points are:\n- The first weakness described above.\n- Validity of the comparison on Figure1 right (My first question).\n- A broader quantitative comparison of video fidelity, consistency, and text-video alignment", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposed a training strategy to advance the original video generation model with better physical consistency. It proposed an REPA learning strategy which aligns the feature from MM-DiT with the feature from a pretrained video model (e.g. VideoMAE).
VideoREPA is applied onto CogVideoX model and evaluated on two datasets: VideoPhy and VideoPhy2. On both semantic consistency and physical commonsense metrics, the proposed method performs better than original model.

STRENGTHS AND WEAKNESSES:
Strengths
1. The proposed method shows better performance on both VideoPhy and VideoPhy2 datasets

Weakness
1. The performance gain from VideoREPA compared to original CogVideoX is very marginal, especially in the performance on SA (semantic adherence)

2. from the qualitative results, it seems the results from VideoREPA has more regular-shaped objects compared to original CogVideoX model, is this the reason why the physical related metrics could be improved? Because the objects in the generated videos have simpler shapes, leading to better physically interaction results

3. There is no physical performance evaluation on VideoMAE or VJEPA to validate why the features from MMDiT should be aligned to VideoMAE features.

4. The ablation study is not very comprehensive, e.g. it didn't explore different VFM but only uses VideoMAE. And it also didn't explore different alignment learning strategy but only tried REPA

QUESTIONS:
Please see the weakness part

LIMITATIONS:
1. VideoREPA is a general technique which is compatible with all video generation foundation model. But the performance is only reported on CogVideoX model. It didn't show any generalization capability to other base models.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes __VideoREPA__, a video diffusion model finetuning method which introduces physics knowledge into diffusion-based pretrained video generation models, where, instead of the token-level features, the relations between token features are aligned with those of a self-supervised video understanding foundation model using the RPEA loss. The proposed TRD loss aligns the pair-wise spatial and temporal token similarities. The experiments are conducted on two different datasets and the results demonstrates that the proposed VideoREPA is better at generating physical-commonsense video contents wrt the physical plausibility metrics.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n1. the motivation is clear, where this work is motivated by the REPA in T2I methods and the information gap is analysed between video diffusion models and the self-supervised pretrained models.\n2. the proposed solution takes the finetuning over pretrained VDM models into consideration, instead of directly use the original REPA loss, this work shifts it into spatial-temporal soft alignment. Note that the original REPA loss is introduced for accelerating training convergence.\n3. the experimental results demonstrates empirical gains on two different datasets wrt the quantitative metrics.\n4. the appendix is comprehensive, including implementation details and refinement strategy.\n\n\n__Weakness__\n1. This work highlights the improvement about the physical-commonsense, which is majorly validated with predefined metrics, however, considering the complexity of real world motions, these metrics may not accurately measure the plausibility of the generated motions. Detailed user study is required to demonstrated VideoREPA actually learns physical-commonsense.\n2. The impacts of different VFMs seem significant (Table 4), is there any possibility to take use multiple VFMs to complement each other.\n2. Are the shape of inputs of two models the same, considering the patch size of them may be different?\n\nQUESTIONS:\nThe questions have been summarized in the weakness part.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}