VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models
Authors: Xiangdong Zhang
Jiaqi Liao
Shaofeng Zhang
Fanqing Meng
Xiangpeng Wan
Junchi Yan
Yu Cheng
Venue: NeurIPS 2025 poster ()
Keywords: Video Generation
Video Diffusion Models
Primary area: deep_learning

Abstract: Recent advancements in text-to-video (T2V) diffusion models have enabled high-fidelity and realistic video synthesis. However, current T2V models often struggle to generate physically plausible content due to their limited inherent ability to accurately understand physics. We found that while the representations within T2V models possess some capacity for physics understanding, they lag significantly behind those from recent video self-supervised learning methods. To this end, we propose a novel framework called {VideoREPA}, which distills physics understanding capability from video understanding foundation models into T2V models by aligning token-level relations. This closes the physics understanding gap and enables more physics-plausible generation. Specifically, we introduce the {Token Relation Distillation (TRD) loss}, leveraging spatio-temporal alignment to provide soft guidance suitable for finetuning powerful pre-trained T2V models—a critical departure from prior representation alignment (REPA) methods. To our knowledge, VideoREPA is the first REPA method designed for finetuning T2V models and specifically for injecting physical knowledge. Empirical evaluations show that VideoREPA substantially enhances the physics commonsense of baseline method, CogVideoX, achieving significant improvement on relevant benchmarks and demonstrating a strong capacity for generating videos consistent with intuitive physics. Code and more video results are available at https://videorepa.github.io/.

VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models
Xiangdong Zhang" Jiaqi Liao, Shaofeng Zhang', Fanqing Meng', Xiangpeng Wan', Junchi Yan', Yu Cheng School of Al & School of CS, Shanghai Jiao Tong University NetMind.AI, The Chinese University of Hong Kong
Abstract
Recent advancements in text-to-vided (T2V) diffusion models have enabled high- fidelity and realistic video symthesis. However, current T2V models often struggle generate physically plausible content due to their limited inherent ability accurately understand physics. We found that while the representations within T2V models possess some capacity for physics understanding, they lag signifi- cantly behind those from recent video self-supervised learning methods. To this end, we propose a novel framework called VideoREPA, which distills physics understanding capability from video understanding foundation models into T2V models by aligning token-level relations. This closes the physics understanding gap and enables more chysics-plausible generation. Specifically, we introduce the Token Relation Distillation (TRD) loss, leveraging spatio-temporal alignment provide soft guidance suitable for finetuning powerful pre-trained T2V models-a critical departure from prior representation alignment (REPA) methods. To our knowledge, VideoREPA is the first REPA method designed for finetuning T2V models and specifically for injecting physical knowledge. Empirical evaluations show that VideoREPA substantially enhances the physics commonsense of baseline method, Cog VideoX, achieving significant improvement on relevant benchmarks and demonstrating strong capacity for generating videos consistent with intuitive physics. Code and more video results are available at Project Page
CegVidooX
OagVatenX REPA
Copvitooo
VideoREEA (ours)
Phyyii undessiningg ewilarin Pbysion (cunton sabuct).
pinz carking
tyn daizing pancaize
CagVidouX
Capeoriit
vineeEES
CagVidooX
VideoREPA
(b Prysic undkasneding cnbnstteinngggaaaatiio Plysin (olll scbsct).
cvaluttion
Glaw demtre Foor.
rime and condees brighly onlone thsben.
Figure Left: Visual comparison of video generation results from CogVideoX (56] (baseline),
Cog VideoX finetuned with REPA [58], and our proposed VideoREPA. Red rectangles denote phe- nomena that violate physical commonsense for easier distinguish. Our VideoREPA generates videos that most closely adhere to real-world physical laws. Right: Evaluation of physics understanding on the Object Contact Prediction (OCP) task within the Physion benchmark ( The plots illustrate significant gap in physics understanding between the SSL video encoder VideoMAEx>> and the T2V model Cog VideoX. The proposed VideoREPA substantially narrows this understanding gap.
! Interns at Shanghai Al Laboratory. 'Corresponding author. chengyu&cse. cuhk. edu hk.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Vidco diffusion models (VDMs) have recently gained significant attention, demonstrating remarkable
advancements [1 45 24 56 31 41] These generative models increasingly applied in diverse
domains, including movie-level video generation [51], animation [49] and advertising 12]. However, critical challenge remains: the physical plausibility (e.g., shape regularity and motion rationality)
of videos generated by even state-of-the VDMs is often severely limited [4] 33. While existing
VDMs (e.g.. VideoCrafter2 (0] CogVideoX (56). Hunyuan [24], Cosmos [1.. Wan (45) have
shown improvements in physics capabilities, typically achieved by scaling training data, refining model architectures, or collecting higher-quality video-text pairs A. these strategies have inherent drawbacks. The substantial expense of scaling datasets and the limited focus of current architectural designs on explicit physies modeling make significant advancements in physically plausible video generation through these avenues difficult and costly.
Current methods for enhancing the physical plausibility of generated videos can be divided into two main categories: simulation-based approaches (29) (27 (33 52, 28, 60) and non-simulation-bant based approaches [46] 10 [55] Simulation-baase methods, which have seen significant development, typically integrate external physics simulators for guidance or direct generation. However, their effectiveness is constrained by the complexity of simulations and the challenge of modeling diverse open-domain phenomena, limiting their potential for creating powerful, general-purpose generative models. con-simulattionbuan baseeddn--dddddddee methods, on the other hand, have received less attention. For example, WISA [46] decomposes textual descriptions into physical phenomena and employs Mixture-of- Physical-Expert Attention. Yet. WISA only show improvement when training with dedicated collected dataset WISA-32K [46] where each relates to individual explicit physical phenomena. It struggles to generalize to open-domain data which much easily to collected and scale. To this end, this paper explores enhancing the physics-plausible video generation of T2V models using non-simulation strategies on open-domain datasets, aiming to achieve robust generalization and broad applicability.
Regarding generating physical coherence videos, it is acknowledged that in generative modeling, improved understanding often benefits generation quality 122 54] 16 23 26]. DynamiCrafter [54]. for example, exemplifies this by using enhanced visual encoding to improve its outputs. However. our evaluations on physics understanding benchmark Physion [6] (illustrated in Figure I reveal that the text-to-video diffusion model VideoX (2B) exhibits poor physics understanding ability, performing significantly worse than the much smaller self-supervised video understanding model, VideoMAEv2 (86M). This notable gap in physics comprehension motivates our core strategy: to improve the physics understanding of VDMs by transferring inherent physics knowledge from ViFMs, and thereby enhance physically coherent video generation.
Recent work has explored bridging the gap between foundation models and generative diffusion models, notably through Representation Alignment (REPA) [58] (2,42) which enhances semanties in image diffusion models via feature alignment. Inspired by REPA, we investigate an approach to improve physical plausibility in video generation by aligning with ViFMs. However, directly applying REPA techniques to inject physics knowledge into text-to-vider models proves infeasible due to several critical distinctions: First, the spatial focus of REPA is insufficient for crucial temporal dynamics in videos. Second, REPA targets from- nom-scratch training acceleration, not knowledge transfer via finetuning pre- pre-trained models. Third, its hard alignment mechanism can destabilize pre- trained VDMs during finetuning. Finally, VDM temporal latent compression adds further alignment complexity. Experiments in Figure Taaree[ssuppr support this, showing that finetuning with REPA degrades the performance of Cog VideoX significantly. See detailed discussions in Section 3.3
To address these challenges and enhance physics in video generation by deepening physics under- standing, we introduce VideoREPA. This method distills token-level relations, capturing dynamics from Video Foundation Models (ViFMs) and transferring them to VDMs, thereby improving the physical realism of generated videos. Our method performs well on open-domain video dataset Vpe [35] without relying on the physies explicit dataset WISA-32K [46] which is much harder to collect. Specifically, we propose a Token Relation Distillation (TRD) loss that distills intra- frame spatial relations and fnter-frame temporal dynamics from ViFM representations into VDMs via relational alignment, which closes the physics understanding gap as shown in Figure D Unlike standard REPA, our TRD loss employs a more moderate alignment mechanism tailored to overcome difficulties associated with fine-tuning. We argue that the physical plausibility of a video depends

---

Deffuson Loss
Taken Relation Disiilation Loss
VALU EIRCOVA
MIM-DNT Black
MEM-IRI lllckk
cores durarrood byandyyann chomiis..
MES-INT Block
Spatak-Temoon Takerin Ai gremart
Pre-trained
Video Encoders
+C Tranas
Figure 2: Overview of VideoREPA. Our VideoREPA enhances physics in T2V models by distilling physics knowledge from pre-trained SSL video encoders. We apply Token Relation Distillation (TRD) loss to align pairwise token similarities between video SSL representations and intermediate features in diffusion transformer blocks. Within each representation, tokens form spatial relations with other tokens in the same latent frame and temporal relations with tokens in other latent frames.
not only on the regular shape of objects (spatial dimension) but critically on the motion of subjects (temporal dimension); thus, focus of TRD extends beyond merely spatial alignment capture these crucial temporal aspects. Our main contributions can be summarized as follows:
1) We identify essential gap in physics understanding between self-supervised ViFMs and T2V models. We then propose VidcoREPA, the first method to bridge video understanding models and T2V models, which closes understanding gaps and achieves more physically plausible generation.
2) We introduce VideoREPA, a novel feature alignment framework for video generation. utilizes Token Relation Distillation loss to effectively distill physics knowledge from ViFMs through token-relational alignment, enabling VDMs to generate videos that better adhere to physical laws.
3) The proposed TRD loss overcomes key limitations (detailed in Section 3.3 of directly applying REPA 588 to the video generation domain, particularly for finetuning pre-traines models and capturing essential temporal dynamics.
4) Quantitative and qualitative experiments show the superiority of our VideoREPA over base- lines like Cog VideoX and other methods like WISA VideoREPA achieves state-of-the-art Physical Commonsense (PC) score of 40.1 on VideoPhy (24.1% improvement over its Cog VideoX baseline) and significant physics enhancements on the challenging VideoPhy2 benchmark. Visualizations also confirm VideoREPA generates videos more consistent with physical laws than VideeX.
2 Related works
Self-supervieed learning for physics understanding. Understanding physical interactions (e.g.. predicting object trajectories [17 movements (29) vital for applications like robotics [T] and autonomous driving (38). Self-supervised learning (SSL), as powerful tool for wide range of understanding tasks including classification, segmentation, and detection [37 & 19 21 leverages pretext tasks to pre-train models on large amounts of unlabeled data, thereby enhancing understanding ability and yielding the powerful pre-trained models often called foundation models. Recent studies [15 44 highlight the potent physics understanding capabilities of Video Foundation Models, such as VidcoMAEv2 [47]. and V-JEPA [5], demonstrated through strong performance on physics benchmarks like Physion (6). Notably, these specialized video models can outperform even large Multimodal Language Models like GPT4-V [36] on physics reasoning tasks. Building on the principle that strong understanding facilitates better generation [54]. we investigate how to leverage the physics knowledge captured by video SSL models to enhance the physical realism of T2V generation.
Physics plausible video generation. While initial T2V model improved visual fidelity, motion, and
realism using scaled data and advanced architectures [56] 45 31 59 recent studies 34
32
highlight major problem: physical plausibility remains poor even in state-of-the-art (SOTA) models. This realization has driven emerging research towards physics-aware generation, with
many new methods proposed [46 10
27
28
52
55
60
29
33 Some techniques, exemplified by

---

PhysAnimator [52], PhysGen [29] and MotionCraft [33 these simulation-dependent methods are inherently limited by the simulator's scope and accuracy, making them less suitable for complex real-world scenarios. PhyT2V prompts through multiple rounds of generation and reasoning. maximizing the physies potential in models but not adding new inherent knowledge. WISA [46] decomposes textual descriptions into physical phenomena and uses -ixpers-of-Experts for different physics categories. However, it faces challenges in clearly defining physical components from diverse text prompts and its effectiveness limited to specialized datasets (i.e., WISA-32K (46) containing explicit physics phenomena and fails to generalize to open-domain datasets like Koala-36M [48), making large-scale data application difficult. This paper proposes VideoREPA, training based method that improves the physical realism of generated videos by aligning representations with those learned by video SSL models. VideoREPA features compatibility with open-domain datasets, enhancing its potential for wide adoption.
rely on direct physics simulation. However,
155 uses MILMs to refine
3 Methods
This section first covers preliminary on Latent Diffusion Models and REPA (Section 3.1p. Subse- quently, Section 3.2 analyzes the interplay between model understanding and generation capabilities, thereby establishing the motivation for our work. Building upon this, Section 3.3 presents our core contribution: VideoREPA, a novel framework featuring the Token Relation Distillation (TRD) loss, distilling physies from video foundation models [47] by aligning token-level relations across both spatial and temporal dimensions. VideoREPA, for the first time, applies feature alignment to finetune pre-trained VDMs, leveraging relational distillation to incorporate spatial-temporal dynamics.
3.1 Preliminaries
Latent Diffusion Models. Latent Diffusion Models (LDMs) 40
typically operate in the latent space of pre-trained Variational Autoencoder (VAE). They generate data by learning to reverse a forward diffusion process that gradually adds noise. Our VideoREPA framework is designed for finetuning transformer-basee video LDM, i.e., Cog VideoX (56). The training objective is the mean squared error (MSE) between the added noise and the noise predicted by the model
Lan Et,on.e [Ic- [ co(ar20 + t)c,
where 2p denotes the initial latent input (obtained by encoding video frames, e.g., via a 3D VAE), ~ ((0 I) sampled noise, 1 the diffusion timestep, 04 and a; are schedule-dependent coefficients (e.g., a, = vār, o; vI ā.) and 0 represents the parameters of the denoising transformer.
Representation alignment for generative models. Representation alignment (REPA) (58) is straightforward regularization method, demonstrating that the convergence speed of image diffusion model training process can be significantly improved by aligning representations y. from encoders of pre-trained vision foundation models (e.g.. DINOv2 (37]) with the internal representation. REPA distills the y, of clean image x into the denoising transformer representation h, of noisy input Xt- Specifically, the y. f(x) E NxD and h. Jx(x)) which will then be input to trainable MLP he for dimension alignment. The alignment loss can be formulated as that maximizes the token-wise feature similarities:
CREPA E
N Σ
sim(y!")
!n) h (h((a))
The sim(..-) denotes a similarity metric (e.g., cosine similarity). This method reduces the semantic gap between ht and ye, accelerating the training speed of DìT. The final loss is: L -= Lait + XLREPA REPA has been reported to speed up DiT training by over 17.5x. Subsequent works, such as VA- VAE [57] which improved VAE features, achieved 21 x speedups, and REPA-E [25] which proposed end-to-end training, reached up to 45x acceleration for DIT.
3.2
Understanding vs. generation
Understanding refers to the ability of models to interpret input data and extract meaningful informa- tion, whereas generation involves generating novel data, well-established principle in generative modeling is that enbancing understanding often leads to improved generation quality [22 [3 54 16].

---

DynamiCrafter [54] for example, improves video generation by leveraging CLIP and query trans- former to better understand image conditions. Similarly, SmartEdit [22 and MGIE [13] utilize language models to enhance instruction understanding for more accurate image editing.
This principle motivates work addressing the poor physics plausibility observed in leading T2V generation models [3]. We wonder whether physics plausibility in generation can be improved through deepening the model's understanding to physics and start by comparing the physics understanding abilities of large VDMs against specialized Video Foundation Models (ViFMs), Physion [6) a benchmark for evaluating physics understanding is used (detailed at Appendix C). As shown in Figure I CogVideoX (2B) demonstrates significantly weaker physics understanding compared to smaller VideoMAEV2 (86M) 447] This disparity highlights an opportunity: bridging the physics understanding gap by transferring knowledge from capable ViFMs to powerful VDMs. Therefore, we propose a method based on representation alignment. The following section details our VideoREPA framework, which close the understanding the gap through TRD loss, as shown in Figure]]
3.3 Token Relation Distillation loss
Introducing physics knowledge understanding into VDMs is non-trivial. Unlike text understanding, which can often be enhanced by incorporating more powerful text encoder [22] since the text prompt is provided directly as input, text/image-to-vicn generation lacks direct video input during inference. Consequently, directly leveraging powerful physies understanding video encoder to guide the generation process is generally infeasible.
Recently, REPA [58 has emerged as a method to bridge the semantic gap between pre-trained foundation models and diffusion models. By aligning internal representations during from-scratch training of diffusion models, REPA primarily aims to enhance semantics and accelerate training. This suggests a potential avenue for transferring physics understanding from capable ViFMs into VDMs. approaches are insufficient for effectively achieving this goal, particularly when finetuning pre-trained VDMs.
However, existing REPA
22 42 58
The gap of applying REPA for physics enhancement in video generation model. Though REPA and related methods 25 58 57 build bridge between foundation and generation models, they become expired when aiming at bridging ViFMs and VDMs: 1) Shift in Focus (Spatial vs. Spatio- Temporal): Existing REPA techniques predominantly focus on aligning spatial features within static images. However, physical plausibility in videos relies critically on temporal dynamics, i.e., the rational evolution of motion and interactions over time, in addition to the correct appearance within single frame at spatial dimension. Standard spatial alignment is insufficient to capture or enforce these crucial temporal dynamics. 2) Mismatch in Application Context -rertSh Acceleration vs. Finetuning for Knowledge Transfer): Existing REPA approaches have primarily been validated and utilized for accelerating the training of models from scratch [57 58). optimizing convergence speed. Our goal. however, is different: injecting specific knowledge (physics) into already pre-trained VDMs via finctuning. This needs further discussion and validation. 3) Mechanism Mismatch (Hard Alignment vs. Finetuning Stability): REPA employs a direct feature similarity loss (c.g.. cosine similarity) to train a DiT from scratch. When applied during finetuning, this "hard" alignment objective attempts force potentially incompatible feature spaces- the latent space of pretrained and the SSL-optimized feature space of ViFM to match directly. As demonstrated in our experiments (see Section 4.5). finetuning Cog VideoX with standard REPA leads significant degradation in semantic quality and overall coherence. We attribute this failure to the direct alignment disrupting the well-establisheds internal representations in pre-trained VDMs. 4) Added Complexity (Temporal Compression in Latents): Unlike typical image latents, VDM latent spaces often employ significant temporal compression (c.g., 4x in Cog VideoX (56)). This adds complexity to the design alignment process, as the temporal granularity between the VDM's latent representation and the ViFM's 's feature representation may differ, requiring careful handling during alignment design.
These limitations collectively indicate that a naive application of standard REPA is unsuitable for enhancing the physical plausibility of pre-trained VDMs via finetuning. A different strategy is required-on one that specifically addresses temporal dynamics, ensures stability during finetuning, and effectively manages differences between the VDM's latent space and the ViFM's 's feature space.
To address these challenges, we propose VideoREPA, a framework leveraging novel Token Relation Distillation (TRD) loss to distill physics understanding models for better physics plausible
5

---

generation. Instead of enforcing direct feature similarity (hard alignment), which proved unsuitable for finetuning (Section 4.5). TRD aligns the relational structure (i.e.. pairwise token similarities) between the internal representations in VDMs and those of a capable Video Foundation Model (e.g., VideoMAEv2). This relational alignment provides a softer guidance suitable for finetuning, explicitly incorporates spatio-temporal dynamics by considering relationships both within and across frames, and issues arising from direct feature space incompatibility. Unlike prior REPA work focused on spatial alignment for image generative model acceleration [58 57), VideoREPA represents, to our knowledge, the first representation alignment method developed for finetuning VDMs to inject specific knowledge (physies contained in spatial-temporal dynamics), pushing beyond mere acceleration.
As shown in Figure 2 the TRD loss aligns pairwise token similarities between ViFM and VDM representations to distill spatial constraints within frames and temporal dynamics across frames from the ViFM. Specifically, let Ey be the ViFM encoder processing video V E R It outputs w is the token count over f temporal and
features yv (V) E NR where N h x spatial positions, with feature dimension D. (F/ H/hh, W/w) represent the temporal/spatial compression ratios. For VDMs, the 3D VAE encoder (56) compresses V into latent z. The hidden state he f' (zt) of denoising transformer is derived from noisy latent The h, is input into a trainable MLP h for dimension D aligmment, i.e., h (h,) Rxxxxxx Note: Although the dimensions (ff h, w) might differ from those derived from the ViFM output, we use the same notation here for annotation simplicity, representing the dimensions after ensuring compatibility.
We compute spatial token pairwise similarity matrix first. After reshaping yv to [x(x)))) the relation (i.e., cosine similarity) matrix for spatial dimension at frame d can be expressed
ydd ydJ lly d.i
dr Ispatial
y where i E [w, index spatial positions. This produces spatial Rhwxhwww per frame. Aggregating across f frames yields Yspatil E IR (xhwxhrr Features are normalized before computing similarity.
Then for temporal relation, we compute cross-frame similarities between each token in frame d and all tokens from other frames & d. Let yv E x be reshaped foundation model features. For each frame and token position
y you
d.i.j.e
VA [ j [1, hw]
lly |"1
This produces a 4D tensor y semp E RR f wwwxwxxxxxxxxIx Corresponding spatial similarity matrix hagatial and temporal similarity matrix hemp are computed identically using the VDM features h (h,).
The TRD loss then quantifies the difference between VDM and ViFM by calculating the average L1 distance using the corresponding spatial and temporal similarity values
Awe
Aac
LTDD
I
diij spatial
y d spatial
+
d.iee
d.i.j. temp
(5)
d=1
(
d,e=1 c
i.j
Spatial component
Temporal component
The final loss can be expressed as = Can + AL where À a hyperparameter. The TRD loss can be computed in a unified operation that, given frame, directly calculates its self-similarity and its similarity with other frames, as implemented in the code
3.4 Remaining issues for implementation
Successfully applying the Token Relation Distillation (TRD) loss, as detailed in Section 3.3 requires addressing practical implementation challenges, primarily concerning feature dimensionality and input configuration for Video Foundation Models (ViFMs).

---

Table Results of Videophy. i denotes the results reported from WISA [46] and * denotes detailed prompt ippt, see Section 4.2 Semantic Adherence (SA) measures the video-text alignment and
fidelity. Importantly, Physical physics laws in the real-world.
Commensense (PC) measures whether generated videos follow the
Solid-Solid
Solid-Fluid
Fluid-Fluid
Overall
Methods
SA
PC
SA
PC
SA
PC
SA
PC
VidcoCrafter2
50.4
32.2
50.7
27.4
48.1
29.1
50.3
29.7
DreamMachine
55.1
21.7
59.6
23.3
58.2
18.2
57.5
21.8
LaVIE
40.8
18.3
48.6
37.0
69.1
50.9
48.7
31.5
Dosmos-Difussio-755
18
Hunyuan Video"
55.2
16.1
67.1
30.1
54.5
54.5
60.2
28.2
PhyT2Vi
61
37
WISA (Koala dataset)
62
33
WISA (WISA dataset)
67
38
Cog VideoX-22
37.8
12.6
67.1
30.1
45.5
50.9
51.6
26.2
Cog VideoX-2B
49.6
13.3
71.2
28.1
60.0
50.9
60.5
25.6
VideoREPA-2B"
52.4
18.2
77.4
32.2
60.0
52.7
64.2
29.7
Cog VideoX-5B
60
33
VogVideoX-5B
53.1
18.2
75.3
32.9
56.4
61.8
63.1
31.4
XogVideoX-SB*
62.9
19.6
76.0
33.6
72.7
61.8
70.0
32.3
RideoREPA-SB
58.0
28.0
82.9
39.0
80.0
74.5
72.1
40.1
key issue is the dimensional misalignment between features of ViFM and VDM. After their respective encoding processes, the temporal ff and spatial h x w dimensions often differ. Advanced VDMs [56] 45 (24 frequently employ 3D VAEs with high temporal compression, e.g., 4x 8x. In contrast, ViFMS 47 typically use lower compression ratios, e.g., 2x. This results in ViFM feature maps yv having a larger temporal size, and often different spatial sizes, compared VDM latents hy To reconcile these differences while maximizing the guidance from the ViFM, we adopt the principle of interpolating VDM latent dimensions to match ViFM features, strategy empirically found to be more effective. Another consideration arises from computational resource limitations when processing inputs for ViFMs, which often utilize 3D full attention. Inputting high-resolution video, e.g.. 480x720, as in Cog VideoX directly into a ViFM can prohibitively memory-intensive. This necessitates a trade-off. We explore three strategies and finally decide to process all frames at a lower resolution to preserve the integrity. Experiments, as shown in Appendix D are conducted to support this decision/approach.
large number of frames, e.g., 49 frames, as in Cog VideoX
4 Experiments
4.1
Implementation details
Model setups. We adopt VogVideoX [56]. powerful T2V diffusion model, as the base model and fintune it using the proposed TRD loss. Specifically, we develop VideoREPA-28 and VideoREPA-SB. corresponding to the ones in CogV VideoX. The generated videos consist of 49 frames at a resolution of 420. We adopt VideoMAEv2
as the alignment target encoder.
Training details. Unlike methods such as WISA [46] that require specialized datasets with explicit physical phenomena (e.g.. WISA-32K (46)), we leverage Open Vid [35). large-scale, open-domain video dataset. Videos are center-cropped and resized to 480x VideoREPA-2B is finetuned on 32k use LoRA for efficient finetuning on 64k Open Vid videos for 2,000 steps. The default alignment depth is 18. All experiments utilize
Vid videos for 4,000 steps. For the larger VideoREPA-SB.
NVIDIA A.00 (80GB) with total batch size of 32. More details are shown in Appendix
4.2 Evaluation
We evaluate VideoREPA on two challenging benchmarks designed to comprehensively assess the physical plausibility of videos generated by text-to-video models:
7

---

Cog
Ours
Honey diffinsing into warm milk. (Fluid&causalitt)
A cylindrical metal contniner rolls down grasey hill... (Rigid body)
HY
Cos
Ours
Pencil rolls around flat desk. (Rigid bady&rolling)
crane gently lifts pallet of bricks. (Causality)
Hy
Coc
Ours
Two people stretch bungee cord... (Elastic)
A player kicks soccer ball... (Human&ccusalltty
Figure 3: Qualitative comparison of Hunyuan Video (HY) 24 Cog VideoX (Cog) [56], and Video-
REPA (Ours), exhibiting enhanced physics commonsense of VideoREPA.
VideoPhy [3]. This benchmark uses 344 prompts to test if generated videos adhere to physical commonsense real-world activities, covering diverse material interactions (e.g. solid-solid, solid- fluid) to evaluate whether VDMs can generate videos with plausible physics. The proposed VideoCon- Physics 3 adopted as auto-rater our paper. When testing Cog VideoX and our VideoREPA, we refine the short prompt into a detailed prompt. This is crucial because the models are trained with long prompts, and prompts impact the quality of the video generation according to the official of Cog Vogee (56]. Details are shown in Appendix E Following WISA (46), we set SA = I and PC I when their values are greater than or equal to 0.5. Values less than 0.5 are set as SA 0 and PC
VideoPhy2 [4. VideoPhy2 is an action-centric benchmark designed to evaluate physical common- sense in generated videos. It aims overcome limitations of prior benchmarks, such restricted size, absence of human interaction, and sim-to-real discrepancies. The dataset includes 590 detailed testing prompts covering 200 diverse actions. For automated evaluation, we employ the VideoPhy2-Autoovv model. We utilize the upsampled prompts provided by the VideoPhy2 benchmark.
4.3 Quantitative comparisons
Using auto-evaluators for VidcoPhy and VidcoPhy2, we quantitatively assess our VidcoREPA.
To show its superiority, VidcoREPA is benchmarked against its CogVidcoX (56) baseline, other leading T2V models (VideoCrafter2 [ LaVie [50], DreamMachine [30], Cosmos-Diffusion [
Hunyuan Video
24 and physics-aware methods like PhyT2V [55] and WISA 46).
Results in Table showw VideoREPA achieves maae--ofthee-ee performance across three interaction types. Compared to its baseline CogVideoX, VideoREPA-SB improves the Physical Commonsense (PC) score by 24.1 % overall (specifically, 42.9% for Solid-Solid, 16.7% for Solid-Fluid, and
20.6% for Fluid-Fluid). Our method also surpasses WISA [46). technique designed for enhancing physics commonsense in video generation. Notably, while WISA shows efficacy when trained on the physics-explicit dataset WISA-32K [46), it struggles to generalize to open-domain datascts like

---

Koala-36M 48 In contrast, VideoREPA, trained on an open-domain dataset, demonstrates clear improvements over WISA on such data (e.g., PC score of 40.1 vs. WISA's 33 on koala-36M),
We further assess physical commonsense on VideoPhy2 4 an action-centric benchmark featuring complex human-object interactions. Following its protocol, Semantic Adherence (SA) and Physical Commonsense (PC) scores are the proportion of videos rated 2 4 for each metric. Our VideoREPA (2B) demonstrates a significant improvement over the baseline by 4.57 scores as shown in Table 2 further validating the effectiveness of our proposed method.
4.4 Qualitative comparisons
We present qualitative comparisons of videos generated by different models in Figure Our VideoREPA achieves superior physics plausibility compared to Hunyuan Video and CogVideo) VideoX. Specifically, in the "pencil roll" scenario, videos from Hunyuan Video and Cog Video often depict pencils rolling in manner inconsistent with rigid body motion laws. In contrast, VideoREPA showcases physically consistent and stable motion. Similarly, for the "crane lifting bricks" example, VideoREPA accurately portrays the crane maintaining a physical connection while lifting the pallet. The other methods, however, tend to generate videos where the bricks are implausibly suspended without any visible means of support from the crane.
Table 2: Results of Videophy2
Methods
SA
PC
CogVideoX
21.02
67.97
VideoREPA
21.02
72.54
For clarity, all prompts are shown in the short version, but the models received the detailed. More results are provided in Appendix G Check detailed prompts and videos PProject Page
4.5 Ablation studies
We conduct ablation studies to reveal the properties and validate the effectiveness of our proposed VideoREPA. Performance of VideoREPA-2B on VideoPhy is reported unless otherwise specified.
Ablation study on TRD loss. NaN means only Lair is adopted.
Table 3:
Token Relation Distillation loss. We ablate TRD loss assess its effectiivnesss. Physical plausibility relies on correct spatial ap- pearance (e.g., no irregular deformations) and coherent temporal dynamics (c.g., smooth, accurate motion). We design TRD loss with both spatial and temporal terms address these. The PC scores in Table Sable3com confirm their importance, showing that removing either component degrades performance. Interestingly, focusing alignment on only the spatial or temporal dimension negatively impacts Seman- tie Adherence (SA), likely due harming the integrity of learned representation of VDMs.
Loss Type
SA
PC
NaN
63.6
23.2
TRD loss
64,2
29.7
only spatial
61.0
27.3
only temporal
61.0
27.9
The ineffectiveness of REPA. As discussed in Sec- tion 3.3 directly applying REPA (58) for physics en- hancement via finetuning pre-trained VDMs presents several challenges. Results in Figure demonstrate this: finetuning VDM with the standard REPA loss leads to a significant degradation in video se- mantic quality. This outcome supports our asser- tion that REPA, with its "hard" alignment approach (i.e., token similarity), pre-trained VDMs as it can disrupt their established feature spaces. In contrast, our proposed TRD loss, which offers "soft" guidance, proves substantially more effective for finetuning VDMs.
(DINOx2)
RFPA Laws (VilcMAAE2)
TRD (VilooMAF22)
unsuitable for finetuning
Figure 4: Ablation on REPA loss.
Conclusion and outlook
In this paper, we presented VideoREPA, a framework designed to transfer physics knowledge from Video Foundation Models (ViFMs) to text-to-video diffusion models (VDMs) via token-level relation distillation. We first identified a significant physies understanding performance gap between ViFMs and VDMs. Subsequently, motivated by the principle that enhanced understanding facilitates

---

quallty generation, proposed the Token Relation Distillation (TRD) loss to distill physics understanding capability from pre-trained ViFMs to VDMs, thereby achieving more physically plausible video generation. Extensive experiments demonstrate that VideoREPA achieves state-of- the-art generation results, exhibiting great physical commonsense in generated videos.
Limitations. Although VideoREPA has achieved significant improvement through fine-tuning VDMs, its potential for pre-training VDMs remains unvalidated due to computational resource limitations. Future research could explore incorporating VideoREPA into the pre-training of VDMs and developing targeted innovations to effectively inject physies knowledge during this phase.
10

---

References
[1] N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03572. 2025.
[2] M. Assran, A. Bardes, Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.0998; 2025.
[3] H. Bansal, Z. Lin, T. Xie, Z. Zong, M. Yarom, Y. Bitton, C. Jiang, Y. Sun, K.-W. Chang, and A. Grover. Videophy: Evaluating physical commonsense for video generation. arXiv preprint TTt::2006.035220 2024.
[4] H. Bansal, Peng, Bitton, R. Goldenberg, A. Grover, and K.-W. Chang. Videophy-2: challenging action-centric physical commonsense evaluation in video generation. arXiv preprint arXiv:2503. 06800, 2025.
[5] A. Bardes, Q Garrido, Ponce, X. Chen, M. Rabbat, LeCun, M. Assran, and N. Ballas. V-jepa: Latent video prediction for visual representation learning. 2023.
[6] D. M. Bear, E. Wang, D. Mrowca, F. J. Binder, H.-Y. F. Tung. R. Pramod, C. Holdaway, S. Tao, K. Smith, F.-Y. Sun, al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2106. 112262, 2021.
[7] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts. al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122
2023.
[8] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[9] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310-7320, 2024.
[10] Y. Chen, J. Cao, A. Kag, V. Goel, S. Korolev, C. Jiang, S. Tulyakov, and J. Ren. Towards physical understanding in video generation: A 3d point regularization approach. arXiv preprint arXiv:2502.03639, 2025.
[11] A. Cherian, R. Corcodel, S. Jain, and D. Romeres. Llmphy: Complex physical reasoning using large language models and world models. arXiv preprint arXiv:2411.08022 2024.
[12] A. Ehtesham, S. Kumar, A. Singh, and T. T. Khoei. Movie gen: Swot analysis of meta's generative ai foundation model for transforming media generation, advertising, and entertain- ment industries. In 2025 IEEE 15th Annual Computing and Communication Workshop and Conference (CCWC), pages 00189-00195. IEEE, 2025.
[13] T.-J. Fu, W. Hu, X. Du, W. Y. Wang, Y. Yang, and Z. Gan. Guiding instruction-basee image editing via multimodal large language models. arXiv preprint arXiv:2309. /7/02, 2023.
[14] Y. Gao, H. Guo, T. Hoang, W. Huang, L. Jiang, F. Kong, H. Ji, L. X. Li, et al. Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint arXiv:2506.09113, 2025.
[15] Q Garrido, N. Ballas, M. Assran, A. Bardes, L. Najman, M. Rabbat, E. Dupoux, and Y. LeCun. Intuitive physics understanding emerges from self-supervised pretraining on natural videos. arXiv preprint axXiv:2502.1183: 2025.
[16] Y. Ge. S. Zhao, Z. Zeng, Y. Ge. C. Li, X. Wang, and Y. Shan. Making llama see and draw with seed tokenizer. arXiv preprint arXiv:2310.01218, 2023.
[17] Gerstenberg, M. F. Peterson, N. D. Goodman, A. Lagnado, and B. Tenenhaum. Eye- tracking causality. Psychological science, 28(12):1711-174,, 2017.

---

[18] R. Girdhar, A. El-Nouby, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Omnimae: Single model masked pretraining on images and videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10406-10417, 2023.
[19] J.-B. Grill, F. Strub, F. Altché, C. Tallec, Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-2111 2020.
[20] O. Groth, F. B. Fuchs, Posner, and A. Vedaldi. Shapestacks: Learning based physical intuition for generalised object stacking. In Proceedings of the european conference on computer vision (eccv), pages 702-717, 2018.
[21] K. He, Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[22] Y. Huang, L. Xie, X. Wang, Z. Yuan, X. Cun, Y. Ge, J. Zhou, C. Dong, R. Huang. R. Zhang, al. Smartedit: Exploring complex instruction-based image editing with multimodal large language models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 8362-8371, 2024.
[23] J. Y. Koh, D. Fried, and R. R. Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36:21487-21506 2023.
[24] W. Kong, Q Tian, Zhang, R. Min, Z. Dai, J. Zhou, Xiong, Li, B. Wu, Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603. 2024.
[25] X. LLnn. J. Singh, Y. Hou, Z. Xing, S. Xie, L. Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. arXiv preprint arXiv:2504. 10483, 2025.
[26] Liao, Z. Yang. L. Li, D. Li, K. Lin, Y. Cheng, and L. Wang. Imagegen-cot: Enhancing text-to- image in-context learning with chain-of-thought reasoning. arXiv preprint 2rXiv:2503.19312 2025.
[27] J. Lin, Z. Wang, Y. Hou, Y. Tang, and M. Jiang. Phy 124: Fast physics-driven 4d content generation from a single image. arXiv preprint arXiv:2409.07179, 2024.
[28] J. Lin, Z. Wang, S. Jiang, Y. Hou, and M. Jiang. Phys4dgen: A physics-driven framework for controllable and efficient 4d content generation from single image. arXiv preprint 1xXi:24411166800 2024.
[29] S. Liu, Z. Ren, S. Gupta, and S. Wang. Physgen: Rigid-body physics-groundee image- image-to-viden to-video generation. In European Conference on Computer Vision, pages 360-378. Springer, 2024.
[30] Luma AI. Dream machine dream machine
I video generator, 2024.
ai/
[31] G. Ma, H. Huang, K. Yan, L. Chen, N. Duan, S. Yin, C. Wan, R. Ming, X. Song. X. Chen, et al. Step-video-22v technical report: The practice, challenges, and future of video foundation model. arXiv preprint arXiv:2502.1022
2025.
[32] F. Meng, J. Liao, X. Tan, W. Shao, Q. Lu, K. Zhang, Y. Cheng. D. Li, Y. Qiao, and P. Luo. To- wards world simulator: Crafting physical commonsense- based benchmark for video generation. arXiv preprint arXiv:2410.05363, 2024.
[33] A. Montanaro, I.. Savant Aira, E. Aiello, D. Valsesia, and E. Magli. Motioncraft: Physics- zero-shot video generation. Advances in Neural Information Processing Systems, 37:123155- 1 123181, 2024.
[34] S. Motamed, I. Culp, K. Swersky, P. Jaini, and Geirhos. Do generative video models learn physical principles from watching vidcos? arXiv preprint ArXiv:2501.0903 2025.
12

---

[35] K. Nan, R. Xie, P. Zhou, T. Fan, Z. Yang, Z. Chen, X. J. Yang, and Y. Tai. Openvid- Im: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.02377, 2024.
[36] OpenAl team. Openai: Gpt-4 for vision (chatgpt with image input), 2023. https://openai com/
[37] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199. 2023.
[38] K. Qian, J. Miao, Z. Luo, Z. Fu, Y. Shi, Y. Wang, K. Jiang, M. Yang, D. Yang, et al. Lego- motion: Learning-enhance grids with occupancy instance modeling for class-agnostic motion prediction. arXiv preprint arXiv:2503.07367, 2025.
[39] R. Riochet, M. Y. Castro, M. Bernard, A. Lerer, R. Fergus, V. Izard, and E. Dupoux. Int- phys: A framework and benchmark for visual intuitive physies reasoning. arXiv preprint 1rXiv:1803.07611 2018.
[40] R. Rombach, A. Blattmann, D. Lorenz. P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[41] J. Tian, X. Qu, Z. Lu, W. Wei, S. Liu, and Y. Cheng. Extrapolating and decoupling image- to-video generation models: Motion modeling easier than you think. arXiv preprint arXiv:2503. 00948, 2025.
[42] Y. Tian, H. Chen, M. Zheng, Y. Liang, C. Xu, and Y. Wang. U-repa: Aligning diffusion u-nets to vits. arXiv preprint arXiv:2503. 18414, 2025.
[43] Z. Tong, Y. Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 110000781993 2022.
[44] R. Venkatesh, H. Chen, K. Feigelis, D. M. Bear, K. Jedoui, K. Kotar, F. Binder, W. Lec, S. Liu, K. A. Smith, et al. Understanding physical dynamics with counterfactual world modeling. In European Conference on Computer Vision, pages 368-387. Springer, 2024.
[45] A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang. J. Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint 2rX1v:2503.20314, 2025.
[46] J. Wang, A. Ma, K. Cao, J. Zheng. Z. Zhang, J. Feng, S. Liu, Y. Ma, B. Cheng, D. Leng, et al. Wisa: World simulator assistant for physics-aware text-to-video generation. arXiv preprint arXiv:2503.081 2025.
[47] L. Wang, B. Huang, Z. Zhao, Z. Tong. Y. He, Y. Wang, Y. Wang, and Y. Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14549-1433 14560, 2023.
[48] Q. Wang, Y. Shi, J. Ou, R. Chen, K. Lin, J. Wang, B. Jiang, H. Yang, M. Zheng, X. Tao, et al. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. arXiv preprint 2.Xiv:2410.0822 2024.
[49] X. Wang, S. Zhang, Tang, Y. Zhang, C. Gao, Y. Wang, and N. Sang. Unianimate-dit Human image animation with large-scale video diffusion transformer. arXiv preprint 1222, 2025.
[50] Y. Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y. Wang, C. Yang. Y. He, J. Yu, P. Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, 133(5):3059-307 2025.
[51] W. Wu, 7hu, and M. Z. Shou. Automated movie generation via multi-agent cot planning. arXiv preprint
2025.
13

---

[52] T. Xie, Y. Zhao, Y. Jiang, and C. Jiang. Physanimator: Physics thysics--uided generative cartoon animation, arXiv preprint arXiv:2501.16550 2025.
[53] Xie, Z. Zong, Y. Qiu, X. Li, Y. Feng, Y. Yang, and C. Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4389-4380 4398, 2024.
[54] J. Xing, M. Xia, Y. Zhang, H. Chen, W. Yu, H. Liu, G. Liu, X. Wang, Y. Shan, and T.-T. Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In European Conference on Computer Vision, pages 399-417. Springer, 2024.
[55] Q. Xue, X. Yin, B. Yang, and W. Gao. Phyt2v: Lim-guided iterative self-refinement for physics-grounded text-to-video generation. arXiv preprint arXiv:2412.00596 2024.
[56] Yang, J. Teng, W. Zheng, M. Ding, S. Huang, Xu, Yang, W. Hong. Zhang, G. Feng, et al. Cogvideox: Texe-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.. 06072, 2024.
[57] J. Yao, B. Yang, and X. Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501.08 2025.
[58] S. Yu, S. Kwak, H. Jang, Jeong, J. Huang, J. Shin, and S. Xie. Representation align- ment for generation: Training diffusion transformers is easier than you think. arXiv preprint 10X:::2400.06940, 2024.
[59] S. Zhang, Wang, Y. Zhang, K. Zhao, H. Yuan, Z. Qin, X. Wang. D. Zhao, and J. Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.0414 2023.
[60] T. Zhang, H.-X. Yu, R. Wu, B. Feng, C. Zheng, N. Snavely, J. Wu, and W. T. Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. In European Conference on Computer Vision, pages 388-406. Springer, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: From line 6 to line 20, we introduce the contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the Conclusion section.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
There are no theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We illustrate the experiment details in the Experiments section and Appendix
B.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The dataset that we used is open- and the code has been made public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbllc/guideeee http://oeeubiiissioooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting//etails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In the Experiments section and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We report the performance of different categories in Table along with the average performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We illustrate the computation resources needed in the Experiments section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have read and obey the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No potential societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the creators of models and datasets used in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /dtaaset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 'ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/moddl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research do not include crowdsourcing or involve research with human subjects.
20

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No study participants or potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: LLM does not impact the methodology in paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy http::/e/eerrrpscccc/oooeeeeeee for what should or should not be described.
21

---

Appendix
A
Analysis on why TRD loss contribute to physics knowledge transfer
A key objective of our framework is enhance the physical plausibility of video generation by transferring physics-related knowledge from video foundation models (ViFMs) to video diffusion models (VDMs). Existing VDMs exhibit limited capability in modeling physical dynamics, whereas ViFMs have demonstrated strong emergent understanding of object configurations and motion causality. Distilling relational representations from ViFMs into VDMs therefore serves as an effective mechanism to bridge this capability gap. Rather than being byproduct of training, physical consistency is an explicit optimization objective in our framework, and the Token Relation Distillation (TRD) loss is specifically formulated to encode this objective.
The TRD loss operates by aligning spatial and temporal relational matrices between the ViFM and the VDM. The spatial component enforces consistency in static structural relations such as object integrity, contact relationships, and positional dependencies within each frame. The temporal component captures dynamic interactions, including motion trajectories, causality, and enter-frame dependencies, which are essential for modeling intuitive physics. By aligning these relational structures instead of matching raw features, TRD induces the VDM to internalize physical rules governing object interactions and temporal evolution. Empirical results (see Figure 3) demonstrate that VDMs trained with TRD exhibit improved physics understanding in intermediate feature evaluations and generate videos with significantly enhanced physical plausibility (Tables [ and 2. validating the effectiveness of relational knowledge transfer.
B Detailed training setting
For finetuning, we utilize the Open Vid dataset 33 an open-source, high-quality collection of videos with expressive captions, containing over one million in-the-wild videos. The learning rate is set to le-4 for LoRA-based finetuning of Cog VideoX-BB and 2e-6 for full-parameter finetuning of Cog VideoX-2B. For LoRA, the rank is 128 and alpha is 64. The target encoders explored in our experiments include VideoMAEv2-B B (47), V-JEPA-L [5), OmniMAE-B [18). and VideoMAE-B B [43). Unless otherwise specified, an alignment depth of 18 is used for both VideoREPA-22 and VideoREPA- SB. Inspired by VA-VAE [57]. prevent the alignment of unnecessary noise, we incorporate margin m (typically ranging from to 0.1) into the TRD loss (Equation ( Specifically, values in TRD loss less than this margin are set to O. The appropriate margin value found to vary: 0.1 for VideoMAEv2, 0.05 for V-JEPA, and 0 for both VideoMAE and OmniMAE, reflecting the great fit for each encoder.
C Physion evaluation setting
For the physics understanding evaluation discussed in Section 3.2 we utilize the Physion bench- mark [6]. Physion presents realistic simulations of diverse physical scenarios, where objects are manipulated in various configurations to assess different types of physical reasoning, including stabil- ity, rolling motion, and object linkage, among others. We specifically employ Physion v1.5 (6), the latest version, which features improved rendering quality and more physically plausible simulations.
Physion stands out as a challenging benchmark due to its inclusion of diverse physical phenomena, complex object dynamics, and realistic 3D simulations. These characteristics make it preferable choice over other benchmarks like ShapeStacks 20 and IntPhys 39 which offer comparatively limited object dynamics.
Specifically, for feature extraction from Cog and VideoREPA, we select features from three temporal dimensions, evenly sampled from the twelve available temporal dimensions in their respective latent spaces. All spatial tokens within these selected temporal slices are utilized. We employ the Object Contact Prediction (OCP) task from the Physion for evaluation. The OCP task assesses a model's 'ss capability to predict future contact between two objects based on an initial context video, requiring an implicit understanding of physical dynamics for accurate prediction.
The evaluation procedure involves first extracting features using the VDM. Consistent with our align- ment strategy, we extract these features from the 18th layer of the denoising network. Subsequently,
22

---

PC
PC
30
80 Score
28
Sccore
29
26
28
14
16
20
22
0.4
0.45
0.5
0.55
0.6
Depth
A
Figure 5: Left: Effect of alignment depth. Right:
Effect of X. PC score on VideoPhy is reported.
Table Ablation study on different alignment target video foundation models.
Models
SA
PC
63.6
23.2
VideoMAE
43
59.8
26.2
V-JEPA S
64.5
24.7
OminiMAE 18
61.6
24.7
V-JEPA 2 2
65.1
27.3
VideoMAEv2
47
64.2
29.7
these extracted features are used to train logistic regression classifier to perform the OCP task, i.e., predicting future object contact. For this evaluation, we utilize the "roll" and "contain" subsets of the Physion benchmark, with prediction accuracies reported in Figure] [
D Additional ablation study
Additional ablation studies are conducted, aligning the experiment setting with Section 4.5
Different video foundation models. We evaluate aligning the VDM with various pre-trained ViFMs: VideoMAE [43] V-JEPA (5), V-JEPA 2), and VideoMAEv2 47. Results in Table 4 indicate VideoMAEv2 performs best, likely due to its extensive pre-training on millions of videos and resulting strong generalization. Thus choose to align VDM with VideoMAEv2 in our VideoREPA.
Different alignment depth. We also investigate the effect of aligning different layers of the diffusion models with features extracted from the ViFM. Experiments in Figure B indicate that an alignment depth of 18 yields the best performance, which is adopted in our VideoREPA.
Effect of X. We try different values of À for the weight of TRD loss. The Figure 5 shows that the 0.5 features the best trade-off between the original diffusion loss and the proposed TRD loss.
Dimension alignment. We conduct an ablation study on the dimension alignment issue when applying the TRD loss. The temporal dimension of VideoREPA's latent space typically smaller than that of VideoMAEv2's features, while its spatial dimensions are often larger. Our guiding principle is that interpolating VDM representations (from VideoREPA) to match the dimensions of ViFM features (from VideoMAEv2) best preserves the ViFM's knowledge. The experiments in Table 5 support this choice. Thus, we interpolate the VDM's latent representations to match the feature dimensions of the pre-trained ViFM. Furthermore, considering that the first encoded frame in the latent space of 3D VAE primarily serves maintain semantic information [56), we exclude from the alignment process to focus dynamic content.
Trade off between input frames and resolution. Given the computational expense of full attention mechanisms in ViFMs, directly inputting high-resolution video, e.g., 480x720, as used by Cog VideoX for generation large number frames, e.g., 49 frames into ViFM can be prohibitively memory- intensive. This necessitates careful trade-off between input frame count and resolution for ViFM processing. We explored three common strategies manage this:
1. Processing all video frames uniformly lower resolution.
23

---

Table 5: Dimension alignment target. Target dimension is VDM refers to interpolating ViFM features to match VDM dimensions.
Table 6: Trade-off between resolution and
frames. Corresponding strategies related to in- dexes are stated in the Appendix D
Spatial
Temporal
SA
PC
Index
SA
PC
VDM
VDM
63.6
26.2
1
64.2
29.7
ViFM
VDM
63.1
28.5
2
63.4
29.1
ViFM
VIFM
64.2
29.7
3
54.7
32.0
2. Processing temporally grouped subsets of frames at high resolution.
3. Processing all frames high resolution but with spatial cropping into patches groups.
Based on empirical evaluations in Table 66 we adopted the first strategy: processing all frames at reduced resolution. This approach was found to best preserve the holistic nature of the ViFM's pre-trained representations with the lowest computation resources needed, whereas the latter two strategies (grouping or cropping) tended to degrade either physies plausibility or semantic quality of the generated videos.
E
Generating detailed prompt for VideoPhy
Adhering to guidance from the official VideoX documentation [56]. which emphasizes the criti- cality of refining prompts, we specifically elaborate the often-brief prompts found in the VideoPhy benchmark 3. Poorly formulated prompts can significantly degrade Semantic Adherence, conse- quently hindering validations of the perceived Physical Commonsense. To mitigate ambiguity, we leverage Large Language Models (LLMs) such as GPT-4o or Gemini 2.5 Pro. These LLMs are tasked with clarifying vague expressions and explicating implicit details within the original prompts, thereby minimizing confounding factors. We detail the prompts from VideoPhy using the following template:
You are a reasoning expert. Your task is to examine the given user prompt and identify whether there is any implicit knowledge that should be made explicit in the description. Your goal is to refine the prompt by making all details clear and descriptive, ensuring that no reasoning is required for understanding the outcome, environment, or processes involved. This means removing any assumptions or implicit components, such as environmental context, sequence of actions, the cause-and-effee process, that are not immediately obvious.
Please rewrite the original prompt in a clear, descriptive manner, without including any formulas or unnecessary reasoning, while providing as much detail as possible about the scene, actions, and effects. You should create polished version of the prompt where the outcome is immediately clear to the reader, leaving no room for ambiguity. Some in-context examples are provided for your reference, and you need to finish the current task: ...
Original prompt: A blender spins, mixing squeezed juice within it. Let's think step by step. The refined prompt should be less than 150 words.
F
User study
While quantitative metrics provide useful indicators of video generation performance, they may fully capture suman-perceived physical plausibility. To complement the quantitative evaluation and directly assess perceptual realism, we conducted a user study using the Good SroodSSameBaar Bad (GSB) pairwise comparison protocol [14] Participants were shown videos generated by VideoREPA-SB
and Cog CogVideoX-BB (the finetuning baseline trained on the same dataset using only Lan) for 344 prompts from the VideoPhy dataset. Each video pair was evaluated along two axes: (i) Semantic Adherence (SA), measuring the alignment between the generated video and the input text prompt; and (ii) Physical Commonsense (PC), assessing the plausibility of physical dynamics such as object interactions, motion continuity, and causal consistency.
24

---

Table 7: User study results comparing VideoREPA and CogVideoX with respect Semantic Adherence (SA) and Physical Commonsense (PC). Values indicate the percentage of user preference.
Criterion
VideoREPA Wins
CogVideoX Wins
Tie
Semantic Adherence (SA)
6,4
3.8 8.0
89.8
Physical Commonsense (PC)
21.7
70.3
As summarized in Table 7 the two models perform comparably in semantic adherence and Video- REPA demonstrates substantial improvement physical plausibility, being preferred 21.7% of the time compared to 8.0% for the baseline, representing 2.7x relative gain. These results confirm that VideoREPA effectively captures human- peeneveee physical commonsense while maintaining semantic fidelity.
G More qualitative results
In this section, we present additional video generation results from our VideoREPA-SB model, along with outputs from the baseline Cog VideoX-SB. This comparison aims to further demonstrate the enhanced physical commonsense achieved by our method in the generated videos. Figures 6 to Bo illlutraae these direct comparisons, showcasing the superiority of VideoREPA. Additionally, to highlight its capabilities further, we display more videos generated by VideoREPA that exhibit strong physical plausibility in Figure aand Figure IO Red rectangles denote phenomena that violate physical commonsense for casier distinguish.
25

---

credit card swipes through the machine. (Appearance)
A person uses low heat setting on the hairdryer to gently dry their fine hair. (Reflection)
A man throwing stone acTOSS river. (Causality)
A message contained bottle sails across the open sea. (Buoyancy)
A dog playfully bats at balloon... (Appearance)
Figure 6: Qualitative results. The first row displays the outcomes of CogVideoX, and the second row presents the results of our VideoREPA.
26

---

A single scull rower uses one oar to propel boat. (Commonsensc)
person... pouring the remaining beer into waiting shot glass. (Fluid)
Perfume spraying from perfume bottle. (Fluid&gas)
...a globe is poked, causing it to spin its axis (Rigid body&rotation)
Parallel bars are shown from a side view with an athlete performing dips. (Commonsense)
Figure 7: Qualitative results. The first row displays the outcomes of CogVideoX, and the second row presents the results of our ViPcoREPA.
27

---

hot iron pressing over a crumpled shirt. (Commonsense)
Soap bar sliding off the ceramic dish. (Sliding)
Detergent flowing into bucket of water. (Fluid)
A person mops up puddle of water concrete floor... (Commonsense)
A spray bottle sprays cleaning solution onto countertop. (Causality)
Figure 8: Qualitative results. The first row displays the outcomes of Cog and the second row presents the results of our VideoREPA.
28

---

A wine bottle pours red blend into glass. (Fluid&ccausalitt)
Ball bounces off the floor. (Gravity)
Drops of honey smooth yogurt. (Causality)
large rolls down hillside... (Rigid body&roll)
waterfall cascades over jagged rocks.. (Fluid)
Figure 9: Qualitative results, displaying videos generated by our VideoREPA.
29

---

Perfume mist diffusing through the air. (Gas&causalitty)
Hands wring out water from soaked cloth, (Fluid)
The lifter's feet move slightly during... (Human&ccommonsesse)
Sour cream swirls in hot soup. (Fluid)
A tennis ball rolls down a grassy hill... (Light&shadow)
Figure 10: Qualitative results, displaying videos generated by our VideoREPA.
30


Reviews:
[{"review_id": "LUp4LGBi3q", "reviewer": "Reviewer_CNDL", "summary": "This work proposes __VideoREPA__, a video diffusion model finetuning method which introduces physics knowledge into diffusion-based pretrained video generation models, where, instead of the token-level features, the relations between token features are aligned with those of a self-supervised video understanding foundation model using the RPEA loss. The proposed TRD loss aligns the pair-wise spatial and temporal token similarities. The experiments are conducted on two different datasets and the results demonstrates that the proposed VideoREPA is better at generating physical-commonsense video contents wrt the physical plausibility metrics.", "questions": "The questions have been summarized in the weakness part.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "__Strengths__\n1. the motivation is clear, where this work is motivated by the REPA in T2I methods and the information gap is analysed between video diffusion models and the self-supervised pretrained models.\n2. the proposed solution takes the finetuning over pretrained VDM models into consideration, instead of directly use the original REPA loss, this work shifts it into spatial-temporal soft alignment. Note that the original REPA loss is introduced for accelerating training convergence.\n3. the experimental results demonstrates empirical gains on two different datasets wrt the quantitative metrics.\n4. the appendix is comprehensive, including implementation details and refinement strategy.\n\n\n__Weakness__\n1. This work highlights the improvement about the physical-commonsense, which is majorly validated with predefined metrics, however, considering the complexity of real world motions, these metrics may not accurately measure the plausibility of the generated motions. Detailed user study is required to demonstrated VideoREPA actually learns physical-commonsense.\n2. The impacts of different VFMs seem significant (Table 4), is there any possibility to take use multiple VFMs to complement each other.\n2. Are the shape of inputs of two models the same, considering the patch size of them may be different?", "was_revised": true, "final_justification": "The authors response has addressed my concerns, where the extended experimental results are solid to verify the lacked analysis about the proposed method.", "rebuttal": "We sincerely thank you for your thorough review and insightful comments. We are encouraged by your recognition of our paper's strengths, including its clear motivation, great novelty, strong performance and comprehensive writing. Your feedback is very helpful, and we address your questions below.\n\n> Q1. Detailed user study is required to demonstrated VideoREPA actually learns physical-commonsense.\n\nThank you for your good suggestion. We conducted a user study to directly compare the performance of our **VideoREPA** against the **CogVideoX** baseline. For this study, we adopted the **Good-Same-Bad (GSB) pairwise comparison** method [1], which conducts pairwise comparisons to assess relative video quality. Participants were shown videos generated by both models for the 344 prompts from the VideoPhy dataset and were asked to evaluate them on two criteria:\n1.  **Semantic Adherence (SA):** Which video better aligns with the text prompt?\n2.  **Physical Commonsense (PC):** Which video appears more physically realistic (e.g., regarding causality, object motion, fluid dynamics)?\n\nThe results are summarized in the preference table below:\n\n| User study | VideoREPA Wins | CogVideoX Wins | Tie |\n| - |-|-|-|\n| Semantic Adherence (SA) | 6.4% | 3.8% | 89.8% |\n| **Physical Commonsense (PC)** | **21.7%** | 8.0% | 70.3% |\n\nFor physical realism, **our VideoREPA was preferred nearly 2.7 times more often than the baseline (21.7% vs. 8.0%)**. And VideoREPA and CogVideoX are comparable in semantic adherence, validating the improvement of PC is not at the cost of video-text alignment (SA).  This user study confirms that VideoREPA generates videos that are significantly more physically plausible.\n\n> Q2. The impacts of different VFMs seem significant (Table 4), is there any possibility to take use multiple VFMs to complement each other.\n\nThank you for this insightful question. **The performance disparity observed in Table 4 likely stems from the scale of pre-training data and the resulting generalization capabilities of each VFM.** VideoMAEv2, which delivers the best performance (29.7 PC score), was pre-trained on millions of diverse, in-the-wild videos, enabling it to develop a robust, generalized understanding of physical dynamics. In contrast, models like VideoMAE and OmniMAE were trained on smaller datasets, leading to a comparatively weaker grasp of intuitive physics. We also **experimented with another powerful VFM, V-JEPA 2 [2]**, which also achieved a strong PC score of 27.3, reinforcing the idea that VFMs resulting from large-scale pre-training possibly have a better physical understanding.\n\nFollowing your suggestions, we **conducted new experiments to explore whether combining VFMs could yield complementary benefits.** We implemented this by applying a weighted average of the TRD losses from multiple VFMs. The results of these combination experiments are summarized below:\n\n|Models|SA|PC\n|-|-|-\n|- (baseline) |63.6|23.2\n|VideoMAE|59.8|26.2\n|V-JEPA|64.5 (2)|24.7\n|OminiMAE|61.6|24.7\n|OminiMAE+VideoMAE|62.5|25.6\n|OminiMAE+VideoMAE+VideoMAEv2|63.4|27.9 (2)\n|V-JEPA 2|65.1 (1)|27.3 (3)\n|VideoMAEv2|64.2 (3)|29.7 (1)\n\n*Numbers in parentheses indicate the rank (1st, 2nd, 3rd).\n\nOur findings are quite interesting:\n\n- Complementary Strengths: We found that combining models can indeed balance their individual strengths and weaknesses. For example, using only VideoMAE resulted in a lower SA score (59.8), while OmniMAE had a weaker PC score (24.7). Combining them (OmniMAE+VideoMAE) improved both SA (to 62.5) and PC (to 25.6) relative to their respective weak points.\n\n- Synergy with a Stronger Model: Adding the powerful VideoMAEv2 into the OminiMAE+VideoMAE further boosted performance, achieving a PC score of 27.9. While this is still lower than using VideoMAEv2 alone, it demonstrates that weaker VFMs can benefit from the combination of other stronger models. This opens a promising research direction: investigating whether advanced ensemble techniques or more sophisticated mixing strategies could yield a combined model that outperforms any individual constituent model.\n\n> Q3. Are the shape of inputs of two models the same, considering the patch size of them may be different?\n\nNo, the input shapes for the two models are different. We address this misalignment through a multi-step process, as detailed in Lines 241-256 in the main paper and Lines 559-580 of our appendix.\n\n1.  **Input Pre-processing for VFM:** The input for our base model, CogVideoX, is fixed at a high resolution (49 frames, 480x720). Feeding this directly into the VFM (e.g., VideoMAEv2) is computationally prohibitive. Therefore, we **uniformly downsample the video frames** to a lower resolution before passing them to the VFM. This strategy was chosen after exploring several trade-offs (Lines 568-572) and was found to best preserve the holistic temporal dynamics with manageable computational cost.\n\n2.  **Handling Patch Size and Latent Misalignment:** The downsampled resolution for the VFM is chosen to be a multiple of its patch size to ensure compatibility. Even after this, a dimensional misalignment often remains between the VFM's feature outputs and the VDM's latent representations (due to different temporal compression ratios, etc.). To reconcile this, we **interpolate the VDM's latent features to match the VFM's feature dimensions** before calculating the TRD loss. This approach was empirically found to be the most effective (Table 5).\n\n**To be specific**, while CogVideoX processes the 49x480x720 video, the corresponding input for VideoMAEv2 is downsampled to 48x160x240. (Note we discard the first frame for the VFM alignment, as it primarily serves to maintain static semantic information in the VDM's 3D VAE and our focus is on distilling dynamic content (Lines 565-567)).\n\n[1] Gao Y, Guo H, Hoang T, et al. Seedance 1.0: Exploring the Boundaries of Video Generation Models[J]. arXiv preprint arXiv:2506.09113, 2025.\n\n[2] Assran M, Bardes A, Fan D, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning[J]. arXiv preprint arXiv:2506.09985, 2025."}, {"review_id": "Jj6njVDask", "reviewer": "Reviewer_vo9S", "summary": "This paper proposed a training strategy to advance the original video generation model with better physical consistency. It proposed an REPA learning strategy which aligns the feature from MM-DiT with the feature from a pretrained video model (e.g. VideoMAE).\nVideoREPA is applied onto CogVideoX model and evaluated on two datasets: VideoPhy and VideoPhy2. On both semantic consistency and physical commonsense metrics, the proposed method performs better than original model.", "questions": "Please see the weakness part", "limitations": "1. VideoREPA is a general technique which is compatible with all video generation foundation model. But the performance is only reported on CogVideoX model. It didn't show any generalization capability to other base models.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed method shows better performance on both VideoPhy and VideoPhy2 datasets\n\nWeakness\n1. The performance gain from VideoREPA compared to original CogVideoX is very marginal, especially in the performance on SA (semantic adherence)\n\n2. from the qualitative results, it seems the results from VideoREPA has more regular-shaped objects compared to original CogVideoX model, is this the reason why the physical related metrics could be improved? Because the objects in the generated videos have simpler shapes, leading to better physically interaction results\n\n3. There is no physical performance evaluation on VideoMAE or VJEPA to validate why the features from MMDiT should be aligned to VideoMAE features.\n\n4. The ablation study is not very comprehensive, e.g. it didn't explore different VFM but only uses VideoMAE. And it also didn't explore different alignment learning strategy but only tried REPA", "was_revised": true, "final_justification": "The rebuttal has resolved my questions.", "rebuttal": "Thank you for taking the time to review our paper. We appreciate the chance to clarify our work and address your concerns, including any possible misunderstandings, in the responses below.\n\n### Weakness\n\n> Q1. The performance gain from VideoREPA compared to original CogVideoX is very marginal, especially in the performance on SA (semantic adherence).\n\nOur VideoREPA improves PC significantly and maintain SA. This is reasonable **because the primary goal of VideoREPA is not to enhance video-text alignment (SA), but to improve Physical Commonsense (PC)**.\n\nAs defined in the VideoPhy [1] and VideoPhy2 [2], PC and SA measure two distinct capabilities:\n\n*   **Physical Commonsense (PC):** Evaluates if the video's content (e.g., object motion, fluid dynamics, causality) adheres to real-world physical laws.\n*   **Semantic Adherence (SA):** Measures the alignment between the video content and the text prompt.\n\nThe SA score is primarily monitored to ensure that our method does not achieve better physics at the cost of semantic fidelity by generating physically simpler, but less relevant, content.\n\nThe results clearly demonstrate that VideoREPA achieves its primary goal. We observed a substantial **24.1% relative improvement in the PC**, while maintaining (or improving) the SA.\n\nModel|SA|PC\n-|-|-|\nCogVideoX-5B|70.0|32.3\nVideoREPA-5B (Ours)|72.1|**40.1 (+24.1%)**\n\n> Q2. from the qualitative results, it seems the results from VideoREPA has more regular-shaped objects compared to original CogVideoX model, is this the reason why the physical related metrics could be improved? Because the objects in the generated videos have simpler shapes, leading to better physically interaction results.\n\nThank you for the question. While it is true that VideoREPA generates objects with more regular and stable shapes, which contributes to physical plausibility, this is only **one facet of the overall improvement** in PC. The PC evaluates a wide range of physical phenomena, and our qualitative results demonstrate that VideoREPA enhances many of these, far beyond just object shape.\n\nWe highlight several examples below, **directly contrasting our results with the baseline** to demonstrate these multi-faceted improvements:\n\n*   **Causality and Interaction Logic:**\n    *   A glove correctly catches a baseball, unlike the baseline where the ball is incorrectly caught *behind* the glove (Fig. 1, row 1).\n    *   A glass shatters realistically upon hitting the floor, whereas the baseline shows unnatural motion (Fig. 1, row 2).\n    *   A crane maintains a direct physical connection while lifting bricks, unlike in the baseline (Fig. 3, row 2).\n    *   Detergent realistically mixes with water, causing the water level to rise, which is not depicted correctly in the baseline (Fig. 8, row 3).\n\n*   **Buoyancy:**\n    *   A bottle correctly floats on the sea's surface, while the baseline's bottle unnaturally lifts off the water (Fig. 6, row 3).\n\n*   **Light Reflection:**\n    *   A person using a hairdryer is accurately reflected in the mirror, correcting the distorted reflection seen in the baseline (Fig. 6, row 2).\n\n*   **Motion and Dynamics:**\n    *   Generating plausible rigid body motion, like a container rolling down a hill (Fig. 3, row 1) or a pencil rolling correctly (Fig. 3, row 2).\n    *   Modeling correct fluid dynamics, such as perfume spraying as a gas/fluid mixture (Fig. 7, row 3).\n    *   Adhering to principles like gravity (Fig. 9, row 2).\n\n*   **Shape Integrity:**\n    *   A credit card maintains its shape integrity when swiped, whereas the baseline's card partially vanishes (Fig. 6, row 1).\n\n\nTherefore, the enhanced physical realism stems not from merely generating correct shapes, but from a **comprehensive improvement in modeling plausible motion, causality, and interactions** that adhere to real-world physics. This is a direct result of our method's core mechanism: the **TRD loss**, which distills a deeper, relational understanding of both **spatial configurations (leading to better shapes and integrity) and temporal dynamics (leading to better motion and causality)** from the VFM.\n\n> Q3. There is no physical performance evaluation on VideoMAE or VJEPA to validate why the features from MMDiT should be aligned to VideoMAE features. \n\nThe justification for aligning VDMs with VFMs like VideoMAEv2 is two-fold. First, as we demonstrate in our **Figure 1 evaluation on the Physion benchmark**, the VideoMAEv2 possess a **significantly stronger physics understanding ability** than the baseline VDM. This finding is also supported by recent studies [3, 4] on the emergent physics understanding capability in such models.\n\nSecond, our work is based on the established principle that **enhanced understanding facilitates better generation** (Lines 151-166). Given the clear \"physics understanding gap\" we identified, we proposed VideoREPA to distill this superior understanding from the VFM into the VDM, thereby close understanding gap and enhancing the physical plausibility of the generated videos.\n\n> Q4.1. The ablation study is not very comprehensive, e.g. it didn't explore different VFM but only uses VideoMAE.\n\nWe would like to clarify that an ablation study on different VFMs **was included in Appendix C (Table 4, Lines 550-553)**, where we compared VideoMAE, V-JEPA, OmniMAE, and VideoMAEv2.\n\nTo further address your concern and to make our study more comprehensive, we have now included another powerful VFM, **V-JEPA 2 [5]**, and also explored various **combinations of VFMs**, as suggested by Reviewer CNDL. These new results not only demonstrate the flexibility of our VideoREPA framework but also provide deeper insights into how different VFMs contribute to performance (see answer to Reviewer CNDL Q2).\n\n|Models|SA|PC\n|-|-|-\n|- (baseline) |63.6|23.2\n|VideoMAE|59.8|26.2\n|V-JEPA|64.5 (2)|24.7\n|OminiMAE|61.6|24.7\n|OminiMAE+VideoMAE|62.5|25.6\n|OminiMAE+VideoMAE+VideoMAEv2|63.4|27.9 (2)\n|V-JEPA 2|65.1 (1)|27.3 (3)\n|VideoMAEv2|64.2 (3)|29.7 (1)\n\n*Numbers in parentheses indicate the rank (1st, 2nd, 3rd).\n\n*Note: With comparable SA scores, the PC score is the predominant metric for assessing the physics plausibility.*\n\nLastly, we'd like to clarify a potential point of confusion: our primary VFM used throughout the paper is the powerful **VideoMAEv2, not VideoMAE**. VideoMAEv2's extensive pre-training on millions of videos makes it a significantly stronger foundation model for understanding physics than VideoMAE.\n\n> Q4.2. And it also didn't explore different alignment learning strategy but only tried REPA.\n\nOur contribution is not an direct application of REPA, **but the creation of VideoREPA, a novel framework with a new loss function (TRD loss)** designed specifically for physics knowledge transfer. We found REPA unsuitable as it is spatial-only and its \"hard\" feature alignment destabilizes pre-trained VDMs during finetuning (as shown in Fig. 4). Our \"soft,\" relational TRD loss was developed precisely to overcome these critical failures.\n\nTo the best of our knowledge, no other off-the-shelf alignment methods are designed for our task. Thus no available alternatives for ablation and the exploration of different alignment methods needs further consideration.\n\nHowever, we explored variants of our own strategy in Table 3. By ablating the spatial and temporal components, we evaluated different alignment approaches within our framework and confirmed that the full spatio-temporal design is the most effective.\n\n### Limitations\n\n> Q5. The performance is only reported on CogVideoX model. It didn't show any generalization capability to other base models.\n\nThank you for your suggestion. We mainly validated the effectiveness of VideoREPA on the CogVideoX model at two different scales (2B and 5B), demonstrating its strong performance and scalability in our paper. Following your advice, we have now applied VideoREPA to another powerful T2V model: Wan2.1 (1.3B) [6]. The results on the VideoPhy are shown below:\n\nModels|SA|PC\n-|-|-\nWan2.1|67.1|21.5\nVideoREPA (ours)|66.9|**26.7**\n\nThe results clearly show that VideoREPA significantly improves the PC score while maintaining SA. This provides empirical evidence for the generalization capability of our proposed method, demonstrating its effectiveness beyond the CogVideoX architecture. \n\n### Paper formatting concerns\n\n> Q6. The abbreviation VFM appears earlier than the full name video foundation model.\n\nThank you for the reminder. We initially abbreviated \"Video Foundation Model\" to VFM for clarity and to avoid repeating the lengthy phrase. However, given that VFM can also refer to \"Vision Foundation Model\" [7], we will change the abbreviation to ViFM in the revised manuscript to prevent any conceptual confusion. This change will be shown in next version of our paper.\n\n**Reference**\n\n[1] Bansal H, Lin Z, Xie T, et al. Videophy: Evaluating physical commonsense for video generation[J]. arXiv preprint arXiv:2406.03520, 2024.\n\n[2] Bansal H, Peng C, Bitton Y, et al. Videophy-2: A challenging action-centric physical commonsense evaluation in video generation[J]. arXiv preprint arXiv:2503.06800, 2025.\n\n[3] Bordes F, Garrido Q, Kao J T, et al. IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments[J]. arXiv preprint arXiv:2506.09849, 2025.\n\n[4] Garrido Q, Ballas N, Assran M, et al. Intuitive physics understanding emerges from self-supervised pretraining on natural videos[J]. arXiv preprint arXiv:2502.11831, 2025.\n\n[5] Assran M, Bardes A, Fan D, et al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning[J]. arXiv preprint arXiv:2506.09985, 2025.\n\n[6] Wan T, Wang A, Ai B, et al. Wan: Open and advanced large-scale video generative models[J]. arXiv preprint arXiv:2503.20314, 2025.\n\n[7] Wang W, Chen Z, Chen X, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks[J]. Advances in Neural Information Processing Systems, 2023, 36: 61501-61513."}, {"review_id": "NMyj0TeirB", "reviewer": "Reviewer_ryaw", "summary": "VideoREPA introduces an approach to enhance the physical plausibility of videos generated by T2V diffusion models. The method leverages an improved version of REPA by distilling physics knowledge from video foundation models into video diffusion models, focusing on finetuning rather than training from scratch.", "questions": "- In Figure 1 (right), the evaluation of physics understanding extracts intermediate features from VDMs to predict video futures (according to Appendix B). This approach raises concerns because VDMs are trained for denoising, not for extracting rich features like MAEs. My questions (or concerns) are:\n1. Is it fair to compare **noisy activations** from VDMs with the structure features of MAEs?\n2. Can you claim that large VDMs understand physics worse than smaller VFMs based on this unfair comparison (L47-50)? \n3. Doesn't the enhanced alignment is a trivial results by directly aligning the feature with MAE, rather than the results of truly enhanced physics understanding?\n- Why does the spatial component of the TRD loss not exclude the case $i=j$, while the temporal component does exclude $d=e$? Please clarify / justify this design choice.\n- The quantitative results show VideoREPA-5B outperforming HunyuanVideo, yet visually the latter appears better in quality. Why was HunyuanVideo not used as the baseline model?", "limitations": "The authors addressed the limitations of their work.\n\nCommon comment for the authors: If I couldn't fully understand your methods or am underestimating your contributions, feel free to explain and let's discuss. I will gladly raise the scores if I have misunderstood key points or if my concerns are addressed well.\n\nI think the most critical points are:\n- The first weakness described above.\n- Validity of the comparison on Figure1 right (My first question).\n- A broader quantitative comparison of video fidelity, consistency, and text-video alignment", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n- The paper proposes a novel way to adapt REPA for finetuning, shifting its original scope from training acceleration to knowledge transfer.\n- Experimental results and video supplements demonstrate that the physical plausibility of generated videos improves using the proposed method.\n- The method effectively addresses the potential issue of feature dimension mismatch between VFMs and VDMs through interpolation.\n\n### Weakness\n- It remains unclear why the **physical knowledge** of VFMs should be distilled into VDMs as proposed. The manuscript gives the impression that enhanced physical plausibility is merely a training byproduct, not an explicit goal. The justification for why the TRD loss contributes to physics understanding needs to be clarified.\n- The experiments are limited to CogVideoX-2B and 5B, which share the same structure and are now relatively outdated. Demonstrating the method's integration with more recent and diverse open-source video generation models would strengthen its credibility.\n- A broader quantitative comparison of video fidelity, consistency, and text-video alignment (e.g. using VBench) is lacking. Without this, it is unclear whether the improvement in physical plausibility compromises other aspects of video quality.", "was_revised": true, "final_justification": "I justified the decision to increasing the score in the final comment.", "rebuttal": "Thank you for the time, thorough comments, and questions. We appreciate your endorsement of our method in the aspects of novelty, performance and careful method design. Great to hear that you are willing to raise your score if concerns addressed and we'd be happy to clarify certain points for your.\n\n### Weakness\n\n> Q1. It remains unclear why the physical knowledge of VFMs should be distilled into VDMs as proposed. The manuscript gives the impression that enhanced physical plausibility is merely a training byproduct, not an explicit goal. The justification for why the TRD loss contributes to physics understanding needs to be clarified.\n\nThank you for this question. We want to emphasize that **enhancing physical plausibility is the explicit, primary goal of our framework, not a training byproduct.** Our entire research pipeline—from motivation to method design—is centered around this objective.\n\n**1. Motivation: Why Distill from VFMs to VDMs?**\n\nOur work stems from a clear, two-part observation: 1) (Problem) Poor Physics Plausibility and Understanding in VDMs. 2) (Opportunity) Strong Physics Understanding in VFMs.\n\n**Given that understanding generally benefits generation**, this significant physics understanding gap motivated our core idea: to bridge this gap by **distilling the emergent physical knowledge** from capable VFMs into powerful VDMs.\n\n**2. Method Design: Why TRD Loss Contributes to Physics Knowledge Transfer?**\n\nThe design of our Token Relation Distillation (TRD) loss is tailored specifically to capture the essence of physical knowledge. Real-world intutive physics is fundamentally about relationships: how objects are spatially configured and how these configurations evolve dynamically over time. Our TRD loss directly mirrors this structure:\n\n*   **Spatial Relation Alignment:** This component enforces plausible object configurations, shape integrity, and static interactions (e.g., an object resting *on* a surface).\n*   **Temporal Relation Alignment:** This component captures the dynamics of motion, causality, and state changes (e.g., how an object moves from frame A to B).\n\nBy aligning these relational matrices, we are not just matching features; we are forcing the VDM to learn the underlying **structural and dynamic rules** that govern a physical scene, as understood by the VFM.\n\n**3. Empirically Validating the Transfer of Physics Knowledge into VDMs**\n\nOur extensive experiments validate our claim in 2. The **physics understanding evaluation (Fig. 1)** confirms that TRD loss successfully transfers understanding capability. Subsequently, the **generation benchmarks (Table 1 & 2)** show this enhanced physics understanding induced by TRD loss translates into significantly improved physical plausibility in videos. This aligns with the established principle that understanding helps generation and **further shows that physical knowledge is indeed being distilled effectively.**\n\n> Q2. Demonstrating the method's integration with more recent and diverse open-source video generation models.\n\nFollowing your advice, we integrated VideoREPA with another open-source model: Wan2.1 (1.3B) [1]. The results on the VideoPhy:\n\nModels|SA|PC\n-|-|-\nWan2.1|67.1|21.5\nVideoREPA (ours)|66.9|**26.7**\n\n**It shows that VideoREPA significantly improves upon the Wan2.1 in terms of PC (physical plausibility)**. This validates the generalization capability of our proposed method. Further visulizations will be updated in the next version of our paper.\n\n> Q3. A broader quantitative comparison of video fidelity, consistency, and text-video alignment (e.g. using VBench) is lacking.\n\nTo verify that our improvement in physical plausibility does not compromise other aspects, we conducted experiments on VBench [2].\n\nThe results, presented below, show that **VideoREPA is comparable to the CogVideoX across every dimension**. This provides strong evidence that our method successfully enhances physical plausibility without sacrificing general video fidelity, consistency, or text-video alignment.\n\nModel|Subject Consistency|Background Consistency|Temporal Flickering|Motion Smoothness |Dynamic Degree|Aesthetic Quality|Imaging Quality|Quality score (Overall)\n-|-|-|-|-|-|-|-|-\nCogVideoX|93.31|93.90|97.74|98.04|62.22|62.82|63.06|81.38 \nVideoREPA (Ours)|93.46|93.95|97.76|98.05|61.39|62.72|63.29|81.38 \n\nModel|Object Class|Multiple Objects|Human Action|Color|Spatial Relationship|Scene|Appearance Style|Temporal Style|Overall Consistency |Semantic Score (Overall)\n-|-|-|-|-|-|-|-|-|-|-\nCogVideoX|89.73|71.72|99.00|84.08|75.12|53.13|24.44|25.23|27.28|79.35\nVideoREPA (Ours)|89.60|71.40|98.20|85.37|76.03|53.05|24.49|25.24|27.28|79.45\n\n### Questions\n\n> Q4.1. Is it fair to compare noisy activations from VDMs with the structure features of MAEs?\n\nYes, we argue that this is a fair and informative comparison. \n\nWhile VDMs are trained for denoising, this process is itself a powerful form of self-supervised learning. Previous studies [3, 4] confirm that features from diffusion models are also structured and effective for downstream understanding tasks. **Therefore, evaluating these features for physics understanding is a valid approach.**\n\nAlso, we build our evaluation protocol **following previous studies [5, 6]** which also evaluate the representation in diffusion models for understanding. Using a linear probing protocol for both, **we maximized the information extracted from the VDM's features evaluating different noise levels.** To our knowledge, this is a direct and appropriate way to probe the understanding capability of these specific representations.\n\n> Q4.2. Can you claim that large VDMs understand physics worse than smaller VFMs based on this unfair comparison (L47-50)?\n\nGiven that our comparison is fair (as justified in our response to Q4.1), we can indeed claim that large VDMs understand physics worse than smaller VFMs: Our experiments showed a significantly weaker physics understanding for the large VDM; specifically, the VDM (2B) achieved 62% accuracy on the Physion (roll subset), while the much smaller VFM (86M) achieved 72%.\n\nThis finding aligns with other studies [7, 8], which point out that VFMs learn powerful representations capable of performing physics understanding tasks and achieving strong results.\n\n> Q4.3. Doesn't the enhanced alignment is a trivial results by directly aligning the feature with MAE, rather than the results of truly enhanced physics understanding?\n\nNo, the improvement is not a trivial result of feature alignment for two reasons. \n\nFirst, We show that the **improved physics understanding** (Fig. 1) **translates to enhanced physical plausibility in the generated videos** (Tab. 1). If the understanding improvement were merely an artifact of the evaluation setup, it would not lead to more physically realistic video, which is an independent task.\n\nSecond, our qualitative results demonstrate a **comprehensive improvement across a wide range of physical phenomena**, including motion, causality, and complex interactions (details in answer to Reviewer vo9S, Q2). A trivial alignment would likely only improve superficial aspects, whereas our method achieves a broad and consistent enhancement of physical realism, suggesting a deeper, more fundamental transfer of physics knowledge.\n\n> Q5. Why does the spatial component of the TRD loss not exclude the case $i = j$, while the temporal component does exclude $d = e$?\n\nThis design choice is intentional and reflects the distinct roles of the spatial and temporal components in our TRD loss:\n\n1. The spatial component calculates the cosine similarity between all pairs of tokens $(i, j)$ **within a single frame** $d$. **When $i = j$:** The case represents the self-similarity of a token. It is 1, i.e., $y_{{spatial}}^{d,i,i} = \\frac{{y} _ {d,i} \\cdot {y} _ {d,i}}{||{y} _ {d,i}|| ||{y} _ {d,i}||} = 1$. Thus both the $y$ and $h$ terms equal to 1. The contribution to the TRD loss is always zero: $|h_{{spatial}}^{d,i,i} - y_{{spatial}}^{d,i,i}| = |1 - 1| = 0$. \n\n    Since these terms have no impact on the loss value and gradients, we kept them for notational and implementation simplicity.\n\n2. The temporal component is designed specifically to capture the relationships between tokens **across different frames (i.e., $d\\neq e$)**. Including the $d=e$ means calculating the similarity of tokens within the same frame, which is the **spatial component**, resulting double-counting.\n\n\n> Q6. The quantitative results show VideoREPA-5B outperforming HunyuanVideo, yet visually the latter appears better in quality. Why was HunyuanVideo not used as the baseline model?\n\nWhile HunyuanVideo may exhibit superior visual quality due to its larger scale, our research prioritizes **improving physical plausibility**, which is a distinct and critical challenge. Existing benchmarks like VideoPhy2 [9] show that CogVideoX outperforms HunyuanVideo in physical commonsense, as verified by human evaluation. Thus we chose CogVideoX as a stronger baseline.\n\n[1] Wan: Open and advanced large-scale video generative models[J]. arXiv:2503.20314, 2025.\n\n[2] Vbench: Comprehensive benchmark suite for video generative models[C]//CVPR 2024\n\n[3] Diffusion models as masked autoencoders[C]//CVPR 2023\n\n[4] Emergent correspondence from image diffusion[J]. NuerIPS 2023.\n\n[5] U-repa: Aligning diffusion u-nets to vits[J]. arXiv:2503.18414, 2025.\n\n[6] Representation alignment for generation: Training diffusion transformers is easier than you think[J]. arXiv:2410.06940, 2024. \n\n[7] IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments[J]. arXiv:2506.09849, 2025.\n\n[8] Intuitive physics understanding emerges from self-supervised pretraining on natural videos[J]. arXiv:2502.11831, 2025.\n\n[9] Videophy-2: A challenging action-centric physical commonsense evaluation in video generation[J].arXiv:2503.06800, 2025."}, {"review_id": "XcOkuXPFfT", "reviewer": "Reviewer_JMpo", "summary": "This work focuses on ensuring that text-to-video (T2V) models generate physically plausible content. Authors identify that the main limitation of current models is that the representations within such models have reduced capacity for physics understanding in contrast to self-supervised methods that focus on representational learning. Thus, a method to distill physics understanding from such foundational models into T2V models. This method enables fine-tuning of T2V to inject physical knowledge. The authors perform experiments comparing their results with CogVideoX.", "questions": "Questions were included in the previous section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths \n* The core idea proposed by the authors is interesting and sound. Authors propose to leverage the representation capabilities of foundational video models, e.g., DinoV2 or VideoMAE2 to align the representations used by text-to-video latent diffusion models and thus capture more realistic and consistent dynamics (e.g., consistent object shapes and physical plausibility of how objects move and interact with other objects). At the same time the authors devise a method to achieve this alignment by enforcing relation similarity between token embeddings. Thus the alignment is not done directly over the representations but over their relations. This is a softer condition useful for fine-tuning as opposed to REPA. \n* Qualitative results look good and the authors selected a set of videos that clearly show the difference between the baselines and the proposed method results. Quantitative evaluation is aligned with these results. \n* In general the presentation is good (but writing could be improved). \n\n# Weaknesses\n\nIn order of importance:\n\n**Vague Link Between SSL Encoders and Physics**\nThe method claims to “enhance physics” by distilling knowledge from self-supervised video encoders. However, SSL encoders (e.g., DINOv2, or VideoMAE2) are typically trained to capture general motion or appearance representations — not necessarily physical plausibility or constraints like dynamics, contact, or force interactions.\n\nWhat evidence is there that SSL video encoders actually encode meaningful physics properties (e.g., Newtonian constraints, conservation laws, contact dynamics) rather than just correlation patterns or appearance flow? Also, how do the authors ensure that the distilled relations truly correspond to physically plausible relations rather than dataset-specific visual correlations?\n\n**Ambiguity in ‘Pairwise Token Similarity’ = Physics**\nThe logic assumes that aligning pairwise token similarities (via TRD loss) transfers physics. But pairwise similarity can capture texture, background motion, or trivial frame correlations — it doesn’t inherently encode contact constraints, collision avoidance, or kinematic feasibility. Why should pairwise token similarities enforce physically realistic behavior? Could this encourage spurious correlations instead of actual dynamic consistency?\n\n**Writing and Clarity**\n\nThe writing could be improved to enhance readability and ensure a smooth flow of information and transitions between ideas. For example, Lines 40–41: *“Yet, WISA struggles to generalize to open-domain data, showing improvements primarily when trained on videos with explicit physics.”* This sentence is confusing and makes the paper more difficult to follow. Similar unclear phrasing can be found throughout the manuscript. Clarifying these statements would significantly improve the overall quality and accessibility of the paper.", "was_revised": true, "final_justification": "The response is clear, concise and it addresses my 3 concerns, especially the first 2 ones. Thus, I im inclined to keep my score.", "rebuttal": "Thank you very much for your careful review and the recognition of our novelty, performance and great presentation. We clarify your questions below.\n\n> Q1.1. **Vague Link Between SSL Encoders and Physics** The method claims to “enhance physics” by distilling knowledge from self-supervised video encoders. However, SSL encoders (e.g., DINOv2, or VideoMAE2) are typically trained to capture general motion or appearance representations — not necessarily physical plausibility or constraints like dynamics, contact, or force interactions. What evidence is there that SSL video encoders actually encode meaningful physics properties (e.g., Newtonian constraints, conservation laws, contact dynamics) rather than just correlation patterns or appearance flow?\n\nThanks for your question. It touches upon the core motivation of our work. It is partially correct that VFMs are typically trained to capture general motion or appearance representations. Apart from these features, **recent studies [1, 2, 3] point out that SSL representations capture high-level features (e.g. physics knowledge) beyond low-level appearance or motion.** For example, the paper [3] provides extensive experiments to show that VFMs like V-JEPA and VideoMAEv2 learn representations that align with physical concepts like object permanence and trajectory prediction without any explicit physics supervision, achieving great intuitive understanding ability.\n\n**We also validate that VFMs indeed encode physics information by assessing their ability on the Physion dataset** (Figure 1), a benchmark for tasks like object dynamics and contact prediction. The prediction accuracy of VFM VideoMAEv2 exceeds 70%, which is significantly higher than the 50% chance level and GPT-4 V only achieves 53%. This level of performance would be unattainable if the models were learning only superficial correlation patterns or appearance flow, rather than a deeper understanding of intuitive physics.\n\nThus while it is true that Video SSL encoders like VideoMAEv2 are not explicitly trained with physics constraints (i.e., not learning explicit physics laws like Newtonian law, value of gravity and etc.), there is compelling evidence that they learn a rich, implicit understanding of intuitive physics which makes it able to predict, detect object dynamics and etc.\n\n> Q1.2. Also, how do the authors ensure that the distilled relations truly correspond to physically plausible relations rather than dataset-specific visual correlations? \n\nWe ensure and validate this from two aspects. **Theoretically**, the distilled relations induced by carefully designed TRD loss contain spatial and temporal component which correspond to exactly the physics knowledge contained in videos. Because real-world physics is fundamentally about relationships including how objects are spatially configured (sptial component) and how these configurations evolve dynamically over time (temporal component) (details in answer to Q2).\n\n**Experimentally**, as shown in Table 1, our VideoREPA method leads to a substantial +24.1% improvement in the overall PC score for CogVideoX-5B. Also, further user study shows the supuriority of our VideoREPA in terms of physical realism in generated videos (The details are shown in the answer to Rewiewer CNDL, Q1). If we were merely distilling dataset-specific visual correlations, it is highly unlikely that this would translate into a consistent and significant improvement on a diverse set of prompts designed to test fundamental physical principles (e.g., solid-solid interaction, fluid dynamics, causality).\n\n**In summary of the answer to Q1**, we are not claiming that Video SSL encoders solve explicit physics equations or encode precise physical property / value. We are arguing, with evidence from our own work and the existing works, that they learn a powerful, implicit model of intuitive physics. Our results show that distilling this emergent knowledge into T2V models is an effective strategy for enhancing their physical realism. \n\n> Q2. **Ambiguity in ‘Pairwise Token Similarity’ = Physics** The logic assumes that aligning pairwise token similarities (via TRD loss) transfers physics. But pairwise similarity can capture texture, background motion, or trivial frame correlations — it doesn’t inherently encode contact constraints, collision avoidance, or kinematic feasibility. Why should pairwise token similarities enforce physically realistic behavior? Could this encourage spurious correlations instead of actual dynamic consistency?\n\nBecause we design TRD loss using spatial-temporal pair-wise similarity to capture physics knowledge intentionally for effective transfer. Real-world intutive physics is fundamentally about relationships including how objects are spatially configured and how these configurations evolve dynamically over time. Our TRD loss directly mirrors this structure:\n\n*   **Spatial Relation Alignment:** This component captures plausible object configurations, shape integrity, and static interactions (e.g., an object resting *on* a surface).\n*   **Temporal Relation Alignment:** This component captures the dynamics of motion, causality, and state changes (e.g., how an object moves from frame A to B).\n\nBy aligning these relational matrices, we are not just matching features; we are forcing the VDM to learn the underlying **structural and dynamic rules** that govern a video, as understood by the VFM.\n\nFurthermore, **the empirical success of our method also provides the strongest evidence that we are distilling meaningful physics, but not spurious correlations**. As shown in Figure 1 (right), our method significantly closes the physics understanding gap between the T2V model and the VFM. This transferred understanding directly translates into better generation quality, evidenced by the substantial +24.1% improvement on the PC score in the VideoPhy benchmark (Table 1).\n\n> Q3. **Writing and Clarity** The writing could be improved to enhance readability and ensure a smooth flow of information and transitions between ideas. For example, Lines 40–41: “Yet, WISA struggles to generalize to open-domain data, showing improvements primarily when trained on videos with explicit physics.” This sentence is confusing and makes the paper more difficult to follow. Similar unclear phrasing can be found throughout the manuscript. Clarifying these statements would significantly improve the overall quality and accessibility of the paper.\n\nThanks for your valuable suggestions regarding the writing. We apologize for any unclear phrasing and have carefully revised the manuscript to improve clarity and flow. Below are three examples of such improvements that will be reflected in the next version of our paper:\n\n1. Lines 40-41:\n\n    Original: \"Yet, WISA struggles to generalize to open-domain data, showing improvements primarily when trained on videos with explicit physics.\". \n    \n    Modified: \"Yet, WISA only show improvement when training with dedicated collected dataset WISA-32K where each relates to individual explicit physical phenomena. It struggles to generalize to open-domain data which is much easily to collected and scale.\"\n\n    Comment on Improvement: We replaced the vague phrase \"struggles to generalize\" with a more concrete explanation of WISA's dependency on its specialized dataset (WISA-32K).\n\n2. Lines 66-67:\n\n    Original: \"thereby improving the physical realism of generated videos without relying on physics explicit datasets (e.g., WISA-32K).\"\n\n    Modified: \"thereby improving the physical realism of generated videos. Our method performs well on open-domain video dataset OpenVid without relying on the physics explicit dataset WISA-32K which is much harder to colloct.\"\n\n    Comment on Improvement: We restructured the sentence to first state our method's core benefit and then explicitly contrast its data requirements with WISA's.\n\nWe made revisions throughout the paper to enhance its overall quality and accessibility. We are committed to further improving the text and welcome any additional feedback. Thank you again.\n\n[1] Yu S, Kwak S, Jang H, et al. Representation alignment for generation: Training diffusion transformers is easier than you think[J]. arXiv preprint arXiv:2410.06940, 2024.\n\n[2] Bordes F, Garrido Q, Kao J T, et al. IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environments[J]. arXiv preprint arXiv:2506.09849, 2025.\n\n[3] Garrido Q, Ballas N, Assran M, et al. Intuitive physics understanding emerges from self-supervised pretraining on natural videos[J]. arXiv preprint arXiv:2502.11831, 2025."}]

Author rebuttal:

Meta-review: This paper proposes VideoREPA, a work designed to bridge the “physics understanding gap” in text-to-video diffusion models by distilling relational knowledge from video foundation models (VFMs). The central contribution is the Token Relation Distillation (TRD) loss, which aligns spatial and temporal token relations rather than enforcing hard feature-level alignment. This approach directly targets the challenge of physical plausibility in generated videos and is supported by clear motivation, validated improvements, and demonstrated method design. The work is well-aligned with current trends, given increasing interest in improving controllability and realism in video generation.

Across the review process, the authors provided several clarifications and experiments that addressed initial concerns. They conducted a user study demonstrating clear improvements in physical realism, explored combinations of multiple VFMs to show complementary strengths, and validated generalization by applying VideoREPA to Wan2.1 in addition to CogVideoX. Further, evaluations on VBench and VBench2 confirmed that improvements in physics plausibility were achieved without compromising video quality or semantic adherence. These additions strengthened the robustness, and all reviewers acknowledged that their concerns had been resolved.

In sum, the paper makes a well-motivated and technically contribution by introducing a novel relational distillation strategy for enhancing physical common sense in video generation. While some limitations remain, the overall work is convincing and impactful. The consensus after discussion is positive, and I recommend acceptance.
Decision: Accept (poster)